Realsense SDK2.0 + C++

rs-hello-realsense

rs-hello-realsense‎示例演示了连接到英特尔实感设备以及通过打印到摄像头视野中心物体的距离来利用深度数据的基础知识。

‎假设相机已连接，您应该看到线路不断更新。X是相机视野中心物体的距离（以米为单位）。

#include <librealsense2/rs.hpp> // Include Intel RealSense Cross Platform API

// Create a Pipeline - this serves as a top-level API for streaming and processing frames
rs2::pipeline p;// Configure and start the pipeline
p.start();

// Block program until frames arrive
rs2::frameset frames = p.wait_for_frames();

// Try to get a frame of a depth image
rs2::depth_frame depth = frames.get_depth_frame();

// Get the depth frame's dimensions
float width = depth.get_width();
float height = depth.get_height();

// Query the distance from the camera to the object in the center of the image
float dist_to_center = depth.get_distance(width / 2, height / 2);

// Print the distance
std::cout << "The camera is facing an object " << dist_to_center << " meters away \r";

rs-imshow

该示例将打开一个 OpenCV UI 窗口，并向其呈现彩色深度流:

// Query frame size (width and height)
const int w = depth.as<rs2::video_frame>().get_width();
const int h = depth.as<rs2::video_frame>().get_height();// Create OpenCV matrix of size (w,h) from the colorized depth data
Mat image(Size(w, h), CV_8UC3, (void*)depth.get_data(), Mat::AUTO_STEP);

rs-capture

‎将深度和RGB数据流式传输和渲染到屏幕

‎

‎除高级功能外，所有功能都通过单个标头提供：

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API

#include "example.hpp"          // Include a short list of convenience functions for rendering

// Create a simple OpenGL window for rendering:
window app(1280, 720, "RealSense Capture Example");
// Declare two textures on the GPU, one for depth and one for color
texture depth_image, color_image;

// Declare depth colorizer for enhanced color visualization of depth data
rs2::colorizer color_map; 

// Declare the RealSense pipeline, encapsulating the actual device and sensors
rs2::pipeline pipe;
// Start streaming with the default recommended configuration
pipe.start(); 

rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camera

rs2::frame depth = color_map(data.get_depth_frame()); // Find and colorize the depth data
rs2::frame color = data.get_color_frame();            // Find the color data

// Render depth on to the first half of the screen and color on to the second
depth_image.render(depth, { 0,               0, app.width() / 2, app.height() });
color_image.render(color, { app.width() / 2, 0, app.width() / 2, app.height() });

rs-depth

GitHub 上的源代码 ( 代码太长了，我只能慢慢啃)

#include <librealsense2/rs.h>
#include <librealsense2/h/rs_pipeline.h>
#include <librealsense2/h/rs_option.h>
#include <librealsense2/h/rs_frame.h>
#include "example.h"#include <stdlib.h>
#include <stdint.h>
#include <stdio.h>//                                     These parameters are reconfigurable                                        //#define STREAM          RS2_STREAM_DEPTH  // rs2_stream is a types of data provided by RealSense device           //#define FORMAT          RS2_FORMAT_Z16    // rs2_format identifies how binary data is encoded within a frame      //#define WIDTH           640               // Defines the number of columns for each frame or zero for auto resolve//#define HEIGHT          0                 // Defines the number of lines for each frame or zero for auto resolve  //#define FPS             30                // Defines the rate of frames per second                                //#define STREAM_INDEX    0                 // Defines the stream index, used for multiple streams of the same type //#define HEIGHT_RATIO    20                // Defines the height ratio between the original frame to the new frame //#define WIDTH_RATIO     10                // Defines the width ratio between the original frame to the new frame  //// The number of meters represented by a single depth unit
float get_depth_unit_value(const rs2_device* const dev)
{rs2_error* e = 0;rs2_sensor_list* sensor_list = rs2_query_sensors(dev, &e);check_error(e);int num_of_sensors = rs2_get_sensors_count(sensor_list, &e);check_error(e);float depth_scale = 0;int is_depth_sensor_found = 0;int i;for (i = 0; i < num_of_sensors; ++i){rs2_sensor* sensor = rs2_create_sensor(sensor_list, i, &e);check_error(e);// Check if the given sensor can be extended to depth sensor interfaceis_depth_sensor_found = rs2_is_sensor_extendable_to(sensor, RS2_EXTENSION_DEPTH_SENSOR, &e);check_error(e);if (1 == is_depth_sensor_found){depth_scale = rs2_get_option((const rs2_options*)sensor, RS2_OPTION_DEPTH_UNITS, &e);check_error(e);rs2_delete_sensor(sensor);break;}rs2_delete_sensor(sensor);}rs2_delete_sensor_list(sensor_list);if (0 == is_depth_sensor_found){printf("Depth sensor not found!\n");exit(EXIT_FAILURE);}return depth_scale;
}int main()
{rs2_error* e = 0;// Create a context object. This object owns the handles to all connected realsense devices.// The returned object should be released with rs2_delete_context(...)rs2_context* ctx = rs2_create_context(RS2_API_VERSION, &e);check_error(e);/* Get a list of all the connected devices. */// The returned object should be released with rs2_delete_device_list(...)rs2_device_list* device_list = rs2_query_devices(ctx, &e);check_error(e);int dev_count = rs2_get_device_count(device_list, &e);check_error(e);printf("There are %d connected RealSense devices.\n", dev_count);if (0 == dev_count)return EXIT_FAILURE;// Get the first connected device// The returned object should be released with rs2_delete_device(...)rs2_device* dev = rs2_create_device(device_list, 0, &e);check_error(e);print_device_info(dev);/* Determine depth value corresponding to one meter */uint16_t one_meter = (uint16_t)(1.0f / get_depth_unit_value(dev));// Create a pipeline to configure, start and stop camera streaming// The returned object should be released with rs2_delete_pipeline(...)rs2_pipeline* pipeline =  rs2_create_pipeline(ctx, &e);check_error(e);// Create a config instance, used to specify hardware configuration// The retunred object should be released with rs2_delete_config(...)rs2_config* config = rs2_create_config(&e);check_error(e);// Request a specific configurationrs2_config_enable_stream(config, STREAM, STREAM_INDEX, WIDTH, HEIGHT, FORMAT, FPS, &e);check_error(e);// Start the pipeline streaming// The retunred object should be released with rs2_delete_pipeline_profile(...)rs2_pipeline_profile* pipeline_profile = rs2_pipeline_start_with_config(pipeline, config, &e);if (e){printf("The connected device doesn't support depth streaming!\n");exit(EXIT_FAILURE);}rs2_stream_profile_list* stream_profile_list = rs2_pipeline_profile_get_streams(pipeline_profile, &e);if (e){printf("Failed to create stream profile list!\n");exit(EXIT_FAILURE);}rs2_stream_profile* stream_profile = (rs2_stream_profile*)rs2_get_stream_profile(stream_profile_list, 0, &e);if (e){printf("Failed to create stream profile!\n");exit(EXIT_FAILURE);}rs2_stream stream; rs2_format format; int index; int unique_id; int framerate;rs2_get_stream_profile_data(stream_profile, &stream, &format, &index, &unique_id, &framerate, &e);if (e){printf("Failed to get stream profile data!\n");exit(EXIT_FAILURE);}int width; int height;rs2_get_video_stream_resolution(stream_profile, &width, &height, &e);if (e){printf("Failed to get video stream resolution data!\n");exit(EXIT_FAILURE);}int rows = height / HEIGHT_RATIO;int row_length = width / WIDTH_RATIO;int display_size = (rows + 1) * (row_length + 1);int buffer_size = display_size * sizeof(char);char* buffer = calloc(display_size, sizeof(char));char* out = NULL;while (1){// This call waits until a new composite_frame is available// composite_frame holds a set of frames. It is used to prevent frame drops// The returned object should be released with rs2_release_frame(...)rs2_frame* frames = rs2_pipeline_wait_for_frames(pipeline, RS2_DEFAULT_TIMEOUT, &e);check_error(e);// Returns the number of frames embedded within the composite frameint num_of_frames = rs2_embedded_frames_count(frames, &e);check_error(e);int i;for (i = 0; i < num_of_frames; ++i){// The retunred object should be released with rs2_release_frame(...)rs2_frame* frame = rs2_extract_frame(frames, i, &e);check_error(e);// Check if the given frame can be extended to depth frame interface// Accept only depth frames and skip other framesif (0 == rs2_is_frame_extendable_to(frame, RS2_EXTENSION_DEPTH_FRAME, &e)){rs2_release_frame(frame);continue;}/* Retrieve depth data, configured as 16-bit depth values */const uint16_t* depth_frame_data = (const uint16_t*)(rs2_get_frame_data(frame, &e));check_error(e);/* Print a simple text-based representation of the image, by breaking it into 10x5 pixel regions and approximating the coverage of pixels within one meter */out = buffer;int x, y, i;int* coverage = calloc(row_length, sizeof(int));for (y = 0; y < height; ++y){for (x = 0; x < width; ++x){// Create a depth histogram to each rowint coverage_index = x / WIDTH_RATIO;int depth = *depth_frame_data++;if (depth > 0 && depth < one_meter)++coverage[coverage_index];}if ((y % HEIGHT_RATIO) == (HEIGHT_RATIO-1)){for (i = 0; i < (row_length); ++i){static const char* pixels = " .:nhBXWW";int pixel_index = (coverage[i] / (HEIGHT_RATIO * WIDTH_RATIO / sizeof(pixels)));*out++ = pixels[pixel_index];coverage[i] = 0;}*out++ = '\n';}}*out++ = 0;printf("\n%s", buffer);free(coverage);rs2_release_frame(frame);}rs2_release_frame(frames);}// Stop the pipeline streamingrs2_pipeline_stop(pipeline, &e);check_error(e);// Release resourcesfree(buffer);rs2_delete_pipeline_profile(pipeline_profile);rs2_delete_stream_profiles_list(stream_profile_list);rs2_delete_stream_profile(stream_profile);rs2_delete_config(config);rs2_delete_pipeline(pipeline);rs2_delete_device(dev);rs2_delete_device_list(device_list);rs2_delete_context(ctx);return EXIT_SUCCESS;
}

rs-pointcloud

‎生成和可视化纹理 3D 点云‎

‎应用程序应打开一个带有点云的窗口。使用鼠标，您应该能够与点云交互、旋转、缩放和平移。‎

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API

#include "example.hpp"          // Include short list of convenience functions for rendering

// Struct for managing rotation of pointcloud view
struct state { double yaw, pitch, last_x, last_y; bool ml; float offset_x, offset_y; texture tex; };// Helper functions
void register_glfw_callbacks(window& app, state& app_state);
void draw_pointcloud(window& app, state& app_state, rs2::points& points);

// Create a simple OpenGL window for rendering:
window app(1280, 720, "RealSense Pointcloud Example");
// Construct an object to manage view state
state app_state = { 0, 0, 0, 0, false, 0, 0, 0 };
// register callbacks to allow manipulation of the pointcloud
register_glfw_callbacks(app, app_state);

using namespace rs2;

// Declare pointcloud object, for calculating pointclouds and texture mappings
pointcloud pc = rs2::context().create_pointcloud();
// We want the points object to be persistent so we can display the last cloud when a frame drops
rs2::points points;

// Declare RealSense pipeline, encapsulating the actual device and sensors
pipeline pipe;
// Start streaming with default recommended configuration
pipe.start();

auto data = pipe.wait_for_frames(); // Wait for next set of frames from the camera

// Wait for the next set of frames from the camera
auto frames = pipe.wait_for_frames();
auto depth = frames.get_depth_frame();// Generate the pointcloud and texture mappings
points = pc.calculate(depth);auto color = frames.get_color_frame();// Tell pointcloud object to map to this color frame
pc.map_to(color);// Upload the color frame to OpenGL
app_state.tex.upload(color);

draw_pointcloud(app, app_state, points);

/* this segment actually prints the pointcloud */
auto vertices = points.get_vertices();              // get vertices
auto tex_coords = points.get_texture_coordinates(); // and texture coordinates
for (int i = 0; i < points.size(); i++)
{if (vertices[i].z){// upload the point and texture coordinates only for points we have depth data forglVertex3fv(vertices[i]);glTexCoord2fv(tex_coords[i]);}
}

OpenGL.cpp

#include <iostream>#ifdef _WIN32#define WIN32_LEAN_AND_MEAN 1#define NOMINMAX 1#include <windows.h>
#endif#if defined(__APPLE__)#include <OpenGL/gl.h>#include <OpenGL/glu.h>
#else#include <GL/gl.h>#include <GL/glu.h>
#endif#include "opencv2/core.hpp"
#include "opencv2/core/opengl.hpp"
#include "opencv2/core/cuda.hpp"
#include "opencv2/highgui.hpp"using namespace std;
using namespace cv;
using namespace cv::cuda;const int win_width = 800;
const int win_height = 640;struct DrawData
{ogl::Arrays arr;ogl::Texture2D tex;ogl::Buffer indices;
};void draw(void* userdata);void draw(void* userdata)
{DrawData* data = static_cast<DrawData*>(userdata);glRotated(0.6, 0, 1, 0);ogl::render(data->arr, data->indices, ogl::TRIANGLES);
}int main(int argc, char* argv[])
{string filename;if (argc < 2){cout << "Usage: " << argv[0] << " image" << endl;filename = "../data/lena.jpg";}elsefilename = argv[1];Mat img = imread(filename);if (img.empty()){cerr << "Can't open image " << filename << endl;return -1;}namedWindow("OpenGL", WINDOW_OPENGL);resizeWindow("OpenGL", win_width, win_height);Mat_<Vec2f> vertex(1, 4);vertex << Vec2f(-1, 1), Vec2f(-1, -1), Vec2f(1, -1), Vec2f(1, 1);Mat_<Vec2f> texCoords(1, 4);texCoords << Vec2f(0, 0), Vec2f(0, 1), Vec2f(1, 1), Vec2f(1, 0);Mat_<int> indices(1, 6);indices << 0, 1, 2, 2, 3, 0;DrawData data;data.arr.setVertexArray(vertex);data.arr.setTexCoordArray(texCoords);data.indices.copyFrom(indices);data.tex.copyFrom(img);glMatrixMode(GL_PROJECTION);glLoadIdentity();gluPerspective(45.0, (double)win_width / win_height, 0.1, 100.0);glMatrixMode(GL_MODELVIEW);glLoadIdentity();gluLookAt(0, 0, 3, 0, 0, 0, 0, 1, 0);glEnable(GL_TEXTURE_2D);data.tex.bind();glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);glTexEnvi(GL_TEXTURE_2D, GL_TEXTURE_ENV_MODE, GL_REPLACE);glDisable(GL_CULL_FACE);setOpenGlDrawCallback("OpenGL", draw, &data);for (;;){updateWindow("OpenGL");char key = (char)waitKey(40);if (key == 27)break;}setOpenGlDrawCallback("OpenGL", 0, 0);destroyAllWindows();return 0;
}

Realsense SDK 2.0+OpenCV

用于可视化深度数据的最小 OpenCV 应用程序

下面的示例将打开一个 OpenCV UI 窗口，并向其呈现彩色深度流。The following code snippet is used to create 

cv::Mat

 from

 rs2::frame

// Query frame size (width and height)
const int w = depth.as<rs2::video_frame>().get_width();
const int h = depth.as<rs2::video_frame>().get_height();// Create OpenCV matrix of size (w,h) from the colorized depth data
Mat image(Size(w, h), CV_8UC3, (void*)depth.get_data(), Mat::AUTO_STEP);

GitHub上面的源代码

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <opencv2/opencv.hpp>   // Include OpenCV APIint main(int argc, char * argv[]) try
{// Declare depth colorizer for pretty visualization of depth datars2::colorizer color_map;// Declare RealSense pipeline, encapsulating the actual device and sensorsrs2::pipeline pipe;// Start streaming with default recommended configurationpipe.start();using namespace cv;const auto window_name = "Display Image";namedWindow(window_name, WINDOW_AUTOSIZE);while (waitKey(1) < 0 && getWindowProperty(window_name, WND_PROP_AUTOSIZE) >= 0){rs2::frameset data = pipe.wait_for_frames(); // Wait for next set of frames from the camerars2::frame depth = data.get_depth_frame().apply_filter(color_map);// Query frame size (width and height)const int w = depth.as<rs2::video_frame>().get_width();const int h = depth.as<rs2::video_frame>().get_height();// Create OpenCV matrix of size (w,h) from the colorized depth dataMat image(Size(w, h), CV_8UC3, (void*)depth.get_data(), Mat::AUTO_STEP);// Update the window with new dataimshow(window_name, image);}return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;return EXIT_FAILURE;
}
catch (const std::exception& e)
{std::cerr << e.what() << std::endl;return EXIT_FAILURE;
}

使用GrabCut算法进行简单的背景删除

GrabCut算法

‎GrabCuts ‎‎示例演示了如何使用 3D 数据增强现有的 2D 算法：‎‎GrabCut 算法‎‎通常用于交互式、用户辅助的前景提取。

该算法详细讲解的链接：(129条消息) 图像分割之（三）从Graph Cut到Grab Cut_zouxy09的专栏-CSDN博客_graphcut图像分割

简单来说，GrabCut算法就是利用了图像中的纹理（颜色）信息和边界（反差）信息，只要少量的用户交互操作即可得到比较好的分割结果.

将用户输入替换为基于深度数据的初始猜测

获得对齐的颜色和深度

首先获取一对空间和时间同步的帧：‎

frameset data = pipe.wait_for_frames();
// Make sure the frameset is spatialy aligned
// (each pixel in depth image corresponds to the same pixel in the color image)
frameset aligned_set = align_to.process(data);
frame depth = aligned_set.get_depth_frame();
auto color_mat = frame_to_mat(aligned_set.get_color_frame());

‎生成近/远掩码

继续生成像素区域，以估计近处和远处的物体。我们使用基本的‎‎形态变换‎‎来提高两个掩模的质量：

// Generate "near" mask image:
auto near = frame_to_mat(bw_depth);
cvtColor(near, near, CV_BGR2GRAY);
// Take just values within range [180-255]
// These will roughly correspond to near objects due to histogram equalization
create_mask_from_depth(near, 180, THRESH_BINARY);// Generate "far" mask image:
auto far = frame_to_mat(bw_depth);
cvtColor(far, far, CV_BGR2GRAY);
// Note: 0 value does not indicate pixel near the camera, and requires special attention:
far.setTo(255, far == 0);
create_mask_from_depth(far, 100, THRESH_BINARY_INV);

调用 cv：：GrabCut 算法‎

‎这两个掩码组合成一个猜测：‎

// GrabCut algorithm needs a mask with every pixel marked as either:
// BGD, FGB, PR_BGD, PR_FGB
Mat mask;
mask.create(near.size(), CV_8UC1);
mask.setTo(Scalar::all(GC_BGD)); // Set "background" as default guess
mask.setTo(GC_PR_BGD, far == 0); // Relax this to "probably background" for pixels outside "far" region
mask.setTo(GC_FGD, near == 255); // Set pixels within the "near" region to "foreground"

运行算法：

Mat bgModel, fgModel;
cv::grabCut(color_mat, mask, Rect(), bgModel, fgModel, 1, cv::GC_INIT_WITH_MASK);

‎并生成生成的图像：

// Extract foreground pixels based on refined mask from the algorithm
cv::Mat3b foreground = cv::Mat3b::zeros(color_mat.rows, color_mat.cols);
color_mat.copyTo(foreground, (mask == cv::GC_FGD) | (mask == cv::GC_PR_FGD));
cv::imshow(window_name, foreground);

狗图：

GitHub上面的源代码

#include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
#include <opencv2/opencv.hpp>   // Include OpenCV API
#include "../cv-helpers.hpp"    // Helper functions for conversions between RealSense and OpenCVint main(int argc, char * argv[]) try
{using namespace cv;using namespace rs2;// Define colorizer and align processing-blockscolorizer colorize;align align_to(RS2_STREAM_COLOR);// Start the camerapipeline pipe;pipe.start();const auto window_name = "Display Image";namedWindow(window_name, WINDOW_AUTOSIZE);// We are using StructuringElement for erode / dilate operationsauto gen_element = [](int erosion_size){return getStructuringElement(MORPH_RECT,Size(erosion_size + 1, erosion_size + 1),Point(erosion_size, erosion_size));};const int erosion_size = 3;auto erode_less = gen_element(erosion_size);auto erode_more = gen_element(erosion_size * 2);// The following operation is taking grayscale image,// performs threashold on it, closes small holes and erodes the white areaauto create_mask_from_depth = [&](Mat& depth, int thresh, ThresholdTypes type){threshold(depth, depth, thresh, 255, type);dilate(depth, depth, erode_less);erode(depth, depth, erode_more);};// Skips some frames to allow for auto-exposure stabilizationfor (int i = 0; i < 10; i++) pipe.wait_for_frames();while (waitKey(1) < 0 && getWindowProperty(window_name, WND_PROP_AUTOSIZE) >= 0){frameset data = pipe.wait_for_frames();// Make sure the frameset is spatialy aligned // (each pixel in depth image corresponds to the same pixel in the color image)frameset aligned_set = align_to.process(data);frame depth = aligned_set.get_depth_frame();auto color_mat = frame_to_mat(aligned_set.get_color_frame());// Colorize depth image with white being near and black being far// This will take advantage of histogram equalization done by the colorizercolorize.set_option(RS2_OPTION_COLOR_SCHEME, 2);frame bw_depth = depth.apply_filter(colorize);// Generate "near" mask image:auto near = frame_to_mat(bw_depth);cvtColor(near, near, COLOR_BGR2GRAY);// Take just values within range [180-255]// These will roughly correspond to near objects due to histogram equalizationcreate_mask_from_depth(near, 180, THRESH_BINARY);// Generate "far" mask image:auto far = frame_to_mat(bw_depth);cvtColor(far, far, COLOR_BGR2GRAY);far.setTo(255, far == 0); // Note: 0 value does not indicate pixel near the camera, and requires special attention create_mask_from_depth(far, 100, THRESH_BINARY_INV);// GrabCut algorithm needs a mask with every pixel marked as either:// BGD, FGB, PR_BGD, PR_FGBMat mask;mask.create(near.size(), CV_8UC1); mask.setTo(Scalar::all(GC_BGD)); // Set "background" as default guessmask.setTo(GC_PR_BGD, far == 0); // Relax this to "probably background" for pixels outside "far" regionmask.setTo(GC_FGD, near == 255); // Set pixels within the "near" region to "foreground"// Run Grab-Cut algorithm:Mat bgModel, fgModel; grabCut(color_mat, mask, Rect(), bgModel, fgModel, 1, GC_INIT_WITH_MASK);// Extract foreground pixels based on refined mask from the algorithmMat3b foreground = Mat3b::zeros(color_mat.rows, color_mat.cols);color_mat.copyTo(foreground, (mask == GC_FGD) | (mask == GC_PR_FGD));imshow(window_name, foreground);}return EXIT_SUCCESS;
}
catch (const rs2::error & e)
{std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n    " << e.what() << std::endl;return EXIT_FAILURE;
}
catch (const std::exception& e)
{std::cerr << e.what() << std::endl;return EXIT_FAILURE;
}

用于避免碰撞的深度过滤‎

问题陈述

避免碰撞的问题优先考虑具有可靠的深度而不是高填充率。‎
‎在基于立体的系统中，由于一些光学和算法效应，包括校正过程中的重复几何形状和摩尔纹图案，可能会发生不可靠的读数。‎
‎有几种最广为人知的方法可以删除此类无效的深度值：‎

1. ‎如果可能，增加红外投影仪将在图像中引入足够数量的噪声，并帮助算法正确解决有问题的情况。‎
2. ‎除1之外，使用光学滤光片阻挡可见光以仅省略投影仪图案将消除错误的近邻深度值。‎
3. ‎D400系列相机包括一组控制深度失效的片上参数。加载自定义"高置信度"预设将有助于ASIC丢弃不明确的像素。‎
4. ‎最后，可以应用软件后处理来仅保留高置信深度值。‎

高置信度预设

用于在流式传输之前将自定义预设加载到设备：

rs2::pipeline pipe;rs2::config cfg;
cfg.enable_stream(RS2_STREAM_DEPTH, 848, 480);
cfg.enable_stream(RS2_STREAM_INFRARED, 1);std::ifstream file("./camera-settings.json");
if (file.good())
{std::string str((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());auto prof = cfg.resolve(pipe);if (auto advanced = prof.get_device().as<rs400::advanced_mode>()){advanced.load_json(str);}
}

high_confidence_filter‎类

接下来，我们定义类。继承并实现 SDK 处理块模式使此算法可与其他 SDK 方法（如 和 ） 组合在一起。‎
‎特别是，将消耗同步深度和红外对，并输出新的同步对的缩减采样和滤波深度和红外帧。‎

high_confidence_filter
rs2::filter
rs2::pointcloud
rs2::alignhigh_confidence_filter

该算法背后的核心思想是，在红外图像中具有明确定义的特征的区域更有可能具有高置信度的相应深度。该算法在深度和红外图像上同时运行，并遮罩除边缘和角落之外的所有内容。‎

下采样步骤

缩减采样是任何深度处理算法中非常常见的第一步。关键的观察结果是，缩减采样会降低空间（X-Y）精度，但会保持Z精度( 下图中的绿轴，也就是竖直方向)。‎

它是如此普遍，以至于SDK以‎‎rs2：:d ecimation_filter‎‎的形式提供了内置的缩减采样方法。‎
‎请务必注意，使用标准 OpenCV 缩减像素采样对于深度图像并不理想。‎
‎在此示例中，我们展示了实现深度缩减采样的另一种正确方法。它在概念上类似于 为每个 4x4 块选取一个非零深度值，但与 它不同的是选取‎‎最接近‎‎的深度值而不是中值。这在避免碰撞的背景下是有道理的，因为我们希望保持与物体的最小距离。‎

rs2::decimation_filterrs2::decimation_filter

‎这种方法的朴素实现：‎

for (int y = 0; y < sizeYresized; y++)for (int x = 0; x < source.cols; x += DOWNSAMPLE_FACTOR){uint16_t min_value = MAX_DEPTH;// Loop over 4x4 quadfor (int i = 0; i < DOWNSAMPLE_FACTOR; i++)for (int j = 0; j < DOWNSAMPLE_FACTOR; j++){auto pixel = source.at<uint16_t>(y * DOWNSAMPLE_FACTOR + i, x + j);// Only include non-zero pixels in min calculationif (pixel) min_value = std::min(min_value, pixel);}// If no non-zero pixels were found, mark the output as zeroif (min_value == MAX_DEPTH) min_value = 0;pDest->at<uint16_t>(y, x / DOWNSAMPLE_FACTOR) = min_value;}

‎主过滤器‎

‎核心过滤器按以下操作顺序执行：

filter_edges(&sub_areas[i]); // Find edges in the infrared image
filter_harris(&sub_areas[i]); // Find corners in the infrared image
// Combine the two masks together:
cv::bitwise_or(sub_areas[i].edge_mask, sub_areas[i].harris_mask, sub_areas[i].combined_mask);// morphology: open(src, element) = dilate(erode(src,element))
cv::morphologyEx(sub_areas[i].combined_mask, sub_areas[i].combined_mask,cv::MORPH_OPEN, cv::getStructuringElement(cv::MORPH_ELLIPSE, cv::Size(3, 3)));
// Copy masked depth values:
sub_areas[i].decimated_depth.copyTo(sub_areas[i].output_depth, sub_areas[i].combined_mask);

‎所有 OpenCV 矩阵都拆分为多个部分 - sub_areas[i]。这样做是为了帮助并行化代码，这样每个执行线程都可以在单独的图像区域上运行。‎

边缘过滤是使用 ‎‎OpenCV Scharr 运算符‎‎完成的：‎

cv::Scharr(area->decimated_ir, area->scharr_x, CV_16S, 1, 0);
cv::convertScaleAbs(area->scharr_x, area->abs_scharr_x);
cv::Scharr(area->decimated_ir, area->scharr_y, CV_16S, 0, 1);
cv::convertScaleAbs(area->scharr_y, area->abs_scharr_y);
cv::addWeighted(area->abs_scharr_y, 0.5, area->abs_scharr_y, 0.5, 0, area->edge_mask);
cv::threshold(area->edge_mask, area->edge_mask, 192, 255, cv::THRESH_BINARY);

‎角落滤波是使用‎‎OpenCV Harris探测器‎‎完成的：

area->decimated_ir.convertTo(area->float_ir, CV_32F);// Harris 角点检测API
cv::cornerHarris(area->float_ir, area->corners, 2, 3, 0.04); cv::threshold(area->corners, area->corners, 300, 255, cv::THRESH_BINARY);
area->corners.convertTo(area->harris_mask, CV_8U);|

上面运用了 Harris 角点检测，相关教程：

[opencv_C++] 入门强推！！！【B站最全】_哔哩哔哩_bilibili

一幅图像分为：平坦区域 ，边缘 和角点 。

何为角点？

下面有两幅不同视角的图像，通过找出对应的角点进行匹配。

 再看下图所示，放大图像的两处角点区域：

直观的概括下角点所具有的特征：

>轮廓之间的交点；

>对于同一场景，即使视角发生变化，通常具备稳定性质的特征；

>该点附近区域的像素点无论在梯度方向上还是其梯度幅值上有着较大变化；

角点检测算法基本思想是什么？

算法基本思想是使用一个固定窗口在图像上进行任意方向上的滑动，比较滑动前与滑动后两种情况，窗口中的像素灰度变化程度，如果存在任意方向上的滑动，都有着较大灰度变化，那么我们可以认为该窗口中存在角点。

‎软件开发工具包集成‎

‎该方法负责将输入帧转换为对象，并将生成的对象转换为新对象。这个额外的层确保了算法和 SDK 之间的无缝互操作性。‎
‎算法输出以后可用于点云生成和导出、流对齐、彩色可视化，并与其他 SDK 后处理块结合使用。‎

sdk_handlecv::Matcv::Matrs2::frame

‎检测到新的输入帧类型后，将生成新的 SDK 视频流配置文件，其分辨率大幅下降，并更新了内部函数。

‎sdk_handle

if (!_output_ir_profile || _input_ir_profile.get() != ir_frame.get_profile().get())
{auto p = ir_frame.get_profile().as<rs2::video_stream_profile>();auto intr = p.get_intrinsics() / DOWNSAMPLE_FACTOR;_input_ir_profile = p;_output_ir_profile = p.clone(p.stream_type(), p.stream_index(), p.format(),p.width() / DOWNSAMPLE_FACTOR, p.height() / DOWNSAMPLE_FACTOR, intr);
}

‎输出图像准备就绪后，将其复制到新的：

rs2::frame

auto res_ir = src.allocate_video_frame(_output_ir_profile, ir_frame, 0,newW, newH, ir_frame.get_bytes_per_pixel() * newW);
memcpy((void*)res_ir.get_data(), _decimated_ir.data, newW * newH);

最后，两个生成的帧（深度和红外）一起输出为：

‎rs2::frameset

std::vector<rs2::frame> fs{ res_ir, res_depth };
auto cmp = src.allocate_composite_frame(fs);
src.frame_ready(cmp);

‎一旦包装为一种算法，就可以像任何其他 SDK 处理块一样应用该算法：

‎rs2::filter

rs2::frameset data = pipe.wait_for_frames();
data = data.apply_filter(filter);

关于 Intel Realsense 深度图像处理.1(C++)相关推荐

1. intel realsense 深度相机深度图像处理API总结

   文章目录 intel realsense 深度相机深度图像处理API总结 一.导入库文件 二.开启数据流传输 三.获取图像帧数据 四.使用OpenCV显示深度图像和高度图像 五.使用过滤器对图像数据进 ...

2. 遨博协作机器人ROS开发 - Intel RealSense深度相机ROS驱动

   目录 一.简介 二.环境版本 三.学习目标 四.知识储备 五.任务实施 六.任务拓展 七.课堂小结 八.课后练习 一.简介 大家好,欢迎关注遨博学院带来的系列技术分享文章(协作机器人ROS开发),在前 ...

3. Intel Realsense 深度流向彩色流对齐 color_aligned_to_depth和depth_aligned_to_color

   为啥要对齐 由于RGB图像数据与深度图像数据的空间坐标系是不同的,前者的原点是RGB摄像头,后者的原点是红外摄像头,因此两者会有相应的误差. 对齐原理 深度图上的2D点转换到世界坐标的3D点,世界坐标 ...

4. 在旭日X3派开发板上使用Intel Realsense深度相机

   在前文<多方位玩转"地平线新发布AIoT开发板--旭日X3派(Sunrise x3 Pi)" 插电!开机!轻松秒杀!>中,我们已经成功了使用了这个开发板. 深度相机目前 ...

5. intel realsense 深度视野范围与模组硬件说明

   文章目录 4.4 距离(Z)处的深度视场 4.5 无效深度带 4.6 最小测量深度 4.7 深度质量规范 4.8 深度起点(地面零点参考) 4.8.1 深度原点X-Y坐标 7.2 视觉处理器D4板外形 ...

6. Intel Realsense深度摄像头的自校准功能（self-calibration-for-depth-cameras）

   https://www.intelrealsense.com/self-calibration-for-depth-cameras/

7. intel realsense D400系列相机介绍（一）

   文章目录 intel realsense D4 2 intel realsense D400介绍 2.3 立体视觉深度技术概述 2.4 摄像机系统框图 2.5 英特尔Intel® RealSense™ ...

8. intel realsense d435深度摄像头使用

   intel realsense d435深度摄像头基本环境配置 基本参考官网给的教程来就ok的 ,这里还是发一下我最初配置d435的记录 官网链接: https://github.com/intel- ...

9. 为什么Intel Realsense D435深度摄像头在基于深度的水平方向障碍物检测（避障）方案中，摄像头不宜安装太高？

   补一句,为啥R与b对比就要比R与a对比更容易区分? 因为a与b的右端和障碍物R属同一深度,但b左端明显比a左端深度大,总体比a拥有差异性更大的深度范围,因此b与a相比,对R拥有差异性更大的深度范围, ...

最新文章

1. 百度副总裁马杰：实现元宇宙，技术要过三道坎
2. SharpSvn使用说明
3. ie 不执行回调函时_javascript引擎执行的过程的理解--执行阶段
4. boost::contract模块实现volatile的测试程序
5. 音视频技术开发周刊 | 162
6. Pytorch(4)-模型保存-载入-eval()
7. kafka 脚本发送_Kafka笔记归纳（第五部分:一致性保证，消息重复消费场景及解决方式）...
8. java getdelay_java中DelayQueue的一个使用陷阱分析
9. 拼多多上架专供湖北平价口罩 每天300万只
10. stm32产生100k时钟信号_stm32f105/107系统时钟变慢
11. ncurses窗口机制：wprintw(), wrefresh()
12. 简单分析FFT坐标轴的生成方式和幅值大小
13. java中equals，hashcode和==的区别
14. Java中多态的实例
15. 数字图像处理 冈萨雷斯 (内含算法链接)
16. 如何让企业微信的消息在微信里显示？
17. 软件测试分类、分级与软件缺陷管理
18. 厦门大学计算机保研学校,厦门大学计算机科学系(专业学位)计算机技术保研夏令营...
19. win10计算机磁盘图标,电脑win10系统磁盘分区ioc图标修改技巧
20. 春节快过腻了？不妨关心下太空探索

热门文章

1. 树莓派更换源后update/upgrade更新后无法启动
2. 中国科学技术大学2021计算机考研分数线,中国科学技术大学2021考研分数线已公布...
3. 姿态解算基础知识（二）-旋转矢量坐标变换的四元数描述的验证
4. 2020-2021 ICPC 银川站【ACMer退役纪念】
5. 12306订票助手java_GitHub - lifes/12306-hunter: 开源免费Java Swing C/S版本12306订票助手
6. 基于Java+springmvc+mysql+jquery实现企业员工管理系统
7. JS中常用正则转换和信息验证的封装（80个）
8. 华大单片机HC32L136定时器时间如何计算
9. 单向链表在O(1)时间内删除一个节点
10. c语言数据结构对学生信息折半查找,数据结构实训报告-二分查找学生管理实训报告.doc...