CN105339981B - Method for using one group of primitive registration data - Google Patents

Abstract

A method of carrying out registration data using one group of primitive for including point and plane.Firstly, this method selects first group of primitive from the data in the first coordinate system, wherein first group of primitive includes at least three primitives and at least one plane.It predicts from the first coordinate system to the transformation of the second coordinate system.Using the transformation by first group of basis element change to the second coordinate system.Second group of primitive is determined according to first group of primitive for transforming to the second coordinate system.Then, using first group of primitive in the first coordinate system and second group of primitive in the second coordinate system, by the second coordinate system and the first co-registration of coordinate systems used.Registration can be used for tracking the posture for obtaining the camera of data.

Description

Methods for registering data using a set of primitives

technical field

The present invention relates generally to computer vision and, more particularly, to estimating the pose of a camera.

Background technique

Systems and methods for tracking the pose of a camera while reconstructing the 3D structure of a scene are widely used in augmented reality (AR) visualization, robotic navigation, scene modeling, and computer vision applications. Such processing is often referred to as simultaneous localization and mapping (SLAM). A real-time SLAM system can use conventional cameras that acquire two-dimensional (2D) images, depth cameras that acquire three-dimensional (3D) point clouds (a set of 3D points), or red, green, blue, and Depth (RGB-D) cameras (such as, ). Tracking refers to the process of using the predicted motion of the camera for sequentially estimating the pose of the camera, while relocalization refers to the process of using some feature-based global registration for recovery from tracking failures.

SLAM systems using 2D cameras are usually successful for textured scenes, but are likely to fail for textureless regions. With the help of iterative closest point (ICP) methods, systems using depth cameras rely on geometrical changes in the scene (such as surfaces and depth boundaries). However, ICP-based systems often fail when geometry changes are small (such as in planar scenes). Systems using RGB-D cameras can utilize both texture and geometric features, but they still require unique textures.

Many approaches do not explicitly address the difficulty of reconstructing 3D models larger than a single-occupancy room. In order to extend those methods to larger scenarios, better memory management techniques are required. However, memory constraints are not the only challenge. Often, room-scale scenes include many objects with textured and geometric features. To scale to larger scenes, camera poses need to be tracked in areas with limited texture and insufficient geometric variation, such as corridors.

camera tracking

Given some 3D correspondence, systems that use 3D sensors to acquire 3D point clouds generalize the tracking problem into a registration problem. The ICP method iteratively locates point-to-point or point-to-plane correspondences, starting from an initial pose estimate given by camera motion prediction. ICP has been widely used for line scan 3D sensors (also known as scan matching) in mobile robots as well as for depth cameras and 3D sensors for generating full 3D point clouds. US20120194516 applies point-to-plane correspondence to the ICP method Camera pose tracking. This representation of the map is a set of voxels. Each voxel represents a truncated signed distance function for the distance to the closest surface point. This method does not extract planes from 3D point clouds; instead, point-to-plane correspondences are established by using local neighborhoods to determine the normals of 3D points. Such ICP-based methods require scenes with sufficient geometric variation for accurate registration.

Another approach extracts features from RGB images and performs descriptor-based point matching to determine point-to-point correspondences and estimate camera poses, which are then refined using ICP methods. The method uses texture (RGB) and geometric (depth) features in the scene. However, using only point features for textureless regions and regions with repetitive textures is still problematic.

Using flat SLAM

Planar features have been used in many SLAM systems. To determine the camera pose, its normals are required to span at least three planes of ^R3 . Therefore, especially when the field of view (FOV) or the range of the sensor is small (such as in ), using only planes leads to many degeneracy issues. The combination of a large FOV line scan 3D sensor and a small field of view (FOV) depth camera can avoid degradation with additional system cost.

The method described in the related application uses point-plane SLAM, which uses both points and planes to avoid failure modes common in methods using one of these primitives. The system does not use any camera motion prediction. Instead, the system performs relocation for all frames by locating point and plane correspondences globally. As a result, the system was only able to process about three frames per second and encountered failures with some duplicate textures due to descriptor-based point matching.

The methods described in the related application also propose the use of both point-to-point and plane-to-plane correspondences to register 3D data in different coordinate systems.

SUMMARY OF THE INVENTION

In indoor and outdoor scenes including man-made structures, planes are predominant. Embodiments of the present invention provide a system and method for tracking an RGB-D camera using points and planes as primitive features. By fitting a plane, the method implicitly deals with noise in depth data specific to 3D sensors. The tracking method is supported by relocalization and bundle adjustment processing to demonstrate a real-time simultaneous localization and mapping (SLAM) system using a handheld or robot-mounted RGB-D camera.

It is an object of the present invention to enable fast and accurate registration while minimizing degradation problems that lead to registration failures. The method uses camera motion to predict anchor and plane correspondences, and provides a tracker based on a prediction and correction framework. The method uses a combination of point and plane relocalization and bundle adjustment processing to recover from tracking failures and continue to refine the camera pose estimate.

Specifically, one method uses a set of primitives including points and planes to register data. First, the method selects a first set of primitives from data in a first coordinate system, wherein the first set of primitives includes at least three primitives and at least one plane.

The transformation from the first coordinate system to the second coordinate system is predicted. Use this transformation to transform the first set of primitives to the second coordinate system. The second set of primitives is determined from the first set of primitives transformed to the second coordinate system.

Then, the second coordinate system is registered with the first coordinate system using the first set of primitives in the first coordinate system and the second set of primitives in the second coordinate system. The registration can be used to track the pose of the camera that acquired the data.

Description of drawings

1 is a flowchart of a method for tracking a pose of a camera according to an embodiment of the present invention; and

2 is a schematic diagram of a process for establishing point-to-point and plane-to-plane correspondences between a current frame and a map using a camera's predicted pose, according to an embodiment of the present invention.

Detailed ways

Embodiments of the present invention provide a system and method for tracking the pose of a camera. The present invention extends the embodiment described in our related US application Sn. 13/539,060 by using camera motion prediction for faster correspondence search and registration. We use the point-to-point and plane-to-plane correspondence established between the current frame and the map. The map includes points and planes from frames previously registered in the global coordinate system. Here, we focus on using camera motion prediction to establish plane-to-plane correspondence and a mixture of both point-to-point and plane-to-plane correspondences.

System Overview

In a preferred system, the RGB-D camera 102 is or Xtion PRO LIVE, which acquires a series of frames 101 . We use a keyframe-based SLAM system, where we select multiple representative frames as keyframes and store the keyframes registered in a single global coordinate system in the map. Compared to the state-of-the-art SLAM that only uses points, we use both points and planes as primitives in all processing in the system. Points and planes in each frame are called measurements, and measurements from keyframes are stored in the map as landmarks.

Given a map, we use a prediction and correction framework to estimate the pose of the current frame: we predict the pose of the camera, and use this pose to determine the correspondence between point and plane metrics and point and plane landmarks, then use the to determine the camera pose.

Tracking may fail due to incorrect or insufficient correspondence. After a predetermined number of consecutive track failures, we relocalize, where we use a global point and plane correspondence search between the current frame and the map. We also apply bundle adjustment using points and planes to asynchronously refine landmarks in the map.

Method overview

The current frame 101 is acquired 110 by the red, green, blue and depth (RGB-D) camera 102 of the scene 103 as described in FIG. 1 . The pose of the camera when the frame was acquired is predicted, and the pose of the camera is used to locate 130 the point and plane correspondence between the frame and the map 194 . Point and plane correspondences are used in a random sampling agreement (RANSAC) framework 140 to register frames to a map. If 150 registration failed, count the number of consecutive failures and if 154 is false (F) continue to the next frame, else if 154 is true (T) use global registration method without camera motion Prediction to relocate 158 cameras.

If the RANSAC registration is successful, the pose 160 estimated in the RANSAC framework is used as the pose of the frame. Next, it is determined 170 whether the current frame is a key frame, and if false, at step 110, proceed to the next frame. Otherwise, the additional points and planes are extracted 180 in the current frame, the map 194 is updated 190, and the process continues for the next frame. Asynchronous refinement of the map using bundle adjustment 198.

These steps may be performed in a processor connected to memory and input/output interfaces known in the art.

camera pose tracking

As mentioned above, our tracking uses features that include both points and planes. The tracking is based on a prediction and correction scheme, which can be summarized as follows. For each frame, we use the camera motion model to predict the pose. Based on the predicted pose, we locate point and plane metrics in the frame that correspond to point and plane landmarks in the map. We use point and plane correspondences to perform RANSAC-based registration. If the pose is different from the pose of any keyframe currently stored in the map, we extract the additional point and plane metrics, and add the frame to the map as a new keyframe.

camera motion prediction

We denote the pose of the kth frame as

Among them, R _k and t _k represent the rotation matrix and translation vector, respectively. We use the _first frame to define the map's coordinate system; thus, T1 is the identity matrix, and Tk represents the pose of the _kth frame relative to the map.

We predict the pose of the kth frame by using the constant velocity assumption Let ΔT denote the previously estimated motion between the (k-1)th frame and the (k-2)th frame, ie, ΔT=T _k-1 T _k-2 ^-1 . Then, we predict the pose of the kth frame as

Anchor point and plane correspondence

As shown in Figure 2, we use the predicted pose to locate the point and plane measure in the kth frame corresponding to the landmark in the map. Considering the predicted pose 201 of the current frame, we locate the correspondence between the point and plane landmarks in the map 202 and the point and plane metrics in the current frame 203 . We first transform landmarks in the map to the current frame using the predicted pose. Then, for each point, we perform a local search using an optical flow process starting from the predicted pixel location in the current frame. For each plane, we first locate the parameters of the predicted plane. Then, we consider a set of fiducials on the predicted plane and locate the pixel connected to each fiducial lying on the predicted plane. The fiducial point with the largest number of connected pixels is chosen, and all connected pixels are used to refine the plane parameters.

Point correspondence: Let p _i =( _xi ,y _i ,z _i ,1) ^T denote the ith point landmark 210 in the map, which is represented as a homogeneous vector. The 2D image projection 220 of _pi in the current frame is predicted as

in, is the 3D point transformed to the coordinate system of the kth frame, and the function FP(·) uses the internal camera calibration parameters to determine the forward projection of the 3D point onto the image plane. We use Lucas-Kanade's optical flow method from The initial position of , starts to locate the corresponding point measurement. Make is the determined optical flow vector 230 . Then, the corresponding point measure for

where the function BP(·) backprojects 2D image pixels to 3D rays, and D(·) refers to the depth value of the pixel. If the optical flow vector is not determined or the pixel position With an invalid depth value, the feature is considered missing.

Plane correspondence: Instead of performing a time-consuming plane extraction process (prior art) for each frame independently of other frames, we utilize the predicted pose to extract the plane. This results in faster plane metric extraction and also provides plane correspondence.

Let π _j =(a _j ,b _j ,c _j ,d _j ) ^T denote the plane equation for the jth plane landmark 240 in the map. We assume that planar landmarks and corresponding metrics have some overlapping areas in the image. To locate such corresponding plane metrics, we randomly select a number of fiducial points 250q _j,r (r=1,...,N) from the inliers of the jth plane landmark, and transform the fiducial points to The kth frame as 255

We also transform πj to the kth frame as 245

we start from the plane datum point after each transition on Connected pixels 260 are located, and the pixel with the largest inlier is selected. These interior points are used to refine the plane equation, resulting in the corresponding plane measure If the number of inliers is less than a threshold, the planar landmark is declared missing. For example, we use N=5 fiducial points, use a threshold of 50mm for point-to-plane distance to determine inliers on the plane, and use 9000 as the threshold for the minimum number of inliers.

Landmark selection

It may be inefficient to perform the above processing with all landmarks in the map. Therefore, we use the landmark that occurs in the single keyframe closest to the current frame. Before the tracking process, the closest keyframe is selected by using the pose Tk _-1 of the previous frame.

RANSAC registration

The prediction-based correspondence search provides candidates for point-to-point and plane-to-plane correspondences, which may include outliers. Therefore, we perform RANSAC-based registration to determine the inliers and determine the camera pose. To unambiguously determine the pose, we need at least three correspondences. Therefore, if there are less than three corresponding candidates, we immediately determine that the tracking has failed. For accurate camera tracking, we also determine that tracking fails when there are only a few corresponding candidates.

If there is a sufficient number of candidates, we use closed-form mixed correspondence to solve the registration problem. Because the number of planes is usually much less than the number of points, and the planes are less noisy due to support from many points, this process prioritizes plane correspondences over point correspondences. Tracking is considered successful if RANSAC locates a sufficient number of interior points (eg, 40% of the number of all point and plane metrics). This method produces the corrected pose Tk for the _kth frame.

map update

If the estimated pose Tk is sufficiently different from the pose of any existing keyframes in the map, we determine the _kth frame as a keyframe. To check this, we can eg use a threshold of 100mm translation and 5° rotation. For the new keyframe, point and plane metrics located as inliers in the RANSAC-based registration are associated with corresponding landmarks, while those located as outliers are discarded. We can then extract the additional point and plane metrics that were most recent in that frame. Additional point metrics are extracted using keypoint detectors such as Scale-Invariant Feature Transform (SIFT) and Accelerated Robust Features (SURF) on pixels that are not close to any existing point metrics. Additional plane metrics are extracted by using a RANSAC-based plane fit on pixels that are not inliers of any existing plane metrics. Add additional points and plane measures to the map as new placemarks. Additionally, we extract feature descriptors (such as SIFT and SURF) for relocalization for all point metrics in the frame.

Claims (14)

1. A method for registering data using a set of primitives, wherein the data has three dimensions (3D), and both points and planes are used as the primitives, the method comprising the steps of: selecting a first set of primitives from the data in a first coordinate system, wherein the first set of primitives includes at least three primitives, at least one of which is a plane; predicting a transformation from the first coordinate system to the second coordinate system, wherein the transformation is predicted using a camera motion model; transforming the first set of primitives to the second coordinate system using the predicted transformation; determining a second set of primitives from the first set of primitives transformed to the second coordinate system; and Using the first set of primitives in the first coordinate system and the second set of primitives in the second coordinate system that correspond to each other, the second coordinate system is aligned with the first coordinate system Registration, wherein the registration is used for simultaneous localization and mapping (SLAM), and the above steps are performed in a processor. 2. The method according to claim 1, wherein, among the at least three primitives included in the first group of primitives, at least one primitive is at least one point in the first coordinate system, and at least one primitive is is at least one plane in the first coordinate system, and among the at least three primitives included in the second group of primitives, at least one primitive is at least one point in the second coordinate system, and at least one A primitive is at least one plane in the second coordinate system. 3. The method of claim 1, wherein the data is acquired by a movable camera. 4. The method of claim 1, wherein the data includes texture and depth. 5. The method of claim 1, wherein the registration uses random sampling consensus (RANSAC). 6. The method of claim 1, wherein the data is in the form of a series of frames acquired by a camera. 7. The method of claim 6, further comprising: selecting a set of frames from the series of frames as keyframes; and The keyframes are stored in a map, wherein the keyframes include the points and the planes, and the points and the planes are stored in the map as landmarks. 8. The method of claim 7, further comprising: predict the pose of the camera for each frame; and From the registration, the pose of the camera is determined for each frame to track the camera. 9. The method of claim 1, wherein the registration is in real time. 10. The method of claim 7, further comprising: The application refines the landmarks in the map using the bundle adjustment of the points and the plane. 11. The method of claim 8, wherein the pose of the kth frame is Among them, R _k and t _k represent the rotation matrix and translation vector, respectively. 12. The method of claim 8, wherein the prediction uses a constant speed assumption. 13. The method of claim 6, wherein the point in the frame is located using an optical flow process. 14. The method of claim 1, wherein the correspondence of the plane is prioritized over the correspondence of the points.