CN117456003A - Category-level object 6D pose estimation method and system based on dynamic key point detection - Google Patents

Abstract

The invention provides a category-level object 6D pose estimation method and system based on dynamic key point detection, and relates to the field of computer vision. The category-level object 6D pose estimation method based on dynamic key point detection comprises the steps of receiving image data of an object, wherein the image data comprise RDB images and point clouds, and the point clouds are formed by randomly sampling pixels in a depth image by combining internal parameters of a camera and then projecting the pixels into a scene; respectively extracting image features of the RDB image and point cloud features of the point cloud, and carrying out splicing and fusion on the image features and the point cloud features to obtain fused features; inputting the fused characteristics into a preset dynamic key point detection network to extract key points of an object; and inputting the key points into a preset multi-scale pose prediction network, and outputting the final pose of the object 6D. The method can adaptively extract the key points of the object from the observed scene, and can achieve better effect even if more noise points exist in the scene and the shielding condition is serious.

Description

Category-level object 6D pose estimation method and system based on dynamic key point detection

Technical field

The present invention relates to the field of computer vision technology, specifically a method and system for 6D pose estimation of category-level objects based on dynamic key point detection.

Background technique

Object 6D pose estimation technology is an important technology in the fields of computer vision and robotics. It is used to accurately determine the pose of a three-dimensional object in six degrees of freedom, that is, three-dimensional translation and rotation. This technology has broad uses in many applications, such as automated manufacturing, robotic operations, augmented reality, virtual reality, driverless cars, and more.

Although the current instance-level 6D object pose estimation method based on fixed key point detection has good performance and robustness, its only effective feature for a single instance makes this type of method still far from being implemented in reality. Therefore, a category-level 6D object pose estimation method based on the normalized category coordinate space is proposed. This type of method can detect the object poses of different instances of the same category of objects, has high versatility, and is closer to actual production. need.

Current category-level 6D object pose estimation methods can be roughly divided into two categories. The first is the direct regression pose parameter method based on the extracted features. However, due to the complexity and non-convexity of the three-dimensional space rotation matrix group, this type of method is difficult to optimize and often fails to achieve the accuracy and robustness required in actual application scenarios. The second is a method based on dense object coordinate system coordinate prediction, which aims to detect the position of each point in the scene under the normalized object coordinate system, and through these correspondences, use PnP or pose prediction network for post-processing and output object attitude. These methods convert the regression of object pose into a prediction of the position of each point in the scene in the object coordinate system, making the network easier to fit and optimize. Although the above method avoids the problem of difficult optimization of 3D rotation groups, 3D point clouds often contain a lot of noise. Directly predicting the coordinates of all points in the scene in the object coordinate system will affect the performance of the network due to the presence of noise points. Not only that, the scale of the scene point cloud is often huge, and the method of predicting all points will cause excessive computational overhead and high storage requirements for the device, which are not conducive to the practical implementation of the algorithm. Therefore, a category-level object 6D pose estimation method based on dynamic key point detection is proposed to solve the problems of the existing methods mentioned above.

Contents of the invention

(1) Technical problems solved

In view of the shortcomings of the existing technology, the present invention provides a category-level object 6D pose estimation method and system based on dynamic key point detection, which can adaptively extract key points of objects from the observation scene, even when there are many noise points in the scene. , it can achieve better results even when the occlusion situation is serious.

(2) Technical solutions

In order to achieve the above objectives, the present invention is achieved through the following technical solutions:

The first aspect provides a category-level object 6D pose estimation method based on dynamic key point detection, including:

Receive image data of the object. The image data includes RDB images and point clouds. The point clouds are formed by randomly sampling the pixels in the depth map and projecting them into the scene in combination with internal parameters of the camera;

Extract the image features of the RDB image and the point cloud features of the point cloud respectively, and splice and fuse the image features and point cloud features to obtain the fused features;

Input the fused features into the preset dynamic key point detection network to extract the key points of the object;

Input the key points into the preset multi-scale pose prediction network, aggregate the local structure information into key points to obtain key points with multi-scale information, and predict the key points in the object through the key point features with multi-scale information. The position in the spatial coordinate system, and the output key point's position in the scene, the key point's characteristics in the scene, and the key point's position and characteristics in the object space coordinate system are spliced to form multiple sets of correspondences, through multiple layers The perceptron outputs the final 6D pose of the object.

Preferably, the feature extractor of the RDB image adopts Resnet18 convolutional neural network.

Preferably, separately extracting the image features of the RDB image and the point cloud features of the point cloud specifically includes:

Send the input RGB image to the Resnet18 convolutional neural network, and extract the feature map f _rgb ∈R ^h×w×c of the image;

Input the point cloud into the Pointnet++ point cloud feature extraction network to extract the structural features f _point ∈R ^{N ×C} of the point cloud; where N is the number of point clouds.

Preferably, the image features and point cloud features are spliced and fused to obtain the fused features, specifically including:

Project the structural features of the point cloud onto the feature map of the image through the internal parameters of the camera, and extract the corresponding features F _point→rgb ∈R ^N×C of the structural features of the point cloud on the feature map of the image through bilinear interpolation;

By splicing the feature map of the image and the structural features of the point cloud and outputting it through a multi-layer MLP, the fused feature f _fusion ∈R ^N×C is obtained.

Preferably, the input of the fused features into a preset dynamic key point detection network to extract the key points of the object specifically includes:

Introduce attention mechanism and Transformer Layer to dynamically detect key points of objects. Represents N _s KPT queries that are randomly initialized and continuously updated with the training process, and are used to represent N _s key points in the scene;

The query representing different key points interacts with the fused features f _fusion ∈R ^N×C extracted from the scene through the crossattention layer, and the KPT query is updated scene adaptively:

f _kpt =MHCA (f _fusion ; f _kpt ),

Using a similarity-based heat map generation strategy, after calculating the similarity between each KPT query and scene points, the 3D positions and 3D features of key points are generated through heat map weighting:

heatmap=Softmax(Similarity(f′ _kpt ,f _fusion ))

in A weight map that represents the similarity calculation of each key point detector in the scene,/> are the key point coordinates of the final detection.

Preferably, the key points are input into a preset multi-scale pose prediction network, and local structural information is aggregated into key points to obtain key points with multi-scale information, specifically including:

For each detected 3D keypoint By extracting/> The fusion features of the nearest neighbor scene points aggregate local structural information into key points through cross attention:

Among them, knn represents the k-nearest neighbor point in the Euclidean space, and index represents the index operation.

Preferably, the key point features with multi-scale information are used to predict the position of the key point in the object space coordinate system, and the output key point's position in the scene, the key point's characteristics in the scene and the key point's position in the object are The positions and features in the spatial coordinate system are spliced to form multiple sets of correspondences, and the final 6D pose of the object is output through the multi-layer perceptron, including:

Predict the position of key points in object coordinate space through key point features:

And the positions of the output key points in the scene, the characteristics of the key points in the scene, and the positions and characteristics of the key points in the object space coordinate system are spliced together to form N _s sets of correspondences, and the final object is output through the multi-layer perceptron. 6D pose:

In the second aspect, a category-level object 6D pose estimation system based on dynamic key point detection is provided, which is characterized in that the system includes:

A receiving module, configured to receive image data of an object. The image data includes RDB images and point clouds. The point clouds are formed by randomly sampling the pixels in the depth map and projecting them into the scene in combination with internal parameters of the camera;

The feature extraction and fusion module is used to extract the image features of the RDB image and the point cloud features of the point cloud respectively, and splice and fuse the image features and point cloud features to obtain the fused features;

The key point extraction module is used to input the fused features into the preset dynamic key point detection network to extract the key points of the object;

The processing and output module is used to input the key points into the preset multi-scale pose prediction network, aggregate the local structure information into the key points to obtain key points with multi-scale information, and use the key points with multi-scale information to The point feature predicts the position of the key point in the object space coordinate system, and splices the output key point's position in the scene, the key point's characteristics in the scene, and the key point's position and characteristics in the object space coordinate system to form a multi-point Group correspondence relationship, output the final object 6D pose through the multi-layer perceptron.

In a third aspect, a computer-readable storage medium is provided that stores one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the any of the methods.

A fourth aspect provides a computing device, including:

one or more processors, memory, and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including Instructions for performing any of the methods described.

(3) Beneficial effects

The present invention is a category-level object 6D pose estimation method and system based on dynamic key point detection. It can adaptively extract the key points of objects from the observation scene. Even when there are many noise points in the scene and the occlusion situation is serious, it can achieve a higher accuracy. Good results. Secondly, this patent designs two modules that consider local features of key points and pose prediction networks based on correspondence relationships. It can better extract the local spatial geometric features around key points and use the corresponding relationship to return the object pose. And it is trained in the digital twin simulation system, which can ultimately greatly improve the accuracy of 6D pose estimation of category-level objects on the existing data set.

Description of the drawings

Figure 1 is a flow chart of the 6D pose estimation method of category-level objects based on dynamic key point detection according to the present invention;

Figure 2 is an analysis diagram of the method in the embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

Example

As shown in Figure 1-2, embodiments of the present invention provide a category-level object 6D pose estimation method based on dynamic key point detection, including:

Receive image data of the object. The image data includes RDB images and point clouds. The point clouds are formed by randomly sampling the pixels in the depth map and projecting them into the scene in combination with internal parameters of the camera;

Extract the image features of the RDB image and the point cloud features of the point cloud respectively, and splice and fuse the image features and point cloud features to obtain the fused features;

Input the fused features into the preset dynamic key point detection network to extract the key points of the object;

Input the key points into the preset multi-scale pose prediction network, aggregate the local structure information into key points to obtain key points with multi-scale information, and predict the key points in the object through the key point features with multi-scale information. The position in the spatial coordinate system, and the output key point's position in the scene, the key point's characteristics in the scene, and the key point's position and characteristics in the object space coordinate system are spliced to form multiple sets of correspondences, through multiple layers The perceptron outputs the final 6D pose of the object.

Furthermore, for input RGBD images, Resnet18 is used as the feature extractor for RGB images. For the depth map D, the pixels in the depth map are randomly sampled and projected into the scene by combining camera internal parameters to form a point cloud. For input data of different modalities, the network first sends the input RGB image into the Resnet18 convolutional neural network, and extracts the feature map f _rgb ∈R ^h×w×c of the image. For the input point cloud features, the network inputs them into the Pointnet++ point cloud feature extraction network to extract the structural features f _point ∈R ^N×C of the point cloud. Where N is the number of point clouds. Then the scene point cloud is projected onto the image feature map through the camera internal parameters, and the corresponding feature f _point→rgb ∈R ^N×C of the scene point cloud on the feature map is extracted through bilinear interpolation. Finally, the fused feature f _fusion ∈R ^N×C is obtained by splicing the features of the two modalities and outputting it through a multi-layer MLP.

Further, for each scene input, after extracting multi-modal fusion features, we designed a dynamic key point detection network as shown in Figure 1, which is used to adaptively extract object key points in the scene. In order to achieve adaptive dynamic detection of object key points in different scenes, this project plans to introduce the attention mechanism and Transformer Layer to achieve scene adaptation. Represents N _s KPT queries that are randomly initialized and continuously updated with the training process, and are used to represent N _s key points in the scene. These queries representing different key points are interacted with the fused features f _fusion ∈R ^N×C extracted from the scene through the cross attention layer, and the KPT query is updated scene adaptively:

f′ _kpt =MHCA(f _fusion ; f _kpt ),#

The updated KPT query aggregates scene adaptive features and is used in the subsequent key point detection module. Next, a similarity-based heat map generation strategy is used. After calculating the similarity between each KPT query and scene points, the 3D positions of key points and 3D features are generated through heat map weighting. Specifically:

heatmap=Softmax(Similarity(f′ _kpt ,f _fusion ))#

in A weight map that represents the similarity calculation of each key point detector in the scene,/> are the key point coordinates of the final detection. The key points of dynamic detection can adapt to different scenes and changes. No matter how the position, angle, lighting conditions, etc. of the target object change, the key points can be accurately detected, and this set of key points can be generalized to different instances of the same category. objects, making the model more generalizable and more suitable for the task of category-level object pose estimation. This enables better accurate and robust pose estimation in the future.

Furthermore, this patent designs a multi-scale pose prediction network, which is mainly divided into two modules: a local feature aggregation module and a correspondence-based pose prediction network:

Local feature aggregation module. In order to enable each key point to better extract local information in the scene, thereby generating multi-scale features. This project plans to propose a local feature aggregation module for key point locations. Specifically, for each detected 3D keypoint By extracting the fusion features of its nearest neighbor scene points, local structural information is aggregated into key points through cross attention:

Among them, knn represents the k-nearest neighbor point in the Euclidean space, and index represents the index operation. By aggregating local features through local attention calculation, key points have multi-scale information, which can better predict the pose of scene objects.

Correspondence-based pose prediction network. In order to regress the object pose according to the correspondence relationship output by the network, an advanced deep neural network is used to fit the traditional least squares algorithm, making the pose output from the fitted correspondence relationship more robust. This method first predicts the key point features The position of the key point in the object coordinate space:

And the positions of the output key points in the scene, the characteristics of the key points in the scene, and the positions and characteristics of the key points in the object space coordinate system are spliced together to form N _s sets of correspondences, and the final object is output through the multi-layer perceptron. Posture. in particular:

The prediction based on the correspondence relationship is more in line with the relationship between least squares fitting object coordinate pairs in mathematics, which can make it easier for the network to learn the mapping relationship of pose, and in this method, the correspondence relationship generated by adaptive key points can be used to remove the scene In order to eliminate the influence of noise points, only the most representative key points are selected, making the overall calculation more accurate, more robust and more efficient.

This patent also designs a method to use digital twin simulation technology to assist model training and verify model performance. In order to collect the 6D pose data of objects, we deploy virtual RGBD sensors into the digital twin environment. These sensors simulate sensing devices such as cameras and lidar in the real world. These virtual sensors will record the position and attitude information of objects in real time and generate a large amount of simulation data. We use these simulated data to train and validate deep learning models. By conducting simulation experiments in a digital twin environment, the performance of the model, including its accuracy, robustness and generalization ability, is verified.

In this patent, the Resnet convolutional neural network and pointnet++ point cloud feature extraction network in the RGBD multi-modal feature extraction backbone network are existing technologies in the background technology. This patent adds an adaptive key point detection network based on this technology. , and designed a new pose estimation network based on local feature aggregation, and conducted simulation experiments in a digital twin environment to verify the effectiveness of the technology.

Yet another embodiment of the present invention provides a category-level object 6D pose estimation system based on dynamic key point detection, which is characterized in that the system includes:

A receiving module, configured to receive image data of an object. The image data includes RDB images and point clouds. The point clouds are formed by randomly sampling the pixels in the depth map and projecting them into the scene in combination with internal parameters of the camera;

The feature extraction and fusion module is used to extract the image features of the RDB image and the point cloud features of the point cloud respectively, and splice and fuse the image features and point cloud features to obtain the fused features;

The key point extraction module is used to input the fused features into the preset dynamic key point detection network to extract the key points of the object;

The processing and output module is used to input the key points into the preset multi-scale pose prediction network, aggregate the local structure information into the key points to obtain key points with multi-scale information, and use the key points with multi-scale information to The point feature predicts the position of the key point in the object space coordinate system, and splices the output key point's position in the scene, the key point's characteristics in the scene, and the key point's position and characteristics in the object space coordinate system to form a multi-point Group correspondence relationship, output the final object 6D pose through the multi-layer perceptron.

Embodiments of the present application may be provided as methods or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the literal scripting language JavaScript.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations are mutually exclusive. any such actual relationship or sequence exists between them. Furthermore, the terms "comprises," "comprises," or any other variation thereof are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also those not expressly listed other elements, or elements inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes the stated element.

Claims (10)

1. A category-level object 6D pose estimation method based on dynamic key point detection, which is characterized by: Receive image data of the object, where the image data includes RDB images and point clouds, wherein the point cloud is formed by randomly sampling pixels in the depth map and projecting them into the scene in combination with camera internal parameters; Extract the image features of the RDB image and the point cloud features of the point cloud respectively, and splice and fuse the image features and point cloud features to obtain the fused features; Input the fused features into the preset dynamic key point detection network to extract the key points of the object; Input the key points into the preset multi-scale pose prediction network, aggregate the local structure information into the key points to obtain key points with multi-scale information, and predict the key points in the object through the key point features with multi-scale information. The position in the spatial coordinate system, and the position of the output key point in the scene, the characteristics of the key point in the scene, and the position and characteristics of the key point in the object space coordinate system are spliced to form multiple sets of correspondences. Through multi-layer The perceptron outputs the final 6D pose of the object. 2. A category-level object 6D pose estimation method based on dynamic key point detection according to claim 1, characterized in that: the feature extractor of the RDB image adopts Resnet18 convolutional neural network. 3. A method for 6D pose estimation of category-level objects based on dynamic key point detection according to claim 2, characterized in that: respectively extracting image features of RDB images and point cloud features of point clouds, specifically including: Send the input RGB image to the Resnet18 convolutional neural network, and extract the feature map f _rgb ∈R ^h×w×c of the image, where h, w, and c are the feature map height, feature map width, and feature map channel number respectively. ; Input the point cloud into the Pointnet++ point cloud feature extraction network, and extract the structural features f _point ∈R ^N×C of the point cloud; where N is the number of point clouds, and C is the dimension of each point cloud feature output by the Pointnet++ network. 4. A method for 6D pose estimation of category-level objects based on dynamic key point detection according to claim 3, characterized in that: the image features and point cloud features are spliced and fused to obtain the fused features, specifically including Project the structural features of the point cloud onto the feature map of the image through the internal parameters of the camera, and extract the corresponding features f _point→rgb ∈R ^N×C of the structural features of the point cloud on the feature map of the image through bilinear interpolation; By splicing the feature map of the image and the structural features of the point cloud and outputting it through a multi-layer MLP, the fused feature f _fusion ∈R ^N×C is obtained. 5. A method for 6D pose estimation of category-level objects based on dynamic key point detection according to claim 4, characterized in that: the fused features are input into a preset dynamic key point detection network to extract the object. Key points include: Introduce attention mechanism and Transformer Layer to dynamically detect key points of objects. Represents N _s KPT queries that are randomly initialized and continuously updated with the training process, and are used to represent N _s key points in the scene; The query representing different key points interacts with the fused features f _fusion ∈R ^N×C extracted from the scene through the crossattention layer, and the KPT query is updated scene adaptively: f′ _kpt =MHCA(f _fusion ; f _kpt ), Using a similarity-based heat map generation strategy, after calculating the similarity between each KPT query and scene points, the 3D positions and 3D features of key points are generated through heat map weighting: heatmap=Softmax(Similarity(f′ _kpt , f _fusion )) in A weight map representing the similarity calculation of each key point detector in the scene, are the key point coordinates of the final detection. 6. A method for 6D pose estimation of category-level objects based on dynamic key point detection according to claim 5, characterized in that: the key points are input into a preset multi-scale pose prediction network, Aggregate local structural information into key points to obtain key points with multi-scale information, including: For each detected 3D keypoint By extracting/> The fusion features of the nearest neighbor scene points aggregate local structural information into key points through cross attention: Among them, knn represents the k-nearest neighbor point in the Euclidean space, and index represents the index operation. 7. A category-level object 6D pose estimation method based on dynamic key point detection according to claim 6, characterized in that: the key points are predicted by key point features with multi-scale information in the object space coordinate system. The position of the output key point in the scene, the characteristics of the key point in the scene, and the position and characteristics of the key point in the object space coordinate system are spliced to form multiple sets of correspondences. The final output is through the multi-layer perceptron. 6D pose of the object, including: Predict the position of key points in object coordinate space through key point features: And the positions of the output key points in the scene, the characteristics of the key points in the scene, and the positions and characteristics of the key points in the object space coordinate system are spliced together to form N _s sets of correspondences, and the final object is output through the multi-layer perceptron. 6D pose: 8. A category-level object 6D pose estimation system based on dynamic key point detection, characterized in that the system includes: A receiving module, configured to receive image data of an object. The image data includes RDB images and point clouds. The point clouds are formed by randomly sampling the pixels in the depth map and projecting them into the scene in combination with internal parameters of the camera; The feature extraction and fusion module is used to extract the image features of the RDB image and the point cloud features of the point cloud respectively, and splice and fuse the image features and point cloud features to obtain the fused features; The key point extraction module is used to input the fused features into the preset dynamic key point detection network to extract the key points of the object; The processing and output module is used to input the key points into the preset multi-scale pose prediction network, aggregate the local structure information into the key points to obtain key points with multi-scale information, and use the key points with multi-scale information to The point feature predicts the position of the key point in the object space coordinate system, and splices the output key point's position in the scene, the key point's characteristics in the scene, and the key point's position and characteristics in the object space coordinate system to form a multi-point Group correspondence relationship, output the final object 6D pose through the multi-layer perceptron. 9. A computer-readable storage medium storing one or more programs, characterized in that the one or more programs include instructions that, when executed by a computing device, cause the computing device to perform according to claim 1 Any of the methods described in 1-7. 10. A computing device, characterized by: one or more processors, memory, and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including Instructions for performing any of the methods according to claims 1-7.