Objective

2

With the rapid development of 3D scanners and 3D point cloud acquisition technologies， the application of 3D point clouds in computer vision， robot guidance， industrial design， and other fields has become increasingly widespread. As long as the point cloud is sufficiently dense， we can construct accurate models to meet the demands of various advanced point cloud tasks. Accurate point clouds facilitate better performance in tasks such as semantic segmentation， completion， classification. However， due to limitations such as sensor resolution， scanning time， and scanning conditions， the acquired point clouds are often sparse. Existing point cloud upsampling methods only address single low-resolution point clouds， and they yield poor results when upsampling highly sparse point clouds at larger magnification rates. And they do not use additional modalities for assistance. Moreover， tactile information has gradually been applied to 3D reconstruction tasks， reconstructing complete 3D models using multi-modal information such as RGB images， depth images， and tactile information. But tactile point clouds have not been applied to point cloud super-resolution tasks yet.

Method

2

In this study， we proposed a tactile-enhanced graph convolutional point cloud super-resolution network that uses dynamic graph convolution to extract tactile features and fuse them with low-resolution point cloud features to obtain more accurate high-resolution point clouds. This network consists of feature extraction module and upsampling module. The feature extraction module extracts features from low-resolution point cloud and tactile point cloud， while the upsampling module performs feature expansion and coordinate reconstruction to output high-resolution point cloud. The key to this network lies in extracting features from tactile point clouds and fusing them with low-resolution point cloud features. The tactile feature extraction module adopts Multilayer Perceptron （MLP） and 4-layer cascaded dynamic graph convolution. The tactile point cloud is mapped to a high-dimensional space through multilayer perceptron for subsequent feature extraction. The dynamic graph convolution module consists mainly of K-Nearest-Neighbors （KNN） and edge convolution. In each dynamic graph convolution， the KNN algorithm is used to recompute the neighboring points of each point and construct the graph structure. KNN algorithm can effectively aggregates local feature information， and edge convolution extracts features of center points and neighboring points. The k-nearest neighbors of each point vary in different network layers， leading to the graph structure dynamically updated in each layer. The feature extraction for the low-resolution point cloud adopts graph convolution. The graph structure is first constructed using the KNN algorithm and then shared with subsequent layers. After fusing the features of the low-resolution point cloud and the tactile point cloud， the point cloud features undergo further progressive feature extraction， mainly using dense connected graph convolution modules. The bottleneck layer compresses features to reduce computational complexity in subsequent layers. Two parallel dense graph convolutions extract local features， while the global pooling layer extracts global features. Finally， the feature rearrangement module and coordinate reconstruction module map the high-dimensional features back to the three-dimensional coordinate system. Compared to the low-resolution point cloud， the local tactile point cloud is denser and more precise， while the low-resolution point cloud is often sparser and contains less local information. With the assistance of tactile information， enhanced local features can be obtained.

Result

2

In this study， we reconstructed a point cloud super-resolution dataset 3DVT （3D Vision and Touch） with tactile information and trained on this dataset. This dataset contains a diverse range of object categories and a sufficiently large number of samples. Using chamfer distance as the evaluation metric， experimental results show that without adding tactile information， the average chamfer distance is 3.009*

<math id="M4"><msup><mrow><mn mathvariant="normal">10</mn></mrow><mrow><mo>-</mo><mn mathvariant="normal">3</mn></mrow></msup></math>

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257101&type=

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257115&type=

5.16466665

2.53999996

，

with one instance of tactile information added， the average chamfer distance decreases to

1.931*

<math id="M5"><msup><mrow><mn mathvariant="normal">10</mn></mrow><mrow><mo>-</mo><mn mathvariant="normal">3</mn></mrow></msup></math>

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257117&type=

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257116&type=

5.16466665

2.53999996

， and with two instances of tactile information added， the average chamfer distance further reduces to 1.916*

<math id="M6"><msup><mrow><mn mathvariant="normal">10</mn></mrow><mrow><mo>-</mo><mn mathvariant="normal">3</mn></mrow></msup></math>

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257117&type=

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257116&type=

5.16466665

2.53999996

. Tactile point clouds can enhance the quality of high-resolution point clouds and serve as an auxiliary aid in point cloud super-resolution tasks. Visualizations of different objects demonstrate that the distribution of upsampled point clouds becomes more uniform and the edges become smoother with the assistance of tactile information. With the assistance of tactile point clouds， the network can better fill in the holes in the point cloud and reduce the generation of outliers. The quantitative results of chamfer distance and density-aware chamfer distance obtained by super-resolution experiments on different objects confirm the effectiveness of tactile point cloud assistance in the super-resolution task. Furthermore， for complex objects， this improvement is even more pronounced. The noise experiments show that at a noise level of 1%， the average chamfer distance without tactile information is 3.132*

<math id="M7"><msup><mrow><mn mathvariant="normal">10</mn></mrow><mrow><mo>-</mo><mn mathvariant="normal">3</mn></mrow></msup></math>

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257107&type=

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257106&type=

5.16466665

2.53999996

， while with the inclusion of two instances of tactile information， the average chamfer distance is 1.954*

<math id="M8"><msup><mrow><mn mathvariant="normal">10</mn></mrow><mrow><mo>-</mo><mn mathvariant="normal">3</mn></mrow></msup></math>

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257117&type=

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257116&type=

5.16466665

2.53999996

. At a noise level of 3%， the average chamfer distance without tactile information is 3.331*

<math id="M9"><msup><mrow><mn mathvariant="normal">10</mn></mrow><mrow><mo>-</mo><mn mathvariant="normal">3</mn></mrow></msup></math>

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257119&type=

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257118&type=

5.16466665

2.53999996

， and with the inclusion of two instances of tactile information， the average chamfer distance is 2.001*

<math id="M10"><msup><mrow><mn mathvariant="normal">10</mn></mrow><mrow><mo>-</mo><mn mathvariant="normal">3</mn></mrow></msup></math>

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257123&type=

https://html.publish.founderss.cn/rc-pub/api/common/picture?pictureId=65257120&type=

5.16466665

2.53999996

. The experiments demonstrate that with the assistance of tactile information， the impact of noise on the network is reduced， indicating that the network exhibits strong robustness.

Conclusion

2

Dynamic graph convolution can effectively extract initial features from tactile point clouds， and the tactile point cloud features contain rich local information. Through feature fusion， it can effectively assist the point cloud super-resolution task. The proposed tactile-enhanced graph convolutional point cloud super-resolution network in this paper uses dynamic graph convolution to extract tactile features and fuse them with low-resolution point cloud features， effectively improving the quality of high-resolution point clouds and exhibiting strong robustness. The superiority of the method lies in its ability to achieve better results by incorporating tactile information without updating the network architecture. This method can provide high-quality point clouds for advanced visual tasks such as point cloud classification and object detection， laying the foundation for further development and application of point clouds.