.png){kind=icon}
返回
Fusing differentiable rendering and language-image contrastive learning for superior zero-shot point cloud classification
.png){kind=icon}
PRE
AI
DOI：10.1016/j.displa.2024.102773.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
Zero-shot point cloud classification involves recognizing categories not encountered during training. Current models often exhibit reduced accuracy on unseen categories without 3D pre-training, emphasizing the need for improved precision and interoperability. We propose a novel approach integrating differentiable rendering with contrastive language-image pre-training. Initially, differentiable rendering autonomously learns representative viewpoints from the data, enabling the transformation of point clouds into multi-view images while preserving key visual information. This transformation facilitates optimized viewpoint selection during training, refining the final feature representation. Features are extracted from the multi-view images and integrated into a global multi-view feature using a cross-attention mechanism. On the textual side, a large language model (LLM) is provided with 3D heuristic prompts to generate 3D-specific text reflecting category-specific traits, from which textual features are derived. The LLM's extensive pre-trained knowledge enables it to capture abstract notions and categorical features relevant to distinct point cloud categories. Visual and textual features are aligned in a unified embedding space, enabling zero-shot classification. Throughout training, the Structural Similarity Index (SSIM) is integrated into the loss function to encourage the model to discern more distinctive viewpoints, reduce redundancy in multi-view imagery, and enhance computational efficiency. Experimental results on the ModelNet10, ModelNet40, and ScanObjectNN datasets demonstrate classification accuracies of 75.68%, 66.42%, and 52.03%, respectively, surpassing prevailing methods in zero-shot point cloud classification accuracy.
Keyword:
Computer vision
Differentiable rendering
Zero-shot learning
Language-image fusion
Point cloud classification
期刊
IF:
3.4
论文数:
2.3K
被引数:
3.2K
机构
引用论文
A semi-supervised 3D object detection method for autonomous driving一种面向自主驾驶的半监督三维目标检测方法.png)
DISPLAYS
DISPLAYSIF3.4
PREAI
分享收藏Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review自动驾驶中图像和点云融合的深度学习: 综述
OAAI
OAAI分享收藏Mixture reality-based assistive system for visually impaired people基于混合现实的视障人士辅助系统DISPLAYS
DISPLAYSIF3.4
PREAI
分享收藏分享收藏Attention-shift based deep neural network for fine-grained visual categorizationPATTERN RECOGNITION
PATTERN RECOGNITIONIF7.6
PREAI
分享收藏
查看更多内容