Understanding and retrieving information in 3D scenes poses a significant challenge in artificial intelligence (AI) and machine learning (ML), particularly in grasping complex spatial relationships and detailed properties of objects in 3D spaces. While large foundational models such as CLIP [1] have made impressive progress in the 2D domain, their direct application to 3D scene understanding is not straightforward. Therefore, we aim to leverage existing large foundation models to understand 3D scenes and extract 3D information, avoiding building an entirely new 3D model from scratch.
The advancement of large foundation models has inspired recent studies to establish connections among images, text, and 3D data. The work in [2] extracts features from 3D objects with a 3D encoder and aligns them with features extracted by a visual encoder and a text encoder from corresponding rendered 2D images and text, enabling information retrieval of 3D objects from different modalities. However, the model mainly focuses on 3D objects and is limited in understanding complicated 3D spaces like 3D indoor scenes.
[3] Reconstructed compact 3D indoor scenes from multi-view images using DVGO and aligned them with semantic segmentation extracted from corresponding 2D multiview images using the CLIP-LSeg model. A visual reasoning pipeline is applied for reasoning tasks in 3D spaces.
Presently, we are going to develop a ML model that leverages the existing large foundation models to get a better understanding of 3D scenes with lower computational complexity.

Reference:
[1]Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. “Learning transferable visual models from natural language supervision,” In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
[2] Hegde, Deepti, Jeya Maria Jose Valanarasu, and Vishal Patel. “Clip goes 3D: Leveraging prompt tuning for language grounded 3D recognition,” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[3] Hong, Yining, et al. “3D Concept Learning and Reasoning from Multi-View Images,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Image credits:
The image showing the architecture of CG3D is from [2].
The image showing the architecture of 3D-CLR is from [3].