Self-supervised endoscopy depth estimation framework with CLIP-guidance segmentation

Zhuoyue Yang

doi:10.1016/j.bspc.2024.106410

2024-09-01

PRE

DOI：10.1016/j.bspc.2024.106410

原文链接

原文求助

摘要

En 中文

Background and objective: Depth estimation has very broad potential in medical image analysis and is important for applications such as augmented reality surgical navigation and preoperative planning. Compared with segmentation tasks that can obtain ground truth through manual annotation, it is difficult to obtain a large number of real values for depth estimation tasks that are limited by hardware conditions in endoscopic environments. Methods: To address the challenge, we propose a novel framework that utilizes segmentation tasks to improve encoder performance in a self -supervised depth estimation network. For the first time, we leverage the Contrastive Language -Image Pre -training (CLIP) method to improve the performance of endoscopy segmentation models. Depth estimation networks can also benefit from this training process indirectly. In addition, we design a semantic -guidance loss function to improve the performance. Results: Our proposed method is systematically evaluated on three datasets. Experiments have verified that the proposed framework can assist the network model in obtaining smaller errors. Compared with other stateof-the-art methods, our framework obtains 0.081 and 0.097 on absolute relative error metrics in quantitative evaluations on SCARED and SERV -CT datasets respectively. In qualitative experiments on real surgery datasets, our proposed method also shows more ideal results. Conclusion: The experiments in this study illustrate that our proposed method can alleviate the problem of difficulty in improving network performance due to the lack of real values of depth data. The visual performance of our approach illustrates the application potential in the clinic. Our method helps doctors obtain depth perception and visual cues simultaneously, thereby reducing the difficulty of surgery and the pain of patients.

Keyword:

3D reconstruction

Endoscopy

Depth estimation

Semantic segmentation

CLIP