Prompting Vision-Language Model for Nuclei Instance Segmentation and Classification

Jieru Yao

doi:10.1109/TMI.2025.3579214

Prompting Vision-Language Model for Nuclei Instance Segmentation and Classification

2025-06-25

0

PRE

AI

DOI：10.1109/TMI.2025.3579214

原文链接

原文求助

摘要

En 中文

Nuclei instance segmentation and classification are a fundamental and challenging task in whole slide Imaging (WSI) analysis. Most dense nuclei prediction studies rely heavily on crowd labelled data on high-resolution digital images, leading to a time-consuming and expertise-required paradigm. Recently, Vision-Language Models (VLMs) have been intensively investigated, which learn rich cross-modal correlation from large-scale image-text pairs without tedious annotations. Inspired by this, we build a novel framework, called PromptNu, aiming at infusing abundant nuclei knowledge into the training of the nuclei instance recognition model through vision-language contrastive learning and prompt engineering techniques. Specifically, our approach starts with the creation of multifaceted prompts that integrate comprehensive nuclear knowledge, including visual insights from the GPT-4V model, statistical analyses, and expert insights from the pathology field. Then, we propose a novel prompting methodology that consists of two pivotal vision-language contrastive learning components: the Prompting Nuclei Representation Learning (PNuRL) and the Prompting Nuclei Dense Prediction (PNuDP), which adeptly integrates the expertise embedded in pre-trained VLMs and multifaceted prompts into the feature extraction and prediction process, respectively. Comprehensive experiments on six datasets with extensive WSI scenarios demonstrate the effectiveness of our method for both nuclei instance segmentation and classification tasks. The code is available at https://github.com/NucleiDet/PromptNu

Keyword:

Digital pathology analysis

nuclei instance segmentation

nuclei classification

vision-language model

prompt learning

Prompting Vision-Language Model for Nuclei Instance Segmentation and Classification

摘要

期刊

机构

引用论文