Visual Language Models Power Open-Set Detection and Surgical Tool Segmentation

Introduction

Large language models have surged in popularity, and visual language models are emerging as a powerful extension that combines image and text for multimodal tasks. The Huolala data science team organized a DS Zone industry exchange featuring Prof. Shi Miao‑Jing to discuss these models.

Fundamental Vision Tasks

1. Object Detection

Traditional object detection relies on large annotated datasets and struggles with scarce data and transfer learning. Standard pipelines involve bounding‑box annotation and category labeling, as illustrated in Figure 1 and Figure 2.

2. Semantic Segmentation

Semantic segmentation assigns a class label to each pixel, requiring massive pixel‑level annotations. It is crucial for autonomous driving and medical imaging, with typical networks such as FCN, SegNet, and PSPNet (see Figure 3).

3. Visual Language Models

Models like OpenAI’s CLIP align image and text embeddings, enabling zero‑shot learning for tasks such as image captioning and classification. Prompt design critically influences performance, and iterative prompt optimization can further improve results.

Open‑Set Object Detection (OVOD)

To address the scarcity of labeled data, Zareian et al. (CVPR 2021) introduced OVOD, which trains on image‑text pairs to acquire an open vocabulary of concepts. This allows detection of novel categories when only a limited set of base classes is annotated.

Prompt Learning (CoOp)

The CoOp method (IJCV 2022) automates prompt engineering by learning contextual embeddings from a small data subset, improving model adaptability across tasks.

DetPro Framework

DetPro extends OVOD by integrating prompt learning into a region proposal network (RPN). It defines negative proposals (low IoU with ground truth) and positive proposals (high IoU) and introduces a background‑explanation scheme for negatives and a context‑aware weighting for positives. The overall architecture and loss functions are shown in Figures 6‑9.

Promptable Surgical Instrument Segmentation

Segmenting surgical tools at the pixel level is challenging due to diverse instrument types, visual similarity, and limited annotated data. Visual language models can leverage textual prompts to identify and segment instruments, dramatically improving performance over conventional vision‑only models (see Figures 11‑12).

Conclusion

Visual language models provide a versatile solution for both open‑set object detection and promptable surgical instrument segmentation, addressing data scarcity and improving accuracy, thereby enhancing safety and efficiency in real‑world applications.