Cost-efficient Active Learning for Referring Image Segmentation and Grounding
A research paper proposes an active learning framework for visual grounding that uses only raw images without accompanying text. It generates auxiliary region-text pairs using foundation models and introduces Referred Region Ambiguity as an acquisition function to prioritize images with strong cross-region competition. The method is evaluated on referring image segmentation (RIS) and referring expression comprehension (REC) benchmarks.
The paper addresses the annotation bottleneck in visual grounding by formulating active learning under a setting where only raw images are available. It uses foundation models to generate auxiliary region-text pairs and defines Referred Region Ambiguity to measure whether model confidence collapses onto a single region or disperses across multiple candidates. This acquisition function selects images with high visual ambiguity, which are more informative for learning discriminative referring expressions. A referring-expression annotation interface is also designed to help annotators quickly write discriminative language. Experiments on RIS and REC benchmarks show consistent improvements.
The core technical contribution is the Referred Region Ambiguity acquisition function, which quantifies the dispersion of model confidence across candidate regions. By leveraging foundation models to generate pseudo region-text pairs, the method estimates ambiguity without ground-truth text, enabling active sample selection in a text-free setting. This approach targets images where multiple regions compete for the same referring expression, which are likely to yield more informative annotations.
The method reduces annotation costs for visual grounding tasks, which are critical for applications like autonomous driving, robotics, and image editing. By focusing human effort on ambiguous images, it can lower the cost and time required to build high-quality referring expression datasets, potentially accelerating deployment of vision-language models in industry.
For companies developing vision-language models, this active learning framework can reduce data annotation expenses and improve model performance per annotation dollar. It enables more efficient use of human annotators by focusing on ambiguous cases, potentially shortening development cycles and lowering barriers to entry for custom visual grounding applications.
Future work may extend the active learning framework to other vision-language tasks, such as visual question answering or image captioning, where annotation costs are similarly high. The approach could also be integrated with interactive annotation tools to further streamline dataset creation. Observing whether the method is adopted in open-source datasets or commercial annotation platforms would be a key next signal.