GazeRefine: Expert Gaze as a Test-Time Prompt for Training-Free Medical Image Segmentation
GazeRefine is a training-free framework that uses gaze as an inference-time prompt for zero-shot medical image segmentation. Sparse, duration-weighted fixations are converted into foreground and background priors that initialize semantic prototypes in frozen DINOv3 feature space. These prototypes are iteratively refined through foreground-background discrimination, feature-space affinity propagation, and anchoring to the initial gaze guidance. GazeRefine requires no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates. The method was evaluated on gaze-annotated polyp segmentation and prostate MRI segmentation, showing strong performance on colonoscopy images and competitive performance on prostate MRI.
GazeRefine introduces a training-free approach to medical image segmentation that uses expert gaze as a test-time prompt. It converts sparse, duration-weighted fixations into foreground and background priors, initializing semantic prototypes in frozen DINOv3 feature space. Iterative refinement through foreground-background discrimination, feature-space affinity propagation, and anchoring to initial gaze guidance allows segmentation to extend beyond directly fixated regions while limiting semantic drift. The method requires no segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates. Evaluation on gaze-annotated polyp segmentation and prostate MRI segmentation shows strong performance on colonoscopy images and competitive performance on prostate MRI, supporting gaze-guided prototype refinement as a promising approach for segmentation-label-efficient, human-in-the-loop medical image segmentation.
GazeRefine leverages frozen DINOv3 features and duration-weighted gaze fixations to create semantic prototypes without any training. The iterative refinement process combines foreground-background discrimination, feature-space affinity propagation, and anchoring to initial gaze guidance, enabling segmentation beyond fixated regions while controlling semantic drift. This training-free, zero-shot approach eliminates the need for segmentation masks, fine-tuning, adapters, prompt encoders, or gradient updates, potentially reducing annotation burden and enabling rapid deployment in medical imaging tasks.
The approach addresses the scalability challenge in medical image segmentation by reducing reliance on dense expert annotations and task-specific training. By using gaze as an inference-time prompt, GazeRefine could enable more efficient human-in-the-loop workflows in clinical settings, where expert time is limited. The strong performance on colonoscopy images suggests potential for real-world polyp detection applications, while competitive results on prostate MRI indicate broader applicability. The release of tools and code may accelerate adoption and further research in label-efficient medical AI.
GazeRefine offers a training-free, zero-shot segmentation method that could reduce annotation costs and time-to-deployment for medical imaging AI. By leveraging expert gaze instead of dense masks, it may enable faster development of segmentation models in resource-constrained settings. The approach could be valuable for companies developing computer-aided detection systems, particularly in gastroenterology and radiology, where labeled data is expensive and scarce.
Observable next signals include publication of the full paper and code release, independent replication studies on additional medical imaging modalities, and potential integration into clinical annotation tools. Further research may explore combining GazeRefine with other foundation models or extending it to 3D medical volumes. Commercial interest could emerge from medical device or AI-assisted diagnostics companies seeking label-efficient segmentation solutions.