ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives
ViTAMINS integrates synthetic hard negatives into unsupervised vision transformer pretraining. It is benchmarked on ImageNet, transfer learning, image retrieval, copy detection, and image/video segmentation. Learned representations contain explicit semantic content and serve as classifiers, improving up to +11.3% over baselines. ViT-B surpasses V-JEPA with ViT-L while being more resource efficient.
ViTAMINS is a method that integrates synthetic hard negatives into unsupervised vision transformer pretraining to improve representation quality. It is benchmarked on ImageNet and transfer learning, image retrieval, copy detection, and image/video segmentation tasks. The proposed negatives give rise to emergent properties where learned representations contain explicit information about semantic content and serve as excellent classifiers (up to +11.3% over baselines). ViTAMINS achieves these benefits through simple modifications to existing contrastive frameworks and outperforms competing methods while being more resource efficient; for example, ViT-B surpasses V-JEPA with ViT-L.
The method modifies contrastive frameworks by injecting synthetic hard negatives, leading to representations that encode explicit semantic content. This suggests that hard negative mining can induce more structured and interpretable features in self-supervised vision transformers, potentially reducing reliance on generative or self-distillation objectives.
Resource efficiency gains (ViT-B outperforming ViT-L) could lower compute requirements for high-quality visual representations, making advanced self-supervised pretraining more accessible. The emergent classification ability may reduce the need for separate classifier heads in downstream tasks.
Improved representation quality with lower resource usage can reduce training costs and accelerate development of vision-based products. The emergent classification property may simplify model pipelines and enable new applications in retrieval, copy detection, and segmentation.
Watch for adoption of synthetic hard negatives in other self-supervised learning frameworks and domains beyond vision. Further benchmarks on larger models and real-world deployment scenarios will clarify scalability and practical benefits.