MLLM-Routed Heterogeneous Ensembles for Robust Cross-Dataset Image Classification
ARMDIL is an ensemble that uses a multimodal large language model (MLLM) agent to dynamically route each image to the most suitable vision backbone. The ensemble includes convolutional neural networks (ResNets), self-supervised representation learners (SSL), and vision-language models (VLMs), each trained on a unified label space constructed from multiple image datasets. ARMDIL performs competitively with specialized training-based routers and improves adaptability by allowing new information to be integrated via simple prompt modifications, while enhancing interpretability through natural language reasoning traces.
ARMDIL, an Adaptive Router for Multi-Domain Image classification with LLMs, is proposed to address generalization challenges in image classification across domains and difficulty levels. It employs an MLLM agent to route images to the most suitable vision backbone among ResNets, SSL models, and VLMs, all trained on a unified label space from multiple datasets. The approach matches specialized training-based routers, offers easy adaptability through prompt modifications, and provides interpretability via natural language reasoning traces.
The key technical contribution is using an MLLM as a zero-shot router for heterogeneous vision backbones, avoiding the need for training a separate routing model. This leverages the MLLM's cross-modal reasoning to assess image characteristics and select the appropriate backbone, enabling dynamic adaptation to new domains through prompt updates rather than retraining. The unified label space across diverse datasets is critical for ensemble consistency.
This approach could reduce the cost and complexity of maintaining multiple specialized image classification models by enabling a single ensemble to handle varied domains. It may lower barriers for enterprises needing robust classification across heterogeneous image sources, as new domains can be added via prompt changes without retraining. The interpretability from reasoning traces could also aid in compliance and debugging in regulated industries.
ARMDIL offers potential cost savings and faster deployment for image classification across multiple domains by reducing the need for separate models and retraining. Its prompt-based adaptability could shorten time-to-market for new domains and improve model governance through interpretable routing decisions.
Next observable signals include publication of full experimental results, open-sourcing of the ARMDIL code or model, and follow-up work applying MLLM routing to other vision tasks such as object detection or segmentation. Potential adoption by cloud AI platforms offering multi-domain image classification services could be a commercialization signal.