Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching
A research paper introduces DroneCATS-Agent, an architecture where a multimodal large language model (MLLM) is a swappable component in a drone control loop, and DroneCATS, a benchmark treating the model as the independent variable. The agent entrusts the model to yaw, search, deliberate when unsure, and self-declare arrival without fine-tuning or function-calling schemas. Evaluation across four capabilities—approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet—reveals that even the simplest embodied settings are far from solved. The roster scales down to 2B parameters, and findings indicate that small open models often navigate into the success radius, suggesting flying is not the primary failure point.
The paper evaluates multimodal LLMs as generalist vision-language-action agents for drone control. It introduces DroneCATS-Agent, where the MLLM is a swappable component, and DroneCATS, a benchmark treating the model as the independent variable. The agent entrusts the model to yaw and search, deliberate when unsure, and self-declare arrival, all without fine-tuning or function-calling schemas. Evaluation across four core capabilities—approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet—reveals that even the simplest embodied settings are far from solved. The roster scales down to 2B parameters, and findings expose a paradox: it is not the flying that fails; small open models often navigate into the success radius.
The architecture treats the MLLM as a swappable component in the drone control loop, with the entire action space declared solely in the prompt. This contrasts with recent systems that narrow the model's decision-making. The benchmark isolates model capability as the independent variable, enabling comparison across frontier and open models down to 2B parameters. The finding that small models can navigate into the success radius suggests that basic spatial reasoning and control may be achievable with modest model sizes, while higher-level deliberation and search remain challenging.
This work signals a shift toward evaluating generalist models as embodied agents without task-specific fine-tuning or function-calling schemas. The emphasis on open models down to 2B parameters indicates growing interest in deployable, low-cost embodied AI. The benchmark could become a reference for comparing vision-language-action capabilities in robotics and drone applications.
The benchmark and architecture provide a framework for evaluating and potentially deploying MLLMs in drone control without expensive fine-tuning or custom function-calling infrastructure. This could lower barriers for drone automation in inspection, surveillance, and delivery, especially with small open models that reduce compute costs.
Observable next signals include follow-up papers extending DroneCATS to other robotic platforms, adoption of the benchmark by model developers for embodied evaluation, and releases of improved small MLLMs targeting drone control. If small models continue to show competence in navigation, we may see increased focus on deliberation and search capabilities as the key differentiators.