Testing Interchangeability in LLM Agent Teams
A study tested whether LLM agents are interchangeable within multi-agent teams. Eight teams per setting were formed from one base model on the same tasks, with each agent keeping a private notebook across ten formation episodes. Role-matched agents were then traded between teams and performance measured on held-out tasks. Compared to a placebo, swaps cost little in task score but raised communication per unit of progress by 16-63%. In Hanabi, a swapped agent was more expensive than an inexperienced one, suggesting interference from conventions learned with former partners. In Collab-Overcooked, replacing the agenda-setting agent caused most extra communication from the remaining agent. Ablations over base models, decoding temperature, and formation length showed the swap penalty moved with how far independently formed teams drifted apart. Greedy decoding lowered both; doubling team history raised both.
The paper investigates whether LLM agents in multi-agent systems are interchangeable when filling the same role. It finds that swapping agents between independently formed teams incurs little task performance loss but significantly increases communication overhead (16-63% more communication per unit of progress). The effect is linked to team drift: teams that have diverged more in their internal conventions suffer larger swap penalties. Greedy decoding reduces both drift and swap penalty, while longer formation history increases both. The study suggests that agent interchangeability is not free and that team-specific conventions create hidden costs when replacing agents.
The swap penalty correlates with team drift, measured as divergence in independently formed teams. Greedy decoding reduces drift and penalty, while longer formation increases both. This implies that agent policies encode team-specific conventions, and replacing an agent disrupts coordination. The private notebook mechanism across formation episodes likely captures these conventions. The finding that a swapped agent can be more costly than an inexperienced one in Hanabi indicates negative transfer from prior team conventions.
For production multi-agent systems, frequent agent replacement may incur hidden communication costs even if task accuracy remains stable. Teams that have been trained or fine-tuned together may not be modular. Organizations should consider retraining or re-aligning entire teams rather than swapping individual agents, especially in roles involving agenda-setting or coordination. The 16-63% communication overhead could translate to higher latency and token costs in deployed systems.
The findings highlight a potential hidden cost in maintaining multi-agent systems: replacing agents may increase communication overhead, leading to higher operational costs and latency. This is relevant for companies deploying agent teams in customer service, logistics, or collaborative AI. Understanding drift can inform decisions on when to retrain entire teams versus individual agents, potentially saving compute and improving efficiency.
Future work may explore methods to reduce team drift, such as shared convention protocols or periodic re-synchronization. The relationship between decoding temperature and drift suggests inference-time controls could mitigate swap penalties. Longer formation histories increasing drift implies that continual learning in multi-agent systems may exacerbate interchangeability issues. Research may also investigate whether explicit communication protocols can make agents more interchangeable.