A game theory for foundation models shows new paths to rational cooperation through similarity inference
A paper on arXiv (cs.AI) published 2026-08-04 reports that foundation model agents using optimal planning consistently converge to stable cooperation in stylized social dilemmas, contradicting classical game theory predictions of mutual defection. The authors introduce the 'embedded Bayesian agent' model, where agents model themselves as part of the universe and maintain epistemic uncertainty about their own decision-making algorithms. Cooperation emerges when agents infer behavioral similarity with others.
As autonomous agents powered by foundation models are increasingly integrated into social and economic systems, understanding their collective behavior is essential for safety and cooperation. Classical game theory assumes decoupled agency, but modern AI agents jointly predict their own actions and external observations. A new study finds that foundation model agents engaging in optimal planning consistently converge to stable cooperation in social dilemmas, directly contradicting classical predictions of mutual defection. The researchers introduce the 'embedded Bayesian agent' model, shifting from decoupled to embedded agency, where agents model themselves as part of the universe and maintain epistemic uncertainty about their own decision-making algorithms. By inferring whether others are behaviorally similar, these agents achieve rational cooperation.
The embedded Bayesian agent framework replaces the decoupled agency assumption with agents that jointly model self and environment, maintaining uncertainty about their own policies. This allows agents to infer behavioral similarity with others, leading to cooperative equilibria in social dilemmas where classical game theory predicts defection. The approach leverages foundation models' ability to perform optimal planning under uncertainty.
This research suggests that AI agents built on foundation models may naturally tend toward cooperation in multi-agent settings, which could reduce risks of adversarial behavior in autonomous systems. It implies that designing agents with embedded agency and similarity inference could be a path to safer AI deployments in economic and social systems.
If foundation model agents reliably cooperate in strategic interactions, this could enable new applications in automated negotiation, supply chain coordination, and multi-agent financial systems, reducing the need for explicit incentive engineering and lowering transaction costs.
Next signals to watch include empirical validations of the embedded Bayesian agent model in more complex, real-world multi-agent environments, and whether this cooperative tendency scales with model size and capability. Further theoretical work may explore the robustness of cooperation under adversarial perturbations or heterogeneous agent populations.