CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?
CordisBench is a 1,200-question benchmark introduced to evaluate language models' reasoning about component lifecycles in dynamic agent harnesses. It combines a controlled formal setting with programs executed against Cordis, a runtime managing component dependencies and cleanup. The benchmark asks models to identify affected components, predict state after a specified teardown order, determine conditions under all or some orders, and choose reconfigurations that succeed. Three efficiency-oriented models were evaluated at low reasoning effort with 2, 4, 8, 16, 24, or 32 relevant interactions, using deterministic task-specific scoring. Models generally handle small systems well but become less reliable as more interactions become relevant, especially in predicting final state and reasoning across teardown orders. Additional inference effort recovers marked gains for some models. On the 16-interaction subset, GPT-5.6 Luna uses nearly 3,000 reasoning tokens per question.
Researchers introduced CordisBench, a benchmark of 1,200 questions testing language models' ability to reason about component lifecycles in dynamic agent harnesses. The benchmark uses the Cordis runtime to manage dependencies and cleanup, and tasks include identifying affected components, predicting state after teardown, determining conditions under different orders, and choosing successful reconfigurations. Three efficiency-oriented models were tested at low reasoning effort with varying numbers of relevant interactions. Models performed well on small systems but reliability decreased as interactions increased, particularly for final state prediction and reasoning across teardown orders. Additional inference effort improved performance for some models, but at a cost: GPT-5.6 Luna used nearly 3,000 reasoning tokens per question on the 16-interaction subset.
The benchmark reveals a scaling limitation in current language models' compositional reasoning: as the number of relevant component interactions grows, models struggle to maintain accurate state tracking across teardown orders. The use of deterministic task-specific scoring and a controlled formal setting isolates lifecycle reasoning from execution noise. The observed token cost (nearly 3,000 reasoning tokens per question for GPT-5.6 Luna on 16 interactions) suggests that improved performance may require substantial inference-time computation, highlighting a trade-off between accuracy and efficiency in dynamic agent environments.
Dynamic agent harnesses, where language models can modify their own execution environment, are becoming more prevalent in agentic AI systems. This benchmark highlights a critical reliability gap: models may fail to reason about the consequences of local changes in complex dependency graphs. For developers building autonomous agents, this implies a need for robust runtime management (like Cordis) and possibly additional verification layers. The observed performance degradation with more interactions suggests that current models are not yet dependable for managing large-scale, self-modifying software systems without human oversight.
For enterprises deploying autonomous agents, this benchmark provides a measurable way to assess model reliability in dynamic environments, potentially reducing the risk of cascading failures. The Cordis runtime itself may have commercial value as a component management layer for agent platforms. The finding that additional inference effort improves performance suggests a potential premium market for high-reasoning-effort model tiers in safety-critical applications. However, the high token costs could impact the economics of large-scale agent deployments.
Future work may focus on improving models' compositional reasoning through targeted training or architectural changes, or on developing hybrid systems that combine language models with symbolic dependency solvers. The benchmark could be extended to include more complex lifecycle scenarios, such as concurrent modifications or failure recovery. As agentic systems become more autonomous, benchmarks like CordisBench will be essential for measuring progress in safe and reliable self-modification. The cost of reasoning tokens may drive research into more efficient inference methods for such tasks.