Disclosure-Gated User Simulation for Companion-Agent Evaluation
A paper proposes a disclosure-gated user simulation for evaluating companion agents. The simulator uses a large language model to play the user, with a disclosure gate conditioning information release on the companion agent's behavior. The gate has five ordered gates merged onto three observable depth layers. The simulator is trained on a corpus with synthetic and real branches. On the English corpus of CompanionBench, the largest rank displacement across 12 systems under test exceeds the noise band, while per-system scores show no detectable change.
The paper addresses a known failure in LLM-based user simulation for companion-agent evaluation: simulated users are excessively cooperative, allowing systems to score by asking many questions rather than making the user willing to speak. The proposed solution is a disclosure gate that conditions information release on the companion agent's behavior. The gate's state is a ladder of five ordered gates merged onto three observable depth layers. The simulator is trained against this specification, learning gating behavior from the synthetic branch of the training corpus and speech patterns from the real branch. After training, the simulator does not need to be told at runtime which gate each item sits behind. On the English corpus of CompanionBench, the largest rank displacement across 12 systems under test exceeds the noise band set by re-running the environment under a new seed, while per-system scores show no detectable change.
The disclosure gate is a load-bearing component of the environment. The simulator learns gating behavior from the synthetic branch and speech patterns from the real branch, enabling runtime operation without explicit gate labels. The evaluation shows that the gate affects system rankings (largest rank displacement exceeds noise) but not per-system scores, suggesting it changes relative comparisons without altering absolute performance metrics.
This work improves the reliability of scalable evaluation for companion agents by addressing the over-cooperativeness of simulated users. It may influence how AI companion systems are benchmarked and compared, potentially leading to more meaningful evaluation metrics in the consumer AI space.
More reliable evaluation can guide development priorities for companion-agent products, helping companies differentiate based on genuine user engagement rather than superficial interaction counts. This may affect product roadmaps and competitive positioning in the consumer AI market.
Observable next signals include adoption of disclosure-gated simulation in other companion-agent benchmarks, extension to non-English corpora, and application to other interactive agent evaluation domains. Further research may explore more complex gate structures or integration with other evaluation frameworks.