Tencent Hunyuan releases AppCIF-Bench, an application-level complex instruction following benchmark
Tencent Hunyuan released AppCIF-Bench, an open-source benchmark with 466 samples and 3,017 rubrics across five scenarios in Chinese and English. It evaluates LLMs on complex instruction following, with Hard Satisfaction Rates from 1.1% to 45.9% across 12 models. The evaluation code, simulation engines, and dataset are available on GitHub.
China context
- Original name
- 腾讯混元
- Outside China
- Open weights · github.com
- Claims
- Company-reported; not yet independently evaluated
- For builders
- Developers outside China can use the benchmark to evaluate their own models or fine-tuned variants on complex instruction following, and can integrate custom models via OpenAI-compatible endpoints or Python functions.
- For investors
- The release signals Tencent Hunyuan's commitment to open research and may attract developer mindshare, but the benchmark's adoption and impact on model rankings remain to be seen.
AppCIF-Bench is the first application-level complex instruction following benchmark, stratifying difficulty by horizon length and logical depth. It includes executable code simulation engines for real-world simulation tasks and supports LLM-as-a-judge evaluation with SSR and HSR metrics. The repository provides Chinese and English datasets, evaluation scripts, and instructions for custom model integration.
The benchmark uses a Prompt-as-Environment paradigm with system prompts up to ~14k words and code simulation engines averaging ~3.5k lines of Python. It defines two metrics: Soft Satisfaction Rate (rubric-level) and Hard Satisfaction Rate (sample-level all rubrics satisfied). Evaluation supports OpenAI-compatible endpoints and custom Python functions, with a default judge of Doubao-Seed-2.0-lite.
LLM developers and enterprise adopters now have a public benchmark that exposes poor performance on long-horizon, distractor-laden instruction following, with top models achieving only 45.9% Hard Satisfaction Rate. This creates pressure to improve robustness in production-grade applications where complex instructions are common.
For enterprises evaluating LLMs for complex workflow automation, AppCIF-Bench provides a standardized way to measure instruction following under realistic conditions, potentially reducing integration risk. For model providers, strong performance could differentiate offerings in a competitive market.
A specific signal to check is whether model providers begin reporting AppCIF-Bench SSR/HSR scores in their release notes or technical reports. Another observable indicator is the appearance of AppCIF-Bench in third-party leaderboards or evaluation suites.