Tencent Hunyuan open-sources DiagSafe, a diagnostic LLM safety benchmark with prompt-level rubrics
Tencent Hunyuan released DiagSafe, an open-source diagnostic safety benchmark with 3,489 bilingual prompts, an 84-label taxonomy, and prompt-level rubrics. The repository is available on GitHub. Rubric-based judging reached up to 95.34% agreement with human annotations.
China context
- Original name
- 腾讯混元
- Outside China
- Open weights · github.com
- Claims
- Company-reported; not yet independently evaluated
- For builders
- Developers can use DiagSafe to test their own models against instruction attacks and fine-grained safety categories, and to benchmark against the 11 evaluated models.
- For investors
- The release signals Tencent Hunyuan's investment in safety tooling, which may differentiate its models in regulated markets and enterprise deals.
Translated from Chinese. Quotes and facts link to the original sources.
DiagSafe is a diagnostic safety benchmark for large language models, built from 3,489 prompts grounded in an 84-label, three-tier safety taxonomy covering Chinese and English, single- and multi-turn dialogues, and direct harmful requests as well as several attack forms. Every prompt comes with its own safety items and exemptions, combined with eight shared usefulness items; an LLM judge rules on each item and cites evidence. The benchmark includes DiagBase (2,401 prompts) and DiagAttack (1,088 prompts), with DiagAttack rewrites single-turn direct requests into instruction attacks. Evaluation of 11 leading LLMs showed rubric-based judging reaches up to 95.34% agreement with human annotations and stays above 89% across judge models, while on DiagAttack every model's unsafe rate rises to 1.9–8.4× its DiagBase value.
DiagSafe introduces prompt-level rubrics where each prompt has its own safety items and exemptions, plus eight shared usefulness items scored 0–3. An LLM judge rules on each item and cites evidence. The benchmark contains 3,489 prompts (DiagBase 2,401, DiagAttack 1,088) across an 84-label, three-tier taxonomy built from 204 safety provisions distilled from public AI safety guidelines of China, the US, the EU and other regions. Rubric-based judging achieved 89.41%–95.34% agreement with human annotations and cross-judge consistency (Cohen's κ) of 0.828–0.864, outperforming safety classifiers and prompt-based paradigms. On DiagAttack, unsafe rates rose to 1.9–8.4× DiagBase values, with 8 of 11 models exceeding 40% unsafe.
AI safety teams and model evaluators outside China gain a reproducible, open-source benchmark that exposes instruction-attack vulnerabilities—where harmful intent is embedded in benign-looking tasks—which existing benchmarks rarely cover. This gives developers a concrete tool to measure and diagnose safety failures before deployment, potentially reducing compliance and reputational risk.
For enterprises evaluating Chinese LLMs, DiagSafe provides a standardized, evidence-citing safety assessment that can inform procurement and deployment decisions, especially for applications requiring high safety assurance. It may also pressure model vendors to improve safety on instruction attacks, potentially reducing downstream liability for adopters.
Watch whether other model providers adopt DiagSafe for public safety reporting or internal red-teaming, and whether follow-up releases extend the benchmark to more languages or attack types. A specific signal is if Hugging Face leaderboards or independent evaluators incorporate DiagSafe results.