TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors
TIER is a benchmark for behavioral safety evaluation of LLMs, covering four risk domains and four threat levels from explicit harmful requests to sophisticated jailbreaks. Responses are assessed using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs show safety behaviors evolve gradually across threat levels, contextual prompts yield the most diverse behaviors, jailbreaks reveal the largest robustness gaps, and models with similar Attack Success Rates can exhibit distinct response distributions.
Researchers introduced TIER, a Threat Implicitness Benchmark for behavioral safety evaluation of large language models. The benchmark covers four risk domains and four threat levels, ranging from explicit harmful requests to sophisticated jailbreaks. Responses are evaluated using a six-label behavior scale and two independent LLM judges. Experiments on six open-weight LLMs revealed that safety behaviors evolve gradually across threat levels rather than shifting directly from refusal to compliance. Contextual prompts produced the most diverse behaviors, while jailbreaks exposed the largest robustness gaps. Notably, models with similar Attack Success Rates can exhibit distinct response distributions, highlighting the need for behavior-aware LLM safety evaluation.
TIER introduces a multi-level threat implicitness framework and a six-label behavior scale, moving beyond binary safety metrics. The use of two independent LLM judges for response assessment suggests a scalable evaluation approach. The finding that models with similar Attack Success Rates can have different response distributions indicates that aggregate metrics may mask important behavioral differences, and that safety evaluation should consider the full distribution of responses across threat levels.
The benchmark addresses a gap in current LLM safety evaluation by focusing on behavioral responses rather than binary outcomes. This could influence how AI developers and safety researchers assess model robustness, particularly against jailbreaks and contextual prompts. The emphasis on open-weight models suggests relevance for the broader AI community, including those without access to proprietary models.
For AI developers, TIER offers a more nuanced safety evaluation that could help identify vulnerabilities not captured by traditional metrics. This may lead to better risk management and compliance with emerging AI safety regulations. For enterprises deploying LLMs, behavior-aware evaluation could inform model selection and deployment safeguards.
Future work may extend TIER to more models, including closed-weight systems, and refine the behavior scale. The benchmark could become a standard tool for safety evaluation, potentially integrated into model development pipelines. Observing how models' response distributions change with fine-tuning or alignment techniques may provide insights into improving safety without sacrificing utility.