BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing
BLOOM-WILT is a full auditing pipeline that elicits natural multi-turn instances of rare behaviours from deployed language models without training cost or access beyond the target's next-token distribution. It uses an auditor model that revises its conversational strategy across rounds and adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt. Evaluated across 4 target models and 8 behaviours, it beats the baseline auditor in 30 of 32 settings and overturns previous model safety rankings.
Researchers introduced BLOOM-WILT, an automated LLM auditing pipeline designed to surface rare behaviours that standard testing misses. The method combines an adaptive auditor that learns from scored interactions with logit tilting to preferentially sample behaviour-relevant generations. In evaluations across four target models and eight behaviours, BLOOM-WILT outperformed the baseline auditor in 30 of 32 settings and changed the relative safety rankings of the models.
BLOOM-WILT uses logit tilting on the target model's next-token distribution, conditioned on an elicitation prompt, to increase sampling probability for behaviour-relevant tokens without additional training. The auditor model iteratively refines its conversational strategy based on previous scored interactions, enabling efficient multi-turn elicitation of rare behaviours.
Automated auditing tools like BLOOM-WILT could become standard for pre-deployment safety testing, as they scale to many behaviours and models without requiring internal access. The ability to overturn safety rankings suggests current model evaluations may be incomplete, prompting vendors to adopt more adversarial testing methods.
BLOOM-WILT offers a cost-effective way for AI developers and auditors to uncover hidden model behaviours, potentially reducing reputational and regulatory risk. Its training-free design lowers barriers to adoption, making it attractive for companies needing scalable safety evaluations.
Expect further research into sample-efficient elicitation techniques and integration of such auditors into model release pipelines. If BLOOM-WILT's approach proves robust, it may influence regulatory expectations for LLM safety testing and lead to new benchmarks for rare behaviour detection.