Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale
The Vulnerability Localization Benchmark (VLoc Bench) comprises 500 real-world vulnerabilities from 290 repositories across six package ecosystems and 147 CWE categories. Each task pairs repository snapshots immediately before and after a security fix. On the vulnerable snapshot, an agent receives only the CWE description and read-only terminal access and must return the affected files; on the patched snapshot, it must determine that the recorded vulnerability is no longer present. The study evaluates 27 language models and four static-analysis tools under a common agent interface. The strongest system achieves 0.229 File F1, and 38.4% of tasks receive no correct localization from any evaluated model.
Researchers introduced VLoc Bench, a benchmark for vulnerability localization in software repositories. It includes 500 real-world vulnerabilities from 290 repositories across six package ecosystems and 147 CWE categories. Agents are given a CWE description and read-only terminal access on a vulnerable snapshot and must identify affected files; on a patched snapshot, they must confirm the vulnerability is absent. Evaluation of 27 language models and four static-analysis tools shows repository-scale vulnerability localization remains difficult: the best system achieves 0.229 File F1, and 38.4% of tasks have no correct localization from any model.
The benchmark tests agentic security analysis at repository scale, requiring models to navigate unfamiliar codebases using only a CWE description and terminal access. The low File F1 of 0.229 indicates that current models struggle to map vulnerability classes to specific implementation files, and the high rate of tasks with no correct localization (38.4%) suggests a significant gap in code understanding and reasoning across large codebases. The inclusion of patched snapshots tests the ability to verify absence of a vulnerability, adding a binary classification component.
This research highlights a critical limitation in AI-driven security tooling: even the best models cannot reliably locate vulnerabilities in real-world repositories. This has implications for automated code review, vulnerability triage, and security agent deployment. The benchmark provides a standardized way to measure progress, which could drive investment and development in more capable code-understanding models. Static analysis tools are also evaluated, offering a comparison between traditional and AI-based approaches.
For security vendors and enterprises, this benchmark provides evidence that current AI models are not yet reliable for automated vulnerability localization, which may temper expectations and guide investment toward hybrid approaches. However, the benchmark itself is a valuable asset for evaluating and comparing AI security tools, potentially becoming a standard in procurement and development. Companies building code security agents can use VLoc Bench to benchmark their systems and demonstrate improvement.
Future work may focus on improving model architectures for long-context code understanding, integrating retrieval mechanisms, or developing specialized training for vulnerability localization. The benchmark could be expanded to include more languages, ecosystems, and vulnerability types. As models improve, the File F1 metric may become a standard for evaluating security-focused AI agents. The gap between static analysis and LLM performance may narrow with better prompting or fine-tuning.