VICBench: A Multi-Language Benchmark for Code Vulnerability Detection
VICBench is a benchmark of 100 verified vulnerability-inducing commits (VICs) for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. It was created through dual annotation by human experts and an agentic workflow. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines. Evaluation shows state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1.
VICBench is a new benchmark for evaluating security vulnerability detection tools. It contains 100 verified vulnerability-inducing commits (VICs) for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. The benchmark was created through dual annotation by human experts and an agentic workflow. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines, significantly larger than prior work. Evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.
The benchmark's VICs average 252.5 lines, much larger than previous datasets, indicating a focus on complex, multi-line vulnerability introductions. The low F1 scores (33.3%-40.1%) of V-SZZ and LLM4SZZ suggest current automated vulnerability detection methods struggle with such complexity, highlighting a gap in handling real-world code changes.
Security tooling vendors and enterprises relying on automated vulnerability detection may need to reassess their tools' effectiveness, as current state-of-the-art approaches leave significant manual effort. The benchmark's multi-language and multi-CWE coverage could become a standard for evaluating and improving detection tools.
For companies developing security scanning tools, VICBench provides a rigorous evaluation standard to benchmark and improve their products. Enterprises can use it to assess the accuracy of vulnerability detection in their software supply chain, potentially reducing manual review costs.
Expect follow-up research to improve VIC identification algorithms, possibly leveraging larger language models or more sophisticated agentic workflows. Adoption of VICBench in academic and industry evaluations may drive development of more robust vulnerability detection systems.