Can Open Large Language Models Catch Vulnerabilities? Open-source LLMs Strong at Detection but Weak at Classification, Cautious Deployment Needed
In January 2025, a study systematically evaluated the vulnerability detection and classification capabilities of Llama3, Codestral, and Deepseek R1 on the Big-Vul dataset. Results showed that these models perform well in detecting vulnerabilities but have low accuracy when classifying by CWE standards, exhibiting overgeneralization and misclassification issues. The study revealed model-specific biases and common failure modes, emphasizing the need to address these limitations before deployment in security-sensitive environments.
This study reveals the dual nature of open-source LLMs in the security domain: strong detection but weak classification. This means models may misreport or miss specific types of vulnerabilities, leading to misjudgment by security teams. For enterprises relying on LLMs for code auditing, this finding warns against fully trusting model classification results and necessitates human review. The study also points out potential risks of using LLMs as learning tools in educational settings.
The study used a subset of the Big-Vul dataset containing 8 CWE categories, evaluated under a closed-world classification setting. Models performed well in detection (binary classification) but had low accuracy in multi-classification (mapping to correct CWE). Analysis showed models tend to overgeneralize (e.g., grouping multiple vulnerabilities into the same CWE) and misclassify (e.g., misidentifying XSS as SQL injection). Model-specific biases stem from training data distribution and architectural differences. Limitations: the dataset covers only 8 CWEs and is in English, while real-world scenarios are more diverse.
For the security industry, this study reminds that LLM-driven code audit tools need cautious deployment, especially automatic classification features. Security vendors should develop hybrid systems combining LLM detection with traditional rule/ expert system classification. In education, LLMs as programming assistants may propagate incorrect security knowledge, requiring auxiliary materials to correct biases.
Recommend that security teams using LLM code audit tools treat them only as auxiliary detection, with classification results requiring human verification. Investment opportunities lie in companies developing hybrid security analysis platforms. Engineering-wise, domain-specific fine-tuning of LLMs can improve classification accuracy.
Areas to focus on: 1) Methods to improve LLM classification capabilities, such as fine-tuning or retrieval augmentation; 2) More comprehensive benchmarks covering more CWEs and real-world vulnerabilities; 3) Model robustness under adversarial attacks; 4) How security tool vendors integrate these findings.