Vero is introduced as the first benchmark to evaluate joint implementation and proof synthesis at the repository level. It contains 43 multi-module instances sourced from real-world repositories spanning Python, Dafny, Verus, and Coq, covering domains from cryptographic protocols to distributed systems. Each instance consists of a multi-module Lean 4 repository with predetermined API interfaces, manually curated formal specifications, and reference implementations, supporting both proof-only and code-and-proof evaluation modes.
AI agents are increasingly used for programming but do not guarantee correctness of generated code. Verified code generation, where an agent produces both an implementation and a machine-checked proof of its specification, offers a stronger path toward trustworthy AI-generated software. Existing benchmarks focus on individual functions or only evaluate proof generation with provided implementations. Vero is the first benchmark to evaluate joint implementation and proof synthesis at the repository level, with 43 multi-module instances from real-world repositories in Python, Dafny, Verus, and Coq, covering diverse domains. Each instance includes a multi-module Lean 4 repository with predetermined API interfaces, manually curated formal specifications, and reference implementations, supporting proof-only and code-and-proof evaluation modes.
Vero shifts evaluation from single-function proof generation to repository-level joint implementation and proof synthesis. The benchmark uses Lean 4 as the target language for multi-module repositories, with instances sourced from Python, Dafny, Verus, and Coq. It supports two evaluation modes: proof-only (given implementation, generate proof) and code-and-proof (generate both). The inclusion of manually curated formal specifications and predetermined API interfaces ensures consistent evaluation across agents.
The benchmark addresses a gap in trustworthy AI-generated software by requiring agents to produce both code and machine-checked proofs. This could influence how enterprises adopt AI coding agents for safety-critical systems. The multi-language sourcing (Python, Dafny, Verus, Coq) suggests cross-ecosystem relevance, while Lean 4 as the target may indicate a growing role for proof assistants in AI toolchains.
For organizations requiring high-assurance software (e.g., aerospace, finance, cryptography), Vero provides a measurable way to assess AI agents' ability to produce formally verified code. This could reduce manual proof effort and accelerate adoption of AI in regulated domains. For AI vendors, strong Vero performance could become a differentiator in enterprise sales.
Observable next signals include: (1) whether leading AI coding agents (e.g., GitHub Copilot, Codex, Claude Code) are evaluated on Vero and publish results; (2) whether Vero is adopted as a standard benchmark in formal methods or AI conferences; (3) whether new agent architectures specifically target joint code-and-proof synthesis; (4) whether industry consortia or standards bodies reference Vero for certification of AI-generated code.