Tencent Hunyuan open-sources ORAV-Bench for benchmarking audio-video generation from multimodal contexts
Tencent Hunyuan released ORAV-Bench, an open-source benchmark for omni reference audio-video generation. It includes 380 tasks with 2–10 references per task and 9 semantic roles. The repository provides an evaluation pipeline with a judge achieving 86.08% effective agreement with human judgments.
China context
- Original name
- 腾讯混元
- Outside China
- Open weights · github.com
- Claims
- Company-reported; not yet independently evaluated
- For builders
- Developers can integrate ORAV-Bench into their evaluation workflows for audio-video generation models, using the provided pipeline and judge.
- For investors
- Tencent Hunyuan's release of an open-source benchmark indicates strategic investment in multimodal AI evaluation, which may influence industry standards.
Tencent Hunyuan published ORAV-Bench on GitHub under Apache-2.0. The benchmark evaluates audio-video generation from multimodal contexts, where an instruction and heterogeneous image, video, and audio references produce one coherent audio-video event. The repository includes an evaluation pipeline with reference-aware pairwise judging, system ranking, and pointwise diagnostics. The judge achieves 86.08% effective agreement with human judgments on a 10% held-out subset. Benchmark tasks and model generations are released separately as ORAV-Bench and ORAV-Bench-Artifacts.
ORAV-Bench uses a visually blind Listener to provide auditory evidence and a Judge that compares this evidence with timestamped visual panels in both presentation orders. The evaluation pipeline supports Python 3.11 or 3.12, FFmpeg 5.1+, and configurable evaluator models via Responses, Chat Completions, or Gemini APIs. The benchmark includes 30 role compositions across 9 semantic roles.
Developers and researchers outside China can now use a standardized, open-source benchmark to evaluate audio-video generation models, reducing the cost of building custom evaluation pipelines. This gives Tencent Hunyuan a position in the emerging multimodal generation evaluation space, potentially influencing how such models are compared and improved.
For developers, ORAV-Bench provides a ready-to-use evaluation framework, saving time and resources in assessing audio-video generation models. For investors, it signals Tencent Hunyuan's investment in multimodal AI infrastructure and potential influence over evaluation standards.
The next observable signal is whether other model providers adopt ORAV-Bench for reporting audio-video generation results, and whether the benchmark gains traction in academic or industry evaluations. The repository's contribution mechanism for model outputs may lead to a growing set of comparable results.