Tencent Hunyuan open-sources OmniCapBench audio-visual captioning evaluation framework
Tencent Hunyuan released OmniCapBench, an open-source evaluation framework for fine-grained audio-visual captioning, on GitHub. The repository includes an annotation pipeline and scoring toolkit. The source release was published on 2026-10-12.
China context
- Original name
- 腾讯混元
- Outside China
- Open weights · github.com
- Claims
- Company-reported; not yet independently evaluated
- For builders
- Developers can use the annotation pipeline and scoring toolkit to evaluate their own audio-visual captioning models with fine-grained, attributable metrics.
- For investors
- The release signals Tencent Hunyuan's investment in evaluation infrastructure for multimodal models, which may indicate a focus on improving model quality and benchmarking capabilities.
OmniCapBench reframes audio-visual caption evaluation as a deep-structured diagnostic task, scoring atomic evaluation units across entity references, visual shots, and audio events. It uses deterministic verification and localized semantic checking to provide attributable metrics. The framework was applied to 786 densely annotated videos (12.8 hours, 20 level-1 and 125 level-2 categories, yielding 5,818 references, 11,419 shots, 39,160 subshots, and 6,537 events).
The framework splits scoring into deterministic rule-based checks for schema, identifiers, time bounds, and cross-track links, followed by localized LLM judging on aligned field pairs. Metrics include SGC, RefUse, CCC, Subject F1, Scene F1, Shot F1/tIoU, Subshot F1/tIoU, Event F1/tIoU, Speaker F1, EVSA F1, and semantic R&P scores. No aggregate total score is reported, keeping every number attributable to a specific component.
Researchers and developers evaluating audio-visual captioning models gain a reproducible, open-source benchmark that localizes errors to specific units, reducing reliance on opaque global LLM judges and enabling more precise model diagnostics.
The open-source release provides a standardized evaluation toolkit that can lower the cost and effort of benchmarking audio-visual captioning models, potentially accelerating development and comparison of such models.
Adoption can be tracked by monitoring GitHub stars, forks, and issues on the OmniCapBench repository, as well as citations of the associated paper and usage in subsequent model evaluations.