Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges
A paper introduces Graph-Structured Rubrics (GSR), which compiles a rubric into a response-independent typed evaluation graph before observing responses. Criterion nodes elicit judgments; transformation, reduction, and gating operators compose them through named ports; and a task-specific output mapping, termed Readout, converts the unique sink into a score or preference. Compilation rejects malformed or type-incompatible graphs. Pointwise evaluation judges rubric dimensions separately before graph aggregation; pairwise evaluation reuses the graph with one judgment for each candidate under every criterion. Under GPT-OSS-120B, GSR improves exact score agreement by 0.62--6.75 percentage points over Prometheus-style scoring on four pointwise datasets and achieves the numerically highest end-to-end pairwise accuracy on two preference benchmarks under native tie and abstention policies.
Graph-Structured Rubrics (GSR) compiles a rubric into a response-independent typed evaluation graph before observing responses. Criterion nodes elicit judgments; transformation, reduction, and gating operators compose them through named ports; and a task-specific output mapping, termed Readout, converts the unique sink into a score or preference. Compilation rejects malformed or type-incompatible graphs. Pointwise evaluation judges rubric dimensions separately before graph aggregation; pairwise evaluation reuses the graph with one judgment for each candidate under every criterion. Under GPT-OSS-120B, GSR improves exact score agreement by 0.62--6.75 percentage points over Prometheus-style scoring on four pointwise datasets and achieves the numerically highest end-to-end pairwise accuracy on two preference benchmarks under native tie and abstention policies.
GSR introduces a typed evaluation graph that makes criterion composition explicit and verifiable before response evaluation. The graph structure with named ports and operators enables modular, reusable rubric components and rejects malformed graphs, improving reliability. The reported improvements over Prometheus-style scoring suggest that structured composition can reduce ambiguity and improve judge consistency.
This approach addresses a known weakness in LLM-as-a-judge systems: rubric interpretation inconsistency. By formalizing rubric structure, GSR could enable more reliable automated evaluation pipelines, potentially reducing human review overhead in model development and benchmarking.
Improved evaluation reliability can lower the cost and time of model iteration and quality assurance. Organizations using LLM judges for content moderation, grading, or model benchmarking may benefit from more consistent and auditable evaluation processes.
If validated across more models and tasks, GSR could become a standard for rubric-based evaluation, influencing how evaluation datasets and judge models are designed. Adoption may depend on ease of rubric compilation and integration with existing evaluation frameworks.