From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
A research paper presents the 'Grip on LLMs' framework, a systematic evaluation suite for Dutch governmental use developed with a major Dutch municipal organisation. Six evaluation dimensions are identified: factuality, honesty, social bias, energy consumption, cost, and training data transparency. The benchmark covers over 30 multilingual and Dutch-specific models. Results show no single model excels across all dimensions; higher quality increases environmental impact and cost, while bias is largely independent. Factuality and honesty are governed by distinct properties.
The 'Grip on LLMs' framework evaluates large language models for Dutch governmental use, identifying six key dimensions and benchmarking over 30 models. Findings reveal trade-offs between quality, cost, and environmental impact, with bias remaining independent. Factuality and honesty are distinct properties, highlighting the complexity of deploying LLMs in public administration.
The framework operationalizes public administration values into measurable benchmarks, revealing that factuality and honesty are not correlated, suggesting separate model capabilities. The independence of bias from quality and cost indicates that debiasing efforts may not compromise other performance metrics.
Government adoption of LLMs requires balancing accuracy, transparency, and sustainability. The trade-offs identified imply that procurement decisions must prioritize dimensions based on use case, and no single vendor currently offers a universally optimal solution.
The framework provides a structured evaluation method for government LLM procurement, enabling cost-benefit analysis across multiple dimensions. It highlights market opportunities for models that balance factuality, honesty, and low environmental impact.
Next signals include potential adoption of the framework by other Dutch governmental bodies, development of similar benchmarks for other languages, and model providers optimizing for specific dimensions like energy efficiency or bias reduction in response to public sector demands.