CRAFT · Jul 17, 2026
CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data
The CRAFT method transforms rubric-based evaluation datasets into model-specific weak capability diagnoses by extracting capability descriptions, clustering them into a hierarchical capability tree, scoring, and selecting low-scoring nodes to guide the generation of targeted supervised fine-tuning data. It was compared with prompt-level EvalTree clustering and random generation on four open-source models, two specialized domains (finance and law), and 13 held-out benchmarks.
What happened
Researchers propose the CRAFT method, which can transform any rubric-based evaluation dataset into model-specific weak capability diagnoses and generate targeted fine-tuning data. It was validated on four open-source models, finance and law domains, and 13 benchmarks.
Technical significance
CRAFT treats rubrics as capability probes, using hierarchical clustering to pinpoint specific capability nodes where the model fails, enabling fine-grained diagnosis.
Industry impact
This method is expected to improve model iteration efficiency in specialized domains (e.g., finance, law), allowing evaluation to directly drive training data generation.
What to watch
It remains to be seen how the method generalizes to more domains and larger models, and whether it will be adopted in mainstream model training pipelines.
Decision value
By automating weak capability diagnosis and training data generation, it reduces model iteration costs and accelerates vertical domain model optimization.