From Confusion to Clarity: Confusion-Aware Retrieval and Knowledge Injection for Text Classification
A framework identifies confusable label pairs, expands candidate sets, and generates targeted rules to differentiate similar labels for LLM text classification. It requires no fine-tuning and transfers to smaller models. On WOS, Flipkart, and LEDGAR benchmarks, it improves Macro F1 by up to 10.0pp over retrieval baselines, with smaller models (2B-20B) gaining up to 11.5pp via cross-model transfer.
Large language models struggle to classify text into taxonomies with many semantically similar labels. A common approach retrieves top-K candidate labels by embedding similarity, but this does not help distinguish similar candidates. The proposed framework identifies confusable label pairs, expands candidates to include confusable labels, and generates targeted rules to differentiate them. It requires no fine-tuning and transfers to smaller models. On three benchmarks, it improves Macro F1 by up to 10.0pp over retrieval baselines, with smaller models gaining up to 11.5pp via cross-model transfer.
The method augments retrieval with confusion-aware candidate expansion and rule-based knowledge injection, enabling LLMs to resolve fine-grained label distinctions without fine-tuning. The generated rules are transferable across model sizes, suggesting a form of task-specific knowledge distillation.
This approach can reduce the need for expensive fine-tuning when deploying LLMs for large-scale text classification tasks, such as e-commerce product categorization or legal document classification, by improving accuracy with smaller, cheaper models.
Improves classification accuracy for large label spaces while enabling use of smaller, cheaper models, reducing inference costs and deployment complexity for text classification applications.
Potential next signals include open-source implementations, integration into LLM application frameworks, and benchmarks on additional domains with large label spaces. Adoption may be driven by enterprises seeking cost-effective classification solutions.