RPCBench: A Benchmark for Proactive Premise Critique in LLM-based Recommendation
RPCBench is introduced as a benchmark for evaluating Recommender-Premise Critique: the ability of LLMs to detect, diagnose, and properly handle faulty premises in natural-language recommendation requests. It contains evidence-grounded test instances from five recommendation domains and covers ten types of premise failures. Each instance provides a visible recommendation context and a corrupted user query. A fine-grained evaluation framework measures proactive detection, error localization, post-detection handling strategy, and evidence faithfulness. Systematic evaluation of 11 LLMs finds that proactive detection is the main bottleneck in Recommender-Premise Critique.
RPCBench is a new benchmark designed to evaluate large language models' ability to critique flawed premises in recommendation requests. Unlike existing benchmarks that focus on ranking or generation quality, RPCBench tests whether models can detect, diagnose, and handle faulty premises in natural-language queries. It includes test instances from five recommendation domains, covering ten types of premise failures, each with a visible context and corrupted query. The evaluation framework measures proactive detection, error localization, handling strategy, and evidence faithfulness. Evaluation of 11 LLMs reveals that proactive detection is the primary bottleneck.
The benchmark's fine-grained evaluation framework separates proactive detection from error localization and handling, revealing that current LLMs struggle most with the initial detection of flawed premises. This suggests that models may generate plausible recommendations without recognizing invalid assumptions in user queries, indicating a gap in reasoning about user intent and evidence consistency.
As LLMs are increasingly deployed as interactive recommender assistants, the ability to critique user premises becomes critical for trust and safety. RPCBench provides a standardized way to measure this capability, which could influence model selection and development for recommendation systems in e-commerce, content platforms, and other domains.
For companies building LLM-based recommendation systems, RPCBench offers a tool to assess and improve model robustness against flawed user inputs, potentially reducing user dissatisfaction and improving recommendation quality. It may also serve as a differentiator in model procurement and benchmarking.
Future work may focus on improving proactive detection through targeted training or prompting strategies. The benchmark could be extended to more domains and premise failure types, and may become a standard evaluation for recommender LLMs. Observing whether model developers adopt RPCBench in their evaluation pipelines will be a key signal.