Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories
A study evaluates embedding retrieval where surface form and meaning are separated, using competition mathematics (MathNet-Retrieve; 500 queries, 117,088-item corpus) and embodied-agent trajectories (ALFWorld-derived; 118 queries, 336 trajectories). In mathematics, strict Hit@1 at the heaviest disguise tier is 0.0% for both production embedders, while the correct item is in the top 10 nearly always; in 95.2-99.8% of misses the winner is more lexically similar to the query than the correct answer. In trajectories, models land at or near hypergeometric chance when gold involves a different object, and below chance when gold differs in object and receptacle. A lexical reranker control hurts in mathematics and helps in trajectories (closing 26-36% of the gap).
The paper demonstrates that embedding retrieval models rely on surface lexical similarity rather than underlying structure, failing to retrieve structurally equivalent items in mathematics and agent trajectories. In mathematics, Hit@1 is 0% under heavy disguise despite correct items being in top 10, and in trajectories performance is at or below chance when object/receptacle differ. A lexical reranker helps in trajectories but hurts in mathematics, indicating benchmark surface variation type determines reranker effect.
The study isolates surface-form bias by constructing retrieval tasks where query and target share structure but not wording. Production embedders show near-total failure on disguised mathematical problems (Hit@1=0%) and chance-level performance on trajectory tasks requiring object/receptacle generalization. The differential effect of a lexical reranker (hurting in adversarial surface variation, helping in incidental variation) suggests current embedding spaces encode lexical similarity more strongly than structural or semantic equivalence.
This reveals a fundamental limitation in current embedding-based retrieval systems widely used in RAG, code search, and agent memory. Systems relying on embeddings may retrieve lexically similar but structurally irrelevant items, degrading performance in mathematical reasoning, planning, and embodied AI. The finding that rerankers can help or hurt depending on task surface variation implies that retrieval pipelines need task-specific tuning and evaluation beyond standard benchmarks.
For companies building retrieval-augmented systems, this highlights a risk of silent failures in domains requiring structural understanding (e.g., math tutoring, code generation, robotics). Improving retrieval robustness could enhance product reliability and user trust. The paper provides a diagnostic method to assess whether a retrieval system is vulnerable to surface-form bias, enabling better evaluation and potential differentiation.
Future work may focus on developing embedding models that better capture structural and semantic equivalence, possibly through contrastive learning with structural augmentations or hybrid retrieval combining symbolic and neural methods. The benchmark protocol (MathNet-Retrieve and ALFWorld-derived) could become a standard for evaluating retrieval robustness. Watch for follow-up studies on reranker behavior and new training objectives to mitigate surface-form bias.