SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG
SciRet evaluates a fixed scientific RAG pipeline on CORD-19 at three corpus scales (1K, 5K, 15K papers). The pipeline uses sentence-window chunking, BM25, BGE-M3 dense retrieval, reciprocal rank fusion, optional cross-encoder reranking, and grounded answer generation. Hybrid retrieval achieves Recall@10 of 1.000 at 1K and 15K scales. An MS MARCO-trained cross-encoder reranker reduces precision on the scientific corpus. Generation faithfulness (RAGAS) increases with corpus scale. Retrieval evaluation uses pseudo-relevance labels from the hybrid system.
The SciRet study systematically evaluates a retrieval-augmented generation (RAG) pipeline for scientific question answering on the CORD-19 dataset. It tests three corpus sizes—1,034, 5,160, and 15,480 chunks—using a fixed pipeline of sentence-window chunking, BM25 sparse retrieval, BGE-M3 dense retrieval, reciprocal rank fusion, optional cross-encoder reranking, and grounded answer generation. Hybrid retrieval (BM25 + BGE-M3) proves more robust than sparse-only or dense-only, achieving perfect Recall@10 at both the smallest and largest scales. However, a cross-encoder reranker trained on MS MARCO reduces precision, indicating domain mismatch can hurt performance. Faithfulness of generated answers, measured by RAGAS, improves as corpus size grows. The study uses pseudo-relevance labels from the hybrid system for retrieval evaluation, so results are comparative rather than absolute benchmarks. Code, indexes, and outputs are released for replication.
Hybrid retrieval (BM25 + BGE-M3 with reciprocal rank fusion) consistently outperforms sparse-only and dense-only methods across corpus scales, achieving Recall@10 of 1.000 at both 1K and 15K chunks. The failure of an MS MARCO-trained cross-encoder reranker highlights the critical impact of domain mismatch in scientific RAG, where general-domain rerankers can degrade precision. Generation faithfulness (RAGAS) improves with larger corpora, suggesting that more context can enhance grounded answer quality. The use of pseudo-relevance labels from the hybrid system for evaluation means the reported metrics are internally consistent but not externally comparable.
This study provides practical guidance for building scientific RAG systems: hybrid retrieval is a safe default, and off-the-shelf rerankers may need domain adaptation. The observed improvement in faithfulness with corpus scale suggests that scaling up document collections can benefit answer quality, but careful evaluation is needed due to the use of pseudo-labels. The release of code and indexes lowers the barrier for replication and adaptation in scientific and industrial settings.
Organizations building scientific Q&A systems can adopt the SciRet pipeline as a baseline, leveraging hybrid retrieval for robust performance. The finding that general-domain rerankers can hurt precision underscores the need for domain-specific components, potentially creating opportunities for specialized reranker training services. The scalability of faithfulness with corpus size may encourage investment in larger document collections for improved answer quality.
Future work may explore domain-adaptive reranking, such as fine-tuning cross-encoders on scientific data, to mitigate the observed precision drop. The relationship between corpus scale and faithfulness warrants further investigation with human-annotated labels. The methodology could be extended to other scientific domains and larger corpora, and the impact of different chunking strategies and fusion methods could be systematically compared.