Generating Leakage-Free Benchmarks for Robust RAG Evaluation

Researchers have introduced SeedRG, a novel semi-synthetic benchmark generation pipeline designed to address critical flaws in evaluating Retrieval-Augmented Generation (RAG) systems. Current benchmarks often suffer from 'knowledge leakage,' where questions can be answered using a Large Language Model's internal parametric memory rather than retrieved external knowledge. This issue is exacerbated by 'benchmark aging,' as repeated use of datasets in training causes models to memorize answers, rendering evaluations unreliable. SeedRG mitigates this by extracting reasoning graphs from existing question-context pairs and generating new, structurally similar examples through type-constrained entity replacement. This process creates novel instances unlikely to exist in model parameters while preserving original reasoning patterns. To ensure high quality, the pipeline includes two verification steps: a reasoning-graph consistency check to maintain task difficulty and a knowledge-leakage filter to exclude instances solvable without retrieval. This approach aims to provide more robust and accurate assessments of RAG performance by ensuring that retrieval mechanisms are genuinely tested.