Stop Sampling RAG at Random: Map the Difficult Corpus First

Random sampling is often a convenient substitute for an evaluation strategy. It is also a poor fit for retrieval-augmented generation systems whose source corpora differ materially in structure. A clean web article, an image-heavy PDF, an EPUB and a structured record can fail in different ways before a model writes a single word. If those differences alter extraction, chunking, retrieval or grounding, an undifferentiated sample spends test capacity without deliberately covering the conditions that matter.

A September 8 implementation case co-authored by KDDI and Google Cloud offers a more practical alternative. While preparing Buffmee, a consumer RAG application grounded in more than 100 sources including books, magazines and web media, KDDI organized evaluation examples in a two-dimensional grid. One axis was file format: web articles, EPUBs, PDFs and structured data. The other was media composition: text-heavy, image-heavy and mixed. The team selected representative examples from each resulting cell.

KDDI reports that this approach reduced evaluation workload by 75% while maintaining comprehensive test coverage. That figure is a vendor-published result from one implementation, not a universal benchmark: the post does not disclose sample counts or independently validate its coverage claim. But the design principle is stronger than the percentage. RAG teams should allocate finite evaluation capacity against a declared map of the input space, not distribute it uniformly or leave it to ad hoc testing.

The corpus is part of the product surface

Teams commonly define RAG evaluation around questions and expected answers. That remains necessary, but it is incomplete. The same question can produce different outcomes depending on the kind of source that must be found and interpreted. A scanned or image-heavy document may expose extraction weaknesses. Dense structured data may challenge chunking or retrieval logic differently from prose. Mixed-media material can create a gap between what a reader sees and what the retrieval pipeline has represented.

The useful unit of planning is therefore not simply the test prompt or the document. It is the combination of content characteristics that plausibly changes system behavior. KDDI’s grid makes that combination explicit. It turns a varied corpus into evaluation strata: defined groups from which a team can select cases intentionally rather than hoping a random set happens to include them.

The exact axes should not be copied mechanically. File format and media composition suited KDDI’s corpus. Another product may need to separate documents by scan quality, table density, language, version volatility, access restrictions, source authority or retrieval frequency. The standard is concrete: choose characteristics with a credible path to different retrieval or generation performance. A category that cannot change behavior is administrative metadata, not an evaluation stratum.

Build a grid that can be operated

A difficulty grid is useful only if it changes the release routine. Start by inventorying the corpus and assigning each source, or a practical content unit within it, to the chosen axes. List every occupied cell. Empty cells do not require invented tests; they record combinations the product does not currently serve. Occupied cells require named representative cases and an owner responsible for keeping them current as the corpus changes.

1. Select axes from observed content properties that can alter extraction, retrieval, interpretation or answer grounding.
2. Maintain representative test cases for every occupied cell, including cases that historically produced failures or user complaints.
3. Mark critical requirements that demand an unambiguous pass or fail, rather than relying on an averaged quality score.
4. Review the grid whenever new source types, ingestion paths or content patterns enter the product.

This is not a claim that a small test set proves total coverage. No sample can do that on its own. It is a way to make the coverage argument inspectable. A product owner can ask which parts of the corpus are represented, which are not, and why. That is a materially better operating position than reporting a single evaluation score from a sample whose composition nobody can explain.

Automate the checks that have a hard edge

KDDI constructed hundreds of automated tests and used binary pass/fail evaluation for selected critical metrics. The distinction matters. Critical requirements often have a hard operational boundary: an answer cites an appropriate source or it does not; a required response component is present or absent; a request completes within a defined condition or misses it. These are well suited to explicit rubrics analogous to software unit tests.

Not every quality dimension has that shape. Helpfulness, tone and the adequacy of an explanation can require judgment. KDDI retained human product judgment to calibrate automated scoring thresholds. That is the right division of labor: use automation to make repeated checks cheap and consistent, then use sampled human review to test whether the automated measure still corresponds to product quality. A threshold is not validated because it exists; it is validated when people compare it with the cases it classifies.

The case reports a 25% improvement in groundedness scores. It does not state the baseline or whether the figure is absolute or relative, so it should not be read as a percentage-point claim. Nor does improved groundedness establish hallucination-free behavior. The operating value is narrower and more useful: the team created a repeatable evaluation practice that could detect and guide quality work across its declared corpus strata.

Do not confuse evaluation design with latency work

KDDI’s performance results belong to a separate, supporting track. Production-log analysis identified skill division, sub-agent routing and prompt bloat as contributors to time to first token. The team split a system prompt exceeding 800 lines into function-specific ADK Skills, loading only the logic required for a request. KDDI reports a 38% reduction in total application response latency and a nearly 18% improvement in time to first token after this performance work.

Those gains should not be attributed to the difficulty grid. The grid concentrates quality evaluation; production analysis and prompt or routing changes address execution cost and delay. They do reinforce one another operationally. Once a team has representative cases across its hard content conditions, it can assess whether a latency optimization preserves behavior where the product is most likely to break. A faster path that has only been checked against easy text is not a production result.

The managerial question is where testing effort goes

The central lesson from KDDI is not that every RAG product needs a two-by-two matrix, nor that 75% less work should be expected elsewhere. It is that heterogeneous content creates an uneven risk landscape. Testing every document is usually impractical. Randomly testing documents treats that landscape as flat. A difficulty grid states where the terrain changes and ensures the test suite follows it.

“Do not ask whether the RAG system has been evaluated. Ask which content conditions its evaluation can actually see.

For product leaders, this changes the conversation from test volume to test allocation. Before adding more automated cases, identify the corpus characteristics that most change behavior. Make each occupied combination visible. Put durable representative cases in the release suite. Apply binary checks where failure has a clear edge, and keep human judgment in the loop where the measure needs calibration. The result is not certainty. It is a deliberate, maintainable claim about what the product has been tested to handle—and what it has not.