Silver Eval Datasets Without the Bulk-Generation Trap

Every team that ships an agent over a real corpus hits the same wall. The gold dataset, query-answer pairs written and checked by people who know the domain, is the baseline you actually trust for regression. It is also the slowest thing in the pipeline to build. Specialist labeling does not scale with the corpus, so coverage stays small, dozens of cases where you want thousands, and every corner of the product you never wrote cases for is a corner you ship blind.

The cheap fix is well known, so everyone tries it. Prompt a model to generate a few thousand question-answer pairs, run the agent over them, and call the pass rate an evaluation. The dashboard goes green, the number moves when you change the prompt, and it feels like the coverage problem is solved. Then real users arrive and break the agent in the first week, on questions that look nothing like the generated set.

The failure is not that the model did a bad job. It is that the bulk prompt asks for the wrong thing. The method that works conditions generation on three things instead. Personas define who is asking. Scenarios define the situation they are in. The feature being shipped defines the slice of the corpus and the dataset the cases belong to. And the personas and scenarios are not invented from thin air. They are derived from the gold set, the same trusted baseline the rest of the pipeline already leans on. This post is the how, with the prompts and the validation loop included.

The bulk-generation trap

Ask a model for fifty questions about a microfinance institution’s late-payment rules and you get questions that read like a textbook quiz. Well-formed, complete sentences, correct terminology, one topic per question, no typos. They all resemble each other because they come from the same place, the model’s internal picture of what a question about late-payment rules sounds like. That picture is an average. Sample it a thousand times and you get a thousand variations of the same question wearing different words.

Real query distributions look nothing like that. They are wide and lopsided. People ask with half the context missing, with the wrong vocabulary, mid-task under time pressure, in fragments, with typos, in their second language, presupposing things the system cannot know. The bulk prompt samples the mode of an imagined distribution. Production usage lives in the tails. A regression score computed on bulk data measures how well the agent handles the model’s idea of a user, and no user has ever been that.

The cheapest way to see the problem is to filter the output. Run a near-duplicate pass over ten thousand generated pairs by token overlap and watch how much of the set survives. In my experience it is a small fraction, because the generator keeps re-arriving at the same handful of shapes. If one bulk prompt were actually enough, nobody would still be paying labelers.

Let the gold set name its askers

Where does the cast come from? Writing personas from nothing repeats the bulk-generation mistake one level up, because the model’s internal picture of “a user” is the same average the bulk prompt samples. The silver set has to derive at least something from the gold set, and the first thing to derive is the people already implied by it.

Every gold query was written or checked by someone who knows the domain, and each one implies an asker. So put the gold set in front of a model and ask the first question. Who could plausibly be sitting behind the chatbot sending these? For a microfinance assistant the answers come back as a market vendor checking a weekly payment, a rice farmer asking what counts as collateral, a first-time borrower confused by the guarantee rules, a branch clerk looking up a customer’s status. None of these is invented. Each is a person the gold queries already point at.

Then ask the second question for each candidate. In which specific situation would this person send exactly these queries? The vendor asks about penalties the week the market flooded and she paid part of her installment in cash. The farmer asks about collateral the morning the group leader told him his land papers might not qualify. The answers are a rough draft of the persona and scenario lists the silver generation will run on, grounded in the distribution gold captured instead of the model’s average user.

Here is the gold query set for this product:
{gold_queries}

Two questions, in order.
1. Who could plausibly be sitting behind the chatbot sending
   these? List the distinct kinds of people, a short paragraph
   each.
2. For each of them, in which specific situation would they send
   exactly these queries?

Then extrapolate, because gold is small and the people inside it are not the whole population. Take the inferred cast and ask who interacts with these people around the same product yet never appears in the gold queries.

• The guarantor who covers a group member’s missed payment and wants to know what it does to his own record.
• The son working in another province who tops up his mother’s loan through an app he set up for her and left her to use alone.
• The group leader who collects five members’ payments and asks about all of them in one conversation.

These adjacent people and situations are extrapolations, and that is fine. They stay anchored to askers the gold set implies, which keeps them near real usage while widening the cast beyond what gold alone covers. Diversity you can argue for is diversity with an anchor.

Who asks before what gets asked

The derivation hands you a cast list. A persona sheet turns each name on it into someone a model can actually simulate. A persona is a short answer sheet about one kind of user, and every field on it exists to push generation away from the average query and toward the way some real person actually asks.

# personas/market-vendor.yaml
name: market-vendor
role: runs a vegetable stall, has borrowed working capital twice before
seniority: 7 years selling in the same market, keeps no books
vocabulary: says "the weekly payment" and "that money app", never
  "installment" or "digital wallet"
cares_about: whether skipping one week puts her on the penalty list,
  not the interest formula behind it
does_not_know: that group-guarantee and individual loans follow
  different late-payment rules, that the branch ledger and the app
  update on different days
typing_style: dictated on a phone, numbers as digits mid-sentence,
  no punctuation, two short messages instead of one long one

Each field earns its place. vocabulary keeps the generated queries off the polished register, because the generator writes the question with the persona’s words, not the documentation’s. cares_about shapes what a good answer looks like for this person, a direct yes or no about the penalty list beats a paragraph about the interest formula. typing_style carries the messiness deliberately, dictation artifacts, digits scattered through sentences, fragments, the occasional word in another language spilling through. An agent trained and tested only on clean prose meets its first dictated fragment in production.

does_not_know is the underrated field. Questions that presuppose something wrong are everywhere in production and almost absent from bulk-generated data, because a model asked to invent questions invents competent askers. A persona who does not know that the branch ledger and the app update on different days generates the confused “i paid already why does it still say 420” message your real users send, and that is exactly the case you want in the set before it reaches production.

How many personas per product? I would suggest four to eight, adjusted to the diversity the use case needs to cover. A feature with one professional audience sits near the low end. A consumer product spanning markets, literacy levels, and languages drifts toward the high end. Whatever the count, each one stays a real person with a name and a mood rather than a demographic bucket.

Personas need situations

A persona alone still answers in a vacuum. Generate for the vendor without a situation and you get generic loan questions wearing a costume. What makes a query production-shaped is everything happening around it.

# scenarios/missed-payment-after-flood.yaml
situation: the market flooded, the vendor spent this week's payment money
  on restocking, and she paid part of the installment in cash at the
  branch two days ago
pressure: the group meeting is in two days, and the other four members'
  guarantees are linked to hers
context_missing: she cannot tell whether the cash payment reached her
  balance, and the app still shows the old amount
just_happened: her daughter installed the app for her last month and
  then left for another province, so nobody remembers the PIN
expected_shape: two or three short turns, worry in the second turn if
  the first reply quotes policy instead of checking the balance

The scenario fields add the three things a bare query never has. Time pressure changes what a correct answer is, because under pressure the right reply is “do this now, details later”. Missing context makes the right agent move a clarifying question instead of an answer, which is a behavior worth testing on purpose. just_happened manufactures the multi-turn and confusion cases, where the user’s problem is partly yesterday’s unfinished task, and the agent has to untangle that before anything else.

Then the multiplication does the real work. Five personas against ten scenarios is fifty situation seeds, and each seed can yield several exchanges. Coverage grows with the product of the two lists instead of the sum. That combinatorial spread is the volume that bulk generation only pretends to provide.

One dataset per feature

The last move changes the shape of the whole collection. Do not generate one dataset for the agent. Generate one per feature being shipped or evaluated. Loan products get their own set, repayment channels their own, identity checks their own, each grounded in that feature’s slice of the product corpus.

Three things follow. Every feature has eval coverage from the day it ships, because the dataset is built alongside it instead of after it. When a release regresses, the failing cases name the feature, and triage starts at the right door. And dataset design moves into the feature spec, where the engineers and the domain people are already in the room, instead of happening after ship when nobody remembers the edge cases.

The questions you answer building the set are product questions. What does this feature have to handle, asked by whom, in what situation. If you cannot write the persona list and the scenario list for a feature, you do not yet know what the feature is, and the dataset review will surface that faster than the retro will.

The generation prompt

Everything above feeds one prompt. The inputs are the persona sheet, the scenario card, and the source documents for the feature slice. The output contract is structured, and three of its rules do the heavy lifting.

You are simulating one user, not writing exam questions.

Persona, stay in character including typing style:
{persona_sheet}

Situation:
{scenario_card}

Source documents for this feature:
{feature_docs}

Generate {n} exchanges this user would have with the assistant in
this situation. Return JSON with, per exchange:
- turns: the user's messages, in the persona's own words and typing
  style
- expected_answer: what a correct, helpful reply looks like
- grounded_in: the doc ids the expected answer rests on
- unanswerable: true when the documents do not cover the case, with
  expected_answer saying what the assistant should do instead of
  guessing

Rules:
- The user's messages never use vocabulary the persona does not have.
- The expected answer uses only the provided documents.
- If the documents do not answer it, mark it unanswerable. A refusal
  case is a valid case.

The typing-style rule needs enforcement, not just stating. Left alone, generation slides back toward clean prose within a batch, so spot-check every batch for register drift and regenerate the too-polished exchanges. The grounding rule makes each pair carry its own citation, which is what lets the validation loop check answers without a human rereading the whole corpus. The unanswerable flag turns the generator into a source of negative cases too, and refusal behavior is exactly what you want under test, because the worst failure mode is a confident answer where the honest move was “the documents do not say”.

One more lever. Run the same seeds through more than one model family when you generate. Each family has its own prior about how users talk, so one family asked for a market vendor keeps writing the same vendor. The union of two families covers more ground than either alone, and it keeps the set from inheriting one vendor’s blind spots. That last part matters more than it sounds. The model inside the agent you are testing is often one of the families you generated with, and a silver set that shares its blind spots will never catch the failures you built the set to find.

Keeping the silver honest

The obvious objection, and it is fair. The whole set was written by models, so why trust it more than the agent it tests? The answer is that you do not trust it. You gate it. Silver pairs are candidates until they clear a loop.

• Near-duplicate removal by token overlap. Anything too close to a kept case is dropped, which forces spread at the cost of raw volume.
• A diversity report before acceptance. Distinct unigram and bigram ratios per feature set, compared against the previous batch. A falling ratio means the generator is collapsing again, even though the case count keeps rising.
• Citation checks on every pair. The case says which documents ground it, so a cheap automated pass verifies the cited documents exist and actually contain the answer’s key terms.
• A human reads a random slice before any batch enters the regression set. A domain person, twenty cases, a few minutes. If the slice is wrong, the batch is wrong, and the batch dies.
• Calibration against gold. Run the agent on gold and on silver for the same feature. If gold holds while silver tanks, or the reverse, one of the two sets is lying about the release, and it is usually the silver.

After the gate, freeze the batch. A silver set that silently grows is a metric that silently changes. Add cases when a feature grows, version the set alongside the code, and rerun the whole thing on every release candidate. The set is an asset with a changelog, not a log file.

Where silver sits

Silver is one of three assets, and each has one job. Gold is small, trusted, and slow, and it anchors every disagreement. Silver is broad, cheap, frozen per feature, derived from the askers gold implies, and exactly as good as its last review. A live judge over production traffic catches drift and novel failure modes continuously, though judging has its own biases and belongs to its own discussion. The scoring side, which deterministic metric to run on which pair and when an LLM judge is acceptable, is a topic I covered in a previous post on evaluating agent RAG systems.

The three compose. Gold tells you whether to believe the dashboard. Silver tells you where the agent breaks across the feature surface. The live judge tells you when production disagrees with both.

The whole loop, end to end

Two derivation prompts turn the gold set into a cast and its situations. The extrapolation prompt widens the cast with the people gold never saw. Each persona sheet crossed with each scenario card becomes a seed, and the generation prompt turns one seed and one feature’s documents into candidate pairs. The gate described above does the rest, near-duplicate removal, a diversity report, a citation check, a human slice, calibration against gold. What survives is frozen per feature and versioned with the code.

The eval set stops being a folder of questions somebody once thought of and becomes a designed artifact whose coverage you can actually argue for, person by person, situation by situation, feature by feature. The agent under test faces a crowd instead of a mirror, and the crowd never runs out of tight deadlines, missing context, and ways to ask the right question wrong.