Build an AI Evaluation Dataset from Real Product Tasks

Define the task and output criteria

Start from one product task and write what a useful result must contain, omit, or qualify. Avoid scoring only general fluency. A meeting summary may sound polished while missing a decision or inventing an owner. Criteria should connect to the user’s next action, including the consequences of incorrect information. Establish them before comparing output variants.

Imagine a hypothetical knowledge assistant for a workshop organizer. An evaluation task asks whether a session can accommodate a specific equipment request. A good answer identifies the relevant session policy, states unsupported details as unknown, and directs the organizer to clarification when needed. It should not make a confident promise merely because the request resembles a previous example.

Cover meaningful variation and difficult cases

Collect or author examples representing the input conditions the feature will encounter. Include short and long requests, ambiguous names, outdated material, missing context, and unrelated text. Use permitted data and remove unnecessary personal information. Organize cases by the failure mode or context they reveal, rather than assembling a large collection without a clear purpose.

For the workshop assistant, include a request where two sessions share a similar name and a policy document that applies only to one venue. Add a case where the needed fact is absent. These examples test distinction and restraint. Synthetic scenarios can help explore gaps, but keep their origin visible and do not assume they represent the distribution of actual usage.

Create reliable review and held-out material

Write a rubric with concrete examples of acceptable and unacceptable outputs. Some tasks support exact checks, while others require a reviewer to assess support, completeness, or usefulness. Have reviewers compare interpretations of ambiguous cases before relying on their scores. Disagreement may reveal a weak criterion rather than a weak system.

Keep a portion of the dataset separate from routine tuning. Repeatedly adapting instructions to every known case can improve the visible score without improving general behavior. Record which examples were used during development. Where the desired answer changes because the product rule changed, update the criterion and explain the change instead of silently making historical comparisons invalid.

Use results to make bounded decisions

Review aggregate results alongside individual consequential failures. A broad average can hide a small category where behavior is unacceptable. Define release criteria for the task and document remaining uncertainty. Compare changes on the same evaluation material when possible, while periodically adding new relevant cases through a controlled process.

In the workshop example, a variant that produces fewer unsupported venue promises may be preferable even if its answers are slightly less concise. Preserve representative failures for regression review. After release, use appropriate feedback to identify missing cases without automatically storing sensitive user inputs. Evaluation is a maintained product asset, not a one-time collection created only to approve launch.

• Define task-specific acceptance criteria.
• Include missing-context and refusal cases.
• Separate tuning and held-out examples.
• Review consequential failures alongside averages.

Evaluation should also test the people reviewing drafts. Give them inspectable evidence and practical editing tools, then check whether consequential mistakes are actually caught.Design Human Review That Actually Helps an AI Workflow