Define failure in context
A summary can be fluent and still omit the one fact that changes a decision. A generated action can look plausible and still violate a permission boundary. Evaluation must reflect the consequences of being wrong.
I would build a small set of representative examples, edge cases, and known failures before debating models or prompts.
Combine signals
Automated checks can measure structure, required fields, citations, and prohibited content. Human review is still important for usefulness, tone, and ambiguous cases. Production feedback completes the loop.
The goal is not to prove that a model is perfect. It is to know when it is safe enough for a particular task and when to fall back.
Build an evaluation set from reality
Start with successful examples, ambiguous inputs, adversarial requests, long context, missing context, and every production failure you can collect safely. Keep the expected behavior close to the product requirement rather than judging only whether the wording sounds good.
flowchart TD
A[Representative task set] --> B[Deterministic checks]
A --> C[Human review rubric]
A --> D[Model comparison]
B --> E[Release gate]
C --> E
D --> E
E --> F[Production feedback]
F --> A
Test the contract around the model
For extraction, test required fields, valid enums, and source references. For tool use, test that invalid arguments are rejected and that permissions are enforced. For generated text, test omissions and harmful confidence, not only grammar.
it("does not execute an invalid inventory action", async () => {
const response = await interpret("move an impossible quantity");
expect(ActionSchema.safeParse(response).success).toBe(false);
expect(domainCommands.execute).not.toHaveBeenCalled();
});
Watch for regression
A prompt or model change can improve the average while breaking an important edge case. Keep a small non-negotiable set that must never regress, report results by task category, and record the model version with every evaluation. A score without slices hides the failure that matters.