Testing with evals
Write cases, choose a scoring method and a threshold, and compare two versions of a prompt with numbers. Who runs the model in the pre-beta.
An eval answers one question: is this version of the prompt better than that one?
Anatomy of an eval#
- Cases: a list of inputs, each with an id and what you expect.
- A method that scores an answer from 0 to 1.
- A threshold from 0 to 1. The default is 0.8.
- The target: a prompt, workflow or agent.
{
"entity_type": "prompt",
"entity_id": "<id>",
"title": "Reply stays within the rules",
"method": "contains",
"threshold": 0.8,
"cases": [
{ "id": "angry-refund", "input": "I want my money back NOW. Order A-1043.", "expected": "next steps" },
{ "id": "not-found", "input": "Where is order Z-9?", "expected": "could not find" }
]
}The five methods#
| Method | A case passes when |
|---|---|
exact | the answer equals what you expect |
contains | the answer includes what you expect |
structural | the answer has the shape you expect, such as valid JSON with certain keys |
judge | a model reads the answer and scores it against your expectation |
hybrid | a mix of rules and a judge |
Writing good cases#
- Start with the cases that hurt: the angry customer, the missing order, the odd input.
- Ten good cases beat a hundred similar ones.
- Add a case whenever a bug teaches you something.
Who runs the model#
Ask your AI something like:
Use Prompt Verse. Get my eval "Reply stays within the rules". For each case render my prompt "Support reply" with the case input, answer it yourself, and score each answer from 0 to 1 against the expected text. Give me the mean and say whether it beats 0.8.Compare versions#
Run the same cases on two commits and compare the means. Use the result in your commit message when you merge.