The first prompt in a product feature is easy to write. The sixth edit is the dangerous one: someone fixes a complaint by adding a sentence, the new wording quietly breaks a case that worked last week, and nobody finds out until a customer does. Prompt evaluation is how you find out first, and prompt versioning is what makes the comparison possible.
This post builds a small harness for both: prompts stored as versioned files, an eval set of real checks, a runner that scores a prompt version, and a release gate that fails when something that used to pass now fails. It’s written in plain Node.js with no dependencies, and it’s about the engineering around a prompt, not about picking a model. For the wider question of using AI responsibly in a dev team, see my guide to AI-assisted software development.
What this post covers
- Tested versions and what the stub model is
- The feature: summarizing a support ticket
- Storing prompts
- The eval set
- Running an eval
- Comparing versions and gating a release
- What the three versions scored
- The tests
- Swapping in a real model
- What I did not test
- FAQ
Tested versions and what the stub model is
| Tool | Version |
|---|---|
| Node.js | v24.22.1, using the built-in test runner |
| Dependencies | None |
| Tests | test-prompts.mjs: 21 tests, all passing |
| Model | A deterministic stub (stubModel.js), no real LLM |
I want to be upfront about that last row. The tests never call a language model. stubModel.js is a small function that obeys a few instructions it finds in the prompt text, such as a word limit or “do not include the email address”. That lets the harness be tested for free, offline and deterministically. It also means the scores below say nothing about how well any real model follows these prompts. What they demonstrate is how the harness behaves.
The feature: summarizing a support ticket
The example feature is a one-sentence summary of a support ticket, shown to an agent above the thread. The tickets and customer names in the eval set are invented. The requirements are the kind that a product owner would actually write down:
- the summary is one sentence of at most 25 words
- it includes the order number when the ticket has one
- it never includes the customer’s email address
- it leads with the problem, not the customer’s name
Each of those becomes a check.
Storing prompts
Prompts live in the repository as one JSON file per version, next to the code that uses them: prompts/summarize-ticket/v1.json, v2.json, and so on. A version is never edited after it ships. A change is a new file, which gives you a diff in review and a stable name to reference in logs.
{
"name": "summarize-ticket",
"version": 2,
"variables": ["ticket"],
"template": "Summarize this support ticket in one sentence of at most 25 words. Include the order number if there is one. Do not include the customer's email address.\n\nTicket:\n{{ticket}}"
}
The loader checks each file for consistency and adds a short content hash. The hash means that two runs can prove they used the same text, even if a version number was reused by mistake.
export async function loadPrompt(dir, name, version) {
const file = path.join(dir, name, `v${version}.json`);
const prompt = JSON.parse(await readFile(file, "utf8"));
if (prompt.name !== name || prompt.version !== version) {
throw new Error(`${file}: name/version inside the file do not match its path`);
}
const used = placeholdersIn(prompt.template).sort();
const declared = [...prompt.variables].sort();
if (JSON.stringify(used) !== JSON.stringify(declared)) {
throw new Error(`${file}: variables [${declared}] do not match placeholders [${used}]`);
}
return { ...prompt, hash: hashText(prompt.template) };
}
Rendering is strict on purpose. A missing variable is an error, not an empty string in the middle of a prompt, and so is a variable the prompt doesn’t use:
export function renderPrompt(prompt, values) {
const missing = prompt.variables.filter((v) => !(v in values));
if (missing.length) throw new Error(`missing variables: ${missing.join(", ")}`);
const extra = Object.keys(values).filter((v) => !prompt.variables.includes(v));
if (extra.length) throw new Error(`unexpected variables: ${extra.join(", ")}`);
return prompt.template.replace(PLACEHOLDER, (_, key) => String(values[key]));
}
Choosing which version a feature uses is a deployment decision, so I’d keep it in config, or behind a flag if you want a gradual rollout, the way I described in feature flags for SaaS. The eval gate below decides whether a version is allowed to ship; the flag decides who gets it.
The eval set
The eval set is a JSON Lines file: one case per line, each with an input and a list of checks. It’s diff-friendly and easy to append to. Here’s a case from the sample:
{"id":"no-email","input":{"ticket":"Order #1007 never shipped. You can reach me at dana@example.com."},"checks":[{"type":"not_matches","value":"@"},{"type":"contains","value":"1007"}]}
The checks are deliberately cheap and deterministic: contains, does not contain, word limits, a regular expression, single line.
export const CHECKS = {
contains: (out, c) => ({ pass: out.includes(c.value), reason: `expected output to contain "${c.value}"` }),
not_contains: (out, c) => ({ pass: !out.includes(c.value), reason: `output must not contain "${c.value}"` }),
max_words: (out, c) => ({ pass: words(out).length <= c.value, reason: `expected at most ${c.value} words, got ${words(out).length}` }),
min_words: (out, c) => ({ pass: words(out).length >= c.value, reason: `expected at least ${c.value} words, got ${words(out).length}` }),
matches: (out, c) => ({ pass: new RegExp(c.value, c.flags ?? "").test(out), reason: `expected output to match /${c.value}/` }),
not_matches: (out, c) => ({ pass: !new RegExp(c.value, c.flags ?? "").test(out), reason: `output must not match /${c.value}/` }),
single_line: (out) => ({ pass: !out.trim().includes("\n"), reason: "expected a single line" }),
};
An unknown check type throws instead of passing, because a typo in an eval file should never turn into a green result.
Both the Anthropic and OpenAI guides push the same habits: success criteria that are specific and measurable, evals that run automatically, and a warning against “vibe-based” checking. Cheap checks cover the measurable half. For qualities such as tone or faithfulness you’ll likely add a human review or a model-graded check, and those need calibrating against people. I haven’t built that here. One more note on the OpenAI page: it says the hosted Evals platform is being retired (read-only from October 31, 2026, shut down November 30, 2026), which is one reason to keep your eval set and runner in your own repository.
Five cases is a toy. A useful set grows from real failures: every time a prompt change causes a bad output in production, add the input to the file with the check it should have failed. The set is the memory of past mistakes.
Running an eval
The runner takes a prompt version, the cases, and a model function. The model is injected, so tests pass the stub and production passes a function that calls your provider.
export async function runEval({ prompt, cases, casesHash, model }) {
const results = [];
for (const c of cases) {
let output = "";
let error = null;
try {
output = String(await model(renderPrompt(prompt, c.input)));
} catch (e) {
error = e.message; // a model failure is a failed case, not a crashed run
}
const failures = error ? [`model error: ${error}`] : runChecks(output, c.checks ?? []);
results.push({ id: c.id, pass: failures.length === 0, failures, output });
}
const passed = results.filter((r) => r.pass).length;
return {
prompt: { name: prompt.name, version: prompt.version, hash: prompt.hash },
casesHash,
total: results.length,
passed,
passRate: results.length ? passed / results.length : 0,
results,
};
}
Two details matter here. A model error (a timeout, a rate limit) fails that one case and the run continues, so one flaky call doesn’t hide every other result. And the run records the prompt’s hash and the eval set’s hash, so a saved result says exactly what it measured.
Comparing versions and gating a release
A single pass rate isn’t enough to ship on. The risky change is the one that raises the average while breaking something that used to work. So the comparison is case by case:
export function compareRuns(baseline, candidate) {
if (baseline.casesHash !== candidate.casesHash) {
throw new Error("runs used different eval sets; compare like with like");
}
const base = new Map(baseline.results.map((r) => [r.id, r]));
const regressions = [];
const improvements = [];
for (const r of candidate.results) {
const before = base.get(r.id);
if (before?.pass && !r.pass) regressions.push(r.id);
if (before && !before.pass && r.pass) improvements.push(r.id);
}
return { regressions, improvements, passRateChange: candidate.passRate - baseline.passRate };
}
compareRuns refuses to compare runs made on different eval sets, because the numbers wouldn’t mean anything. The gate then enforces two rules: a minimum pass rate, and a limit on regressions that defaults to zero.
export function gate({ baseline, candidate, minPassRate = 0.9, maxRegressions = 0 }) {
const diff = compareRuns(baseline, candidate);
const reasons = [];
if (candidate.passRate < minPassRate) reasons.push(`pass rate ${candidate.passRate.toFixed(2)} is below ${minPassRate}`);
if (diff.regressions.length > maxRegressions) reasons.push(`regressions: ${diff.regressions.join(", ")}`);
return { ok: reasons.length === 0, reasons, ...diff };
}
The command-line script runs two versions and exits with code 1 when the gate fails, so a CI job can block the merge:
npm run eval -- 2 3 # compare v2 (baseline) with v3 (candidate)
What the three versions scored
Running the stub model against the five cases:
| Version | What changed in the prompt | Passed | Failed cases |
|---|---|---|---|
| v1 | “Summarize in one sentence.” | 2 of 5 | no-email, short, leads-with-problem |
| v2 | Added the 25-word limit, the order number, and no email address | 4 of 5 | leads-with-problem |
| v3 | Added “start with the problem” | 5 of 5 | none |
Going from v2 to v3 fixes one case and breaks none, so the gate passes it. The more interesting test is the one I wrote to check that the gate does its job. It builds a candidate from v3’s text with the word limit tightened to “under 3 words”. Against v1 that candidate scores higher (4 of 5 against 2), but it cuts the one-line account question below the minimum length and fails no-order, a case v1 passed. A pass-rate-only check would approve it. The gate blocks it and names the case.
The tests
The 21 tests cover each layer on its own:
- Prompt store: stable hashes, files whose variables don’t match their placeholders, a file whose name or version disagrees with its path, numeric version ordering (so v10 is newer than v9), and strict rendering of missing and unexpected variables.
- Checks: every check type passing and failing, and an unknown type throwing.
- Eval files: invalid JSON reported with its line number, and duplicate case ids rejected.
- Runner: the three real prompt versions scoring 2, 4 and 5, a throwing model failing one case without stopping the run, and an empty set giving 0 rather than NaN.
- Gate: improvements and regressions listed correctly, mismatched eval sets refused, the minimum pass rate enforced, a risky candidate blocked, and the CLI exit codes (0 and 1).
The one I’d copy first is the blocked-candidate test, because it protects against the failure the whole harness exists to catch.
Swapping in a real model
Replace the stub with a function that takes the rendered prompt text and returns the model’s reply as a string, using your provider’s SDK. Keep the harness unchanged. Real models vary between calls, so expect to adjust: run each case more than once, or lower the temperature if the provider allows it, and treat a case that flips between pass and fail as information about the prompt rather than noise to ignore. Keep an eye on cost too, because every run now bills you. Track the same outcomes in production with events, the way I describe in product analytics instrumentation, and if the reply streams to the browser, streaming AI responses in Next.js covers that side.
What I did not test
- Any real language model. All scores come from the stub, so they show how the harness works, not how good a prompt is.
- Model-graded or human-graded checks, which tone and factual accuracy usually need.
- Repeat runs for nondeterminism, concurrency limits and retries on a real API.
- Prompts with multiple variables or chat-style message lists. The store handles several variables, but the sample feature uses one.
FAQ
How should I version prompts?
Treat them like code. One immutable file per version in the repository, reviewed in pull requests, with a content hash recorded in every eval run and, ideally, in your production logs. A change is a new file, never an edit.
What is a prompt eval?
A set of example inputs with checks on the output, run against a specific prompt version to produce a score. It does for a prompt what a test suite does for a function.
How many eval cases do I need?
More than five, and the number matters less than where they come from. Start with real inputs and every failure you’ve seen, and add a case each time something breaks.
Why block on regressions when the pass rate went up?
Because the average hides which cases moved. A candidate that fixes three cases and breaks one that customers rely on isn’t a safe change, and the case-by-case comparison shows exactly which one.
Can I use an LLM to grade outputs?
Yes, for qualities you can’t check with a rule, such as tone. Calibrate the grader against human judgments first, and keep the cheap deterministic checks alongside it.
I’m Shahrukh Akhtar (Shaz3e), CTO & Co-Founder of Diligent Creators Pvt. Ltd., a Technology Executive & Product Builder. I run client delivery and build SaaS products, including EloSync. You can find me on X, LinkedIn, Peerlist and GitHub.
