CI for prompts: how we stop translation quality walking backwards

14 July 2026 · ThaiPo

Prompt engineering has a dirty secret: most teams edit the prompt, eyeball three outputs, feel good, and ship. Two weeks later something that used to work does not, nobody knows which change broke it, and quality has quietly walked backwards. We have been burned enough times that ThaiPo's prompts are now treated exactly like code: versioned, regression-tested, and gated. Here is the pipeline.

Prompts are numbered artifacts, frozen forever

Every translation prompt has a version number, and when a version ships, its exact text is frozen into a history directory that nothing may edit. This sounds bureaucratic until the first time you need it: when v8 scores worse than v6 on some category, you can diff the actual shipped texts rather than reconstructing them from git archaeology and memory. The prompt that produced last month's behavior is a fact, not a recollection.

Back-testing: every version, every time

Before a new prompt version ships, the harness re-runs all historical versions against the current test corpus, not just old-versus-new. That matrix answers the question that actually matters: is the new version better than every ancestor, or merely better than the one we are replacing? More than once the winner on the current corpus was not the incumbent, which is exactly the regression this exists to catch. Language corpora drift as users and features change, so the ranking is recomputed rather than remembered.

The judge is a model, and the gate is real

Scoring is done by a judge model with a rubric (faithfulness, register, direction, staying in character), and the score is a release gate: a change that drops the pass rate does not ship. Two things make an LLM judge workable in practice. First, the rubric asks checkable questions instead of "rate this 1-10", because numeric vibes from a judge are noise with confidence intervals. Second, we spot-audit the judge itself against human reads periodically; a judge that drifts is worse than no judge, because it launders regressions into green checkmarks.

The eval that keeps us honest: could a Thai person tell?

For our native-output feature the headline eval is a blind discrimination test: mix genuine Thai chat messages with model output and ask, per message, whether a Thai person wrote it. Indistinguishability is the product goal, so it is also the metric, and the misclassified examples arrive pre-labeled with the tell that gave them away. Alongside it run targeted guards-as-tests: a calque detector (idioms translated word by word), a direction check, and a stay-in-character suite built from every message that has ever made the bot break character in production. Bugs graduate into eval cases; that is the whole retention policy.

Test data has to include the ugly cases

A lesson we relearned building live call translation: our harness fed the speech model clean synthetic audio, and everything passed while real phones on real networks failed, because pristine test audio cannot catch a regression that only appears on noisy input. The corpus now deliberately includes the ugly slice of reality: slang, code-switching, clipped fragments, messages that are half emoji. A green suite over clean data is a comfortable lie.

The 0% that was actually nothing

Our cautionary tale about trusting dashboards: a quality panel once reported a 0% pass rate, alarmingly, for weeks. The truth: an eval run had errored before scoring anything, persisted its empty result (zero passes of zero attempts), and every later glance read "0%" as catastrophe instead of "N=0". The fix is trivial (never persist an empty run; show sample sizes next to every rate), but the lesson generalizes: an eval pipeline is itself software, and its failure modes look exactly like your product's failure modes. Monitor the monitor.

What it costs and what it buys

The eval suite runs on cheap models where possible and the judge only where it earns its keep; the whole apparatus costs a rounding error next to production traffic, and the expensive comparison modes stay off unless someone is reading them. What it buys is the thing prompt work otherwise never has: monotonicity you can defend. When quality moves, we know which version moved it, in which direction, on which categories, against every prompt we have ever shipped. That is just CI, applied to the least deterministic code we own.

Try it in your own chats

ThaiPo lives inside LINE and translates every message in both directions, free forever. Get started free or add @thaipo.ai as a friend on LINE and it sends your signup link right in the chat.