The glossary that taught itself nonsense: 111,798 rows down to 10,844

10 November 2026 · ThaiPo

For a while, one of our chats was confidently instructing the translator that the Thai word for "Thailand" means a bank. Not as a one-off slip. As a rule, written into that chat's memory, applied to every message it translated.

ThaiPo learns as a chat goes on. A background job reads each chat's history and mines a glossary: nicknames, places, the words this particular group uses in its own way. The glossary is sealed per chat (nothing learned in one chat ever reaches another) and it is injected into that chat's prompt so the translation gets more personal over time. That is the feature. This post is about the summer it spent teaching itself nonsense, and how the table went from 111,798 rows to 10,844.

The symptom: replies about nothing

The report was that some chats were getting translations unrelated to what had been typed. A Thai sentence would arrive with raw English pasted into the middle. Worse, it was intermittent. We ran the identical poisoned prompt twice as an A/B: three of four outputs corrupted the first time, one of four the second. From inside a chat, nothing about that looks systematic, which is how it hid for months.

The cause was the miner. It was handed a flat list of message texts, with no speaker, no language tag, and no pairing, because consecutive chat messages are replies to each other, not translations of each other. In a chat that is mostly one language there are no cross-language pairs to find. A model asked for pairs does not shrug and return an empty list. It produced the only thing the input supported: an English-to-English explanation dictionary.

Those rows were then injected under a heading that said to always use these exact translations. In one group, 148 of its 149 injected terms were this kind of pair. Fleet wide, 35% of 159,718 stored terms were not translations at all.

The first gate is almost embarrassingly simple. Every chat translates Thai to and from exactly one other language, so a real term must cross the Thai boundary. One deterministic check kills the explanation dictionary, identical pairs, and typo pairs (which had been faithfully teaching the model the typo). It runs on write, so the table stops growing, and on read, so every bad row already stored is neutralised the moment the code deploys. Then we deleted 55,853 rows. The mining prompt had implied cross-language pairs all along. The model ignored it for months. A prompt is a request; a filter is a guarantee.

Valid, and still wrong

Two weeks later we looked at what survived. One free-plan group had 4,905 mined terms. The free plan injects the newest 25, so that chat's entire working memory was one day's harvest out of seventy days of mining. Everything it had genuinely learned was buried under thousands of newer rows.

Of those 25 live slots, nine were vocabulary any translator knows cold (the Thai for Spain, the Thai for electricity), three were garbled fragments of voice transcription, and three were simply wrong. เมืองไทย, "Thailand", mapped to a bank whose name resembles it. กราบ, "to bow in respect", mapped to Grab, the ride-hailing app. Perfectly well-formed Thai-to-English pairs. Complete nonsense.

The obvious fix is a novelty check: drop anything the base model already translates correctly. On its own that would have made things worse. A novelty check keeps every entry that disagrees with the model, and disagreement has two causes: real chat-specific knowledge, and garbage. It would have thrown out the nine harmless words and carefully preserved all three poisoned ones.

So the gate is a three-way classifier. Every candidate is sorted into redundant, wrong, or useful, and only useful is stored. The first version of its prompt described the three verdicts and asked for one. Tested on 60 real mined terms it got 44 redundant and 9 wrong correctly, then called all three poisoned entries useful. Shown a plausible mapping between two real things, the model read it as local knowledge, which is exactly what a glossary is for. Adding "be strict" changed nothing.

What worked was ordering. The model must write down what the source term means, in its own words, beforeit is allowed to judge the proposal. Having committed to "the country Thailand", it cannot reconcile that with a bank.

One detail we like: the gate and the purge fail in opposite directions. The gate decides whether to add, so a term with no verdict is not stored. The purge decides whether to delete, so a row with no verdict is kept. Same principle, opposite action. The purge ran through the batch API: 441 requests, under three minutes, zero failures, $6.78, and 111,798 rows became 10,844.

The suspect with an alibi

Then came a case that looked like more of the same. A group's proper names were coming back mangled: a business name rendered as a literal Thai translation of its parts (imagine a shop called Brightside coming back as "the bright side of the road"), an acronym spelled out phonetically in Thai letters. Obviously the glossary again.

That chat had zero glossary rows.

The culprit was the rolling context window, the short list of recent exchanges the model reads to stay consistent. It stores each exchange as source and rendering, and hands them back as worked examples. In a chat whose messages are mostly short labels, every line of that window is a substitution row. The window had quietly become a glossary nobody audited, and one wrong rendering was how that chat spelled the name until the window rolled over. (The doubling spiral in our post on never answering is the same family.)

The fix is two refusals at write time: an exchange where a name was mangled, or where a short label was merely respelled by sound, never enters the window. That refuses 0.8% of exchanges across the fleet. Our first cut refused every label whose English vanished into Thai, which fixed the chat, cost 16.3% of exchanges, and taught the window that English should stay English. Measured on real windows through the live model, bad renderings fell from 34 of 66 to 14 of 66.

The glossary got the matching gate: a mined pair that is the same sounds in two alphabets is refused, by comparing consonant skeletons, since consonants are what survives a person guessing at a spelling. The threshold was swept over the real rows. Of 739 distinct pairs dropped, three were genuine losses.

The disclaimer that did nothing

Along the way we measured the cheap fix, twice. Rewrite the heading on the injected block: these are hints, not rules, a past rendering may be wrong, never copy a line. On the glossary block that scored 8 of 24 against a baseline of 9. The gate that removed rows scored 12, and both together 14. On the context window, the new header scored 33 and 25 bad against the control's 32 and 40 across two A/Bs. Noise.

How you label a block does not travel. What is in the block does. The model reads the contents as examples and mostly ignores the prose about how to read them.

What we took away

• A learning system needs an immune system. Anything that writes to its own future prompts will eventually write something wrong, and then obey it.
• Validate at write, not at read. Disclaimers at read time measured as inert, twice. Refusing the row worked both times.
• A self-training loop amplifies its own errors. A mistake made once is a typo. A mistake stored is a policy.
• Measure the table, not the vibes.The fixes here started with counting rows, and the causal stories that held up were the ones we A/B tested on real data. Two "regressions" we read off a single sample vanished on the second.

ThaiPo is designed, built and run by one developer, Steven Acreman.

Try it in your own chats

ThaiPo lives inside LINE and translates every message in both directions, free forever. Get started free or add @thaipo.ai as a friend on LINE and it sends your signup link right in the chat.