The boring knobs that cut our LLM bill by two thirds

14 July 2026 · ThaiPo

ThaiPo translates live chats with frontier models, runs a background learning pipeline that mines vocabulary and runs quality evals, and the whole Anthropic side of that costs a few dollars a day. None of it came from clever engineering. Every meaningful saving came from a boring knob we had initially set wrong, and each knob was worth double-digit percentages. Here they are, with what each one taught us.

1. Prompt caching does nothing until you design for it

We turned on prompt caching early, saw the line item, and assumed we were saving money. Months later we measured the actual cache hit rate: about 0.1%. Effectively zero. The reason is embarrassing in hindsight: cache entries expire after five minutes by default, and a per-chat prompt for a chat that gets one message every twenty minutes never hits. We were paying the cache-write premium on every call and collecting none of the reads.

The fix had two parts. First, restructure prompts so the large static part (the translation instructions) is a shared prefix, identical across every chat, with the per-chat material appended after it; the shared prefix stays warm because all traffic refreshes it. Second, pay for the one-hour cache TTL on that prefix instead of the default five minutes. Sparse-traffic products, which is most products, should assume caching is doing nothing until a measured hit rate says otherwise.

2. Everything that can wait an hour goes through the batch API

The learning plane (vocabulary mining, summaries, evals) has no user waiting on it, and batch pricing is half of interactive pricing. Flipping the whole background pipeline to batch was a configuration change worth an immediate 50% on that entire category of spend. The general rule: sort every model call into "a human is watching" and "nobody is watching", and be ruthless about the second pile.

3. Mine incrementally, and beware work that rides along with deploys

Early versions re-read far more history than had actually changed. Incremental mining (process only what is new since the last high-water mark) cut that spend to a sliver. The subtler bug: our memory pipeline re-fired on every boot, which means every deploy silently re-ran mining across chats. Deploy-day cost spikes looked like traffic; they were us. If your background jobs trigger on process start, your release cadence is a cost multiplier.

4. Turn off the science you are not currently reading

Benchmark harnesses, A/B comparisons, and side-by-side eval modes are wonderful and cost real tokens on every message they shadow. Ours are behind flags, and the flags are off unless someone is actively looking at the results. Quality gates that block releases stay on; dashboards nobody is reading this week do not get to spend money.

5. A model ladder, priced per tier

Free chats run on a fast, cheap model; paid chats run on the best model we can buy. That sounds obvious, but the discipline is in the routing table: every single call site (live translation, photo reading, background mining, evals) has an explicit model assignment with a counter and a dollars-per-day estimate next to it, changeable at runtime without a deploy. The expensive surprises were never the decisions; they were the call sites nobody had made a decision about.

6. Prepaid credits with auto-reload off, on purpose

Our billing posture is a circuit breaker: prepaid credits, auto-reload disabled. If something runs away (a retry loop, a poisoned queue, a bug in any of the above), the worst case is a few hours of degraded service, not an invoice with a comma in it. We have taken the outage exactly once, when credits ran out faster than expected: translations paused, nothing was lost (every message sits in the log and was translated on refill), and the incident cost us an apology instead of a mortgage payment. For a small company we would make the same trade every time, with one caveat: it only works because the architecture tolerates the pause. If a missed webhook lost data, a hard spend cap would be a data-loss switch.

7. Meter everything, including the invisible half

For months our per-chat cost accounting quietly excluded the background pipeline, so the dashboard under-reported real spend by roughly half, and the gap grew with every feature. Now every model call, foreground or background, records its tokens against the chat that caused it, priced per model. The uncomfortable rule of thumb: whatever your cost dashboard says, audit for the spend category it does not know about, because there is one.

Where it nets out

The baseline (caching designed properly, batch for background, incremental mining, science off) holds the platform at a few dollars a day. What remains is dominated by one deliberate choice: paid chats get the most capable model for live translation, because translation quality is the product. The knobs above exist so that when we spend, it is on that, and not on cache misses and forgotten benchmarks.

Try it in your own chats

ThaiPo lives inside LINE and translates every message in both directions, free forever. Get started free or add @thaipo.ai as a friend on LINE and it sends your signup link right in the chat.