One CPU core, one debug script, four-minute translations

10 November 2026 · ThaiPo

In June 2026, every ThaiPo chat got slow at the same moment. Translations that normally land in a few seconds started arriving two to four minutes late, in every chat at once. Nothing had crashed and nothing was logging errors.

The culprit was us, in a shell, on a production machine, trying to be helpful about something else entirely. This is the post-mortem, including the part where the design made it possible and the part of that design we decided to keep anyway.

A single small server juggling a stream of chat messages while a debugging script piles a mountain of files onto it, and a queue of late messages builds up behind

The one-second budget

When someone sends a message in a LINE chat that ThaiPo is in, LINE calls our webhook and waits for an acknowledgement. It does not wait long: roughly one second. Miss that, and LINE treats the delivery as failed and tries again later. "Later" turns out to be minutes, not milliseconds.

Our webhook path has more hops than you might guess. Only the web app has a public address, so it hosts a small proxy route: it takes LINE's request and forwards the raw bytes over the private network to the ingest service. Ingest verifies the signature, writes the event to object storage (our source of truth), writes a second small marker object, and only then answers. The proxy is forward-and-wait: it holds LINE's request open for that whole round trip and relays whatever status comes back.

So the one-second budget is spent on four things, in series:

  • CPU on the web machine, to accept the request and forward it.
  • A private network hop to the serving plane.
  • CPU on the serving machine, to verify and fan out.
  • Object storage latency, for every write ingest awaits before it says 200.

On a normal day that fits with room to spare. The catch is that none of the four has its own reserve. Anything that slows any one of them is paid for out of the same second.

The debug session

At the time, the web app ran on a single machine with one small shared CPU core. That same machine was also the handiest place to debug from. Our Go services ship as distroless containers with no shell in them at all, while the web container has a shell, a Node runtime and credentials for the bucket. If you want to poke at stored objects, it is right there, already logged in, practically asking.

We were chasing an unrelated translation question and wanted to reconstruct some history. So we ran a script on that machine that listed and fetched thousands of objects from the bucket. Then another one. Each ran for a while and took as much of that one shared core as it could get, which is to say all of it.

The proxy route lives on that same core. It did not crash. It did not log an error. It just got its turn on the CPU a little later than usual, and "a little later" was enough to push acknowledgements past LINE's timeout. LINE did exactly what it promises: it gave up on those deliveries and redelivered them minutes afterwards.

Because the proxy is the front door for everything, this was not one slow chat. It was every chat with a message in flight during those windows.

The tell

What made it diagnosable is a single boolean. LINE marks each webhook event with a redelivery flag, and ingest logs it alongside the event. Lined up by time, the logs told the whole story:

  • Before the scans: events arriving with gaps of four to eight seconds, redelivery flag false.
  • During the scans: gaps of 132 and 250 seconds, redelivery flag true, across many chats simultaneously.

The slow windows lined up exactly with our own shell sessions. There is a particular feeling that comes from correlating an outage against your own command history and getting a perfect fit. We do not recommend it.

The redelivery flag being true everywhere was the important clue. A slow model makes one translation late. A stuck queue makes one shard late. Only a slow front door makes LINE itself resend, for everyone. At least one message had still not reached the serving plane ten minutes later, so we cannot rule out that a webhook was lost outright.

What changed

Two web machines, each with more CPU.One shared core was simply too little margin for a process that sits inside someone else's one-second timeout. The web app now runs on two machines with two shared cores each. It took two attempts: the first CPU bump was quietly reverted to one core while a comment in the config still described the fix, and we only caught it two months later, which is when the second machines arrived.

Two serving machines. The serving plane is on the same path with the same deadline, so the same argument applies to it.

Both counts are written down as load-bearing.The repo's conventions file, the one every contributor (human or otherwise) reads first, says in bold not to put either back to one, and says why. Machine counts are the kind of thing that gets "optimised" on a quiet afternoon by someone looking at a bill. The background learning plane, by contrast, stays at one machine on purpose: nothing waits on it.

Heavy scans never run on the ingress machine. A tightly scoped lookup (one chat, one day, a handful of objects) is tolerable. Anything bigger runs on a throwaway machine that exists only for that job, or the objects get pulled down and processed locally.

Logs are free, shells on production are not.Reading logs puts no load on the machine that produced them. A shell session shares a CPU with whatever that machine does for a living. We reach for logs first now, and treat a production shell as something with a cost attached, even when it is only "reading".

The tradeoff we kept

The obvious structural fix is to make the proxy acknowledge immediately: return 200 to LINE the instant the request arrives, then forward to ingest in the background. CPU contention would no longer matter, because the ack would no longer wait on anything.

We looked hard at that and have not done it.

Forward-and-wait means the 200 that LINE receives is a true statement: the event is durably in object storage. If the serving plane is unreachable, the proxy answers 503. If a storage write fails, ingest answers with an error. In both cases LINE keeps the message and redelivers it, and because ingest keys each stored event by LINE's own event id, a redelivery that does arrive twice is a harmless overwrite.

Ack-first inverts that. The proxy would say "got it" before anyone had got anything, and a serving outage would turn into silently dropped messages, held only in the memory of a web process. We would have to build our own durable buffer first, to replace the retry LINE gives us for free. The incident would have been invisible, which sounds like an improvement until you notice that the failure it hides is the lesser one.

Late is recoverable. Lost is not. So the proxy still waits, and we bought headroom instead of hiding the dependency.

Takeaway

If your process answers inside someone else's timeout, everything sharing its CPU is part of your latency budget, including you, at a prompt, running one quick script. Write the budget down, name every hop in it, give the path spare machines, and do your archaeology somewhere else.

ThaiPo is designed, built and run by one developer, Steven Acreman.

Try it in your own chats

ThaiPo lives inside LINE and translates every message in both directions, free forever. Get started free or add @thaipo.ai as a friend on LINE and it sends your signup link right in the chat.