How we built live translated video calls with no server in the hot path

14 July 2026 · ThaiPo

ThaiPo's newest feature is a video call where two people who share no language talk face to face, and everything each of them says appears on the other's screen in their own language, starting to render well under a second after the words are spoken. You type /callin a LINE chat, everyone taps a link, and that's the whole product.

This post is about how it works, because the interesting part is not a clever model. It is an architectural decision: our servers never touch the call.

The rule: no server in the hot path

Latency budgets for "live captions on a call" are brutal. Every hop you own is a hop you have to keep fast at 3am on hotel wifi. So we own none:

  • Voice and video travel phone-to-phone over WebRTC. Up to four people form a full mesh; with a hard cap of four, a mesh stays cheaper and simpler than an SFU, and there is no media server to scale, secure, or pay for.
  • Speech recognition and translationrun over a WebSocket each phone opens directly to Gemini's Live API. Your phone streams your microphone; the model streams back a translated transcript.
  • Captions reach the other phones over the WebRTC data channel that is already open for the call. No caption ever visits a ThaiPo machine.
  • Our backend does exactly three jobs, all off the hot path: it mints session tokens at join time, relays a handful of signaling messages while the call is being set up, and receives a 30-second heartbeat for metering.

A side effect we like: since audio flows browser-to-model and captions flow peer-to-peer, there is nothing for us to store. Calls are not recorded because there is no place in the architecture where a recording could happen.

Locked ephemeral tokens, and the bug that taught us to verify

If the browser talks to the model directly, what stops a user from opening the connection with their own instructions? The Live API has a good answer: ephemeral tokens that can carry the session configuration locked. Our server composes the system prompt (the translation instructions, plus that chat's learned vocabulary) and bakes it into the token. The browser can connect with the token but cannot change what the session is.

The war story: the SDK accepts a field listing which extra configuration fields the client may still set. We passed an empty list, meaning "lock everything". On the wire, that empty list serialized in a way that meant the opposite: sending the field at all disabled the constraint. Sessions connected fine, took no instruction, and did whatever a friendly multilingual model does with raw audio, which in the field looked like captions in random languages. The fix was deleting one line, and the lesson was permanent: for security properties, test the behavior, not the parameter you think you set.

Each viewer gets exactly one language, enforced twice

Each participant is pinned to a side at join (their LINE profile language is the strongest signal, then the browser locale). Their own Gemini session translates what theysay into the other side's language, and their screen shows only what others said, rendered in their own language.

Instructions alone were not enough. Real calls code-switch, people read words aloud from the other language, and models occasionally drift. So the client enforces the contract at render time: a caption is shown only if its text is actually in the viewer's script, checked by Unicode ranges, otherwise it renders nothing. Thai script has its own block, so for a Thai-English call this check is cheap and exact. It also caught our strangest bug: describing the source language to the model in a way that mentioned a regional dialect made it occasionally write the translation in Lao script, which is a different Unicode block that most Thai readers cannot comfortably read. The render rail rejected it, the eval harness flagged it, and the prompt now pins the output script explicitly.

Captions that mutate

We show the model's partial transcript immediately and let it rewrite itself as the sentence completes, the way a human interpreter revises mid-sentence. Waiting for final, polished text would cost two to three seconds of perceived latency; showing honest drafts costs nothing but humility. The current line renders large and settles in place; history fades above it.

Paying for silence is optional

Streaming audio to a model is metered by the second, and most of a call is silence or the other person talking. A tiny voice-activity gate in an AudioWorklet only forwards audio while its owner is actually speaking, with a 900ms hangover so trailing syllables survive, and tells the model explicitly when the stream pauses. That one gate roughly halves the audio bill. A two-person call costs us about two US cents per minute, all in.

The permission gauntlet

Almost everything that failed in field testing failed before the call even started:

  • Modern browsers silently deny camera and microphone prompts that arrive without a user gesture. Our first design auto-joined on page load, which looked frictionless and guaranteed a gestureless prompt. First-time users got denied with nothing visible in any settings screen. The fix is a single full-screen Join button; returning users with permissions already granted still skip it.
  • Our own security headers had shipped a Permissions-Policy that disabled camera and microphone for the whole site, months before calls existed. iOS Safari ignores that header for top-level pages, so iPhones worked; Android Chrome obeys it, so one tester was unfixably blocked while every settings screen on his phone looked healthy. If you see getUserMedia denied with no prompt and clean settings, check your own response headers first.
  • iOS routes call audio to the earpiece when the microphone is live, which at arm's length reads as "no sound". The Audio Session API opts you back to the loudspeaker.

Avatars for free

One more thing fell out of the architecture. Because captions already ride a data channel, adding a voice-puppet avatar cost no bandwidth at all: the only thing on the wire is a one-line "I am a bunny now" note. Each phone draws the character locally over that person's tile and animates the mouth from the audio level it is already receiving. Camera-shy users can join with the camera never enabled and still be a talking tiger.

Try it

Every ThaiPo plan includes ten call minutes a month, so you can feel the latency yourself: add ThaiPo to a LINE chat and type /call. The full feature is documented in the docs.

Try it in your own chats

ThaiPo lives inside LINE and translates every message in both directions, free forever. Get started free or add @thaipo.ai as a friend on LINE and it sends your signup link right in the chat.