Answering the Past with the Future

Where It Started: Captions That Never Match the Speaker

It all began with a small, familiar frustration — turning on live captions in MS Teams or YouTube and watching text that doesn't match what's actually being said. Wrong words, garbled phrases, captions that rewrite themselves mid-sentence.

The first question was simple: how big a model do you need to understand unclear speech and heavy accents in real time?

The answer was surprising — model size isn't the problem. A 1.5B-parameter speech model already handles accents remarkably well. The real problem is time: a live system must decide what you said before your sentence ends. It sees only the past, never the future. Offline transcription gets the whole sentence, can revise freely, and can think as long as it wants.

Live captions lose to offline transcription every time — not because they're dumber, but because they're forced to answer before seeing the whole exam.

The Truth to Accept First: 100% Doesn't Exist

Before designing anything, the ceiling needs to be clear — 100% accuracy is impossible, and not because the technology isn't there yet:

• Professional human transcribers run ~4–5% error rates, and two people transcribing the same file produce different results — meaning even the "correct answer" is ambiguous.
• Homophones, unfamiliar proper nouns, words genuinely swallowed by the air — no amount of context turns a principled guess into knowledge.
• Extend the pipeline to translation and errors compound. And some layers of meaning — jokes, wordplay, politeness registers — never mapped cleanly across languages to begin with.

Then came the insight that flips the whole framing:

Communication never required 100% per-sentence accuracy — it requires convergence toward shared understanding across the conversation.

Humans misunderstand each other constantly, then ask again, paraphrase, repair mid-sentence. Linguists call this conversational repair. Communication is a naturally self-correcting protocol, so per-message errors aren't fatal as long as the system keeps both sides converging.

The right question isn't "is it 100% accurate?" but "has it crossed the usable line?"

The Architecture: Answering the Past with the Future

The core idea is best explained with a sci-fi image — as if person A knows in advance what person B will say, and comes back to tell the present.

The actual mechanism:

10:00:00 ──── speech begins ────► 10:00:02
                                       │
                                       ▼
                            10:00:03 the model creates a "commit"
                            answering the 00–02 window
                            = at second 1 of the audio,
                              the model "sees the future" by 2s

• The system deliberately delays 1–2 seconds so that every word is decided with "future" audio in view — trading a little latency for a jump in accuracy. (Humans tolerate this easily; professional interpreters lag 2–4 seconds.)
• Results are stored as append-only commits — old ones never disappear; the uncertain tail stays provisional until the next round confirms it.
• Each new commit pulls context from the previous 2–3 commits as a prior — with a helper layer summarizing context per section, extracting names and jargon, and feeding them back to prevent drift.

The remarkable part: compared against actual research, nearly every piece of this maps to techniques the field already uses — lookahead buffering, LocalAgreement, previous-text conditioning, two-pass architecture. Different people, different contexts, same constraints — converging on the same answer. That's not coincidence; it's evidence the design is right.

Placed at the Right Layer, It Just Works

The problem splits into two cases requiring different architectures:

Case 1: Remote Calls — Put the AI on the Outbound Side

Intercontinental calls carry 100–300ms of speed-of-light latency that no network can remove — yet everyone accepts it without noticing.

So why not use that unavoidable window as free compute time? Place transcription/translation on the speaker's side so it overlaps with the time audio already spends crossing the planet. Translation latency no longer adds to network latency — it hides inside it. Pure pipelining, exactly as in a CPU.

The bigger bonus is signal quality: the outbound side hears raw audio before codec compression, before packet loss. And the speaker's own device knows its owner best — accent, frequent vocabulary, names of people around them. Personalized accuracy comes free from being on the right side.

Case 2: Face-to-Face — Cut the Datacenter Out

Face-to-face has no network latency to hide work inside. A cloud round-trip adds 100–300ms for nothing. The only correct path is fully on-device, or direct P2P (Bluetooth/local link) without touching the internet at all.

The whole principle in one sentence:

Good systems don't wait for complete information — they guess first, and pay the correction cost only when the guess is wrong.

CDNs, CPU branch prediction, optimistic UI — all the same pattern. Real-time translation is just the newest member of a very old family.

Why Now: The Moment Two Lines Cross

In the past 1–2 years, something happened that had never been true before — frontier models distilled down to 1–2B parameters with their core capability intact. Speech models run in real time on phone NPUs. Small LLMs are smart enough to repair transcripts from context.

Which means everything above — lookahead commits, on-device, P2P — only recently became possible. What once required a datacenter now fits in a pocket. The wall blocking these ideas already fell, and most people haven't noticed yet.

And that is a very good signal — for building something that genuinely helps people.

What Makes All of This Matter

Behind every technical question here, there are real stakes: I have a hearing impairment, and I'm currently recovering from cochlear implant surgery.

For most people, a wrong caption is an annoyance. For someone whose captions are their primary channel to the world, a wrong caption means misunderstanding the situation entirely. The latency-vs-accuracy trade-off for these users is fundamentally different from a general product — and the person who can tune that balance most correctly is the one who lives with it every day.

Captions for real life must work offline — in elevators, on trains, in hospitals — and must be private, because they listen to every conversation around you. Both requirements point the same way: on-device is the answer, and it just became possible.

Assistive technology has always lacked one kind of person — someone who understands the engineering constraints deeply enough to design the system, and understands the user experience from the inside rather than from interviews. Those two things almost never live in the same person.

Perhaps this is the story of someone who has both.

• Appendix: Technical Notes (for the day implementation begins)

Core Pipeline

    [Microphone / raw audio]
       → VAD (remove silent intervals)
       → Chunked ASR + 1–2s lookahead
       → LocalAgreement (commit only the portion where two consecutive hypotheses agree)
       → previous-text conditioning (feed prior commits back as the prompt)
       → [For translation] MT → TTS on the receiver's side

Production-ready components to build on

• ASR: Whisper large-v3 (~1.5B, VRAM ~10GB fp16 / ~5GB int8 via faster-whisper) — smaller distilled variants can run on a mobile NPU
• Streaming layer: whisper_streaming (github.com/ufal/whisper_streaming) — already implements LocalAgreement and connects directly to faster-whisper
• Context layer: an LLM with 1–7B parameters serves two roles: (1) repair the transcript from context, and (2) maintain a list of domain terms and names, then inject them back as custom vocabulary / prompt bias — reducing proper-noun errors more effectively than moving to a larger model
• Two cases, two deployments: remote = ASR at the outbound edge (raw audio, overlapped with network latency) → send text/latent across the network → TTS at the receiver / face-to-face = fully on-device or P2P over a local link

Reference numbers

• Total pipeline latency budget: target 1–2s, with ~4–5s as the ceiling for a fluid conversation (warning: lookahead + inference + MT + TTS compound)
• Reference WER: 2–4% for clean offline audio (near the human ceiling), 8–15% for live speech + accents + specialized vocabulary
• Thai adds another layer of difficulty through word segmentation (no spaces between words) — acoustics can be correct while segmentation is wrong
• Physical floor: intercontinental round trips always take tens to hundreds of milliseconds → the answer is to hide latency through speculation, not wait for a faster network

Design principles

1. Do not chase 100% — design the system to repair itself (repair > accuracy)
2. Planned delay is a weapon, not a weakness (lookahead)
3. Append-only commits + provisional tail
4. Context flows from the slow path back into the fast path
5. Topology determines placement: remote = outbound edge / face-to-face = on-device
6. Assistive users weigh trade-offs differently from general users — let the people who use it every day tune the balance