An avatar that acts while it speaks lives or dies on timing. The words are generated and voiced in real time, and the actions bound to them (a click, a scroll, a highlight) have to land on the word that motivates them. Miss by a beat and it reads as dubbed: a mouth saying "click here" a second after the cursor already moved. This is an account of how we drive those actions off a per-word timing signal the HeyGen Avatar Realtime API emits alongside the video, and what we learned running it as a 24/7 Twitch show that has delivered 9,269 verdicts on air.
The second stream
Open a HeyGen Avatar Realtime session, push text into it, and two things come back. The first is obvious: a photorealistic talking-head video. The second is easy to overlook and does most of the work here. Over a server-sent-events stream (/v3/avatar-realtime/{id}/words), the API emits a per-word timestamp feed. Each event is a {word, start, end} triple: this word is spoken from second start to second end on the video's own timeline. It isn't in the public API docs; we found it on the wire and built on it.
That second stream is what makes acting-while-speaking possible. For any word the avatar is about to say, it hands over the media-time position where that word will land, a little ahead of the audio. Bind an action to a word and it fires with the speech: the moment the voice reaches "click," the click executes, rather than at a guessed offset.
One number tempts you into doing this wrong. A POST /text to the Realtime API acknowledges in about 70 ms, which tells you the request was accepted, not when the word will be heard. End to end, a pushed word takes several seconds to travel through synthesis, encoding, and streaming before it plays. The per-word timestamps are what make that long, variable pipeline manageable: every word comes back tagged with the media-time position where it will actually play, so you schedule against the speech, not against a wall-clock estimate of it. (The generation stack that produces this stream is its own story: the inference report.)
Why timestamps, and not a timer
The naive approach is a stopwatch. Estimate a speaking rate, say two and a half words a second, start a timer when you send the sentence, and fire the click at the predicted offset. It works until it doesn't: the model synthesizes a little faster or slower than your estimate, a comma adds a pause you didn't budget for, or the network jitters, and the click slides off the word. Reading the video player's own clock (currentTime) instead of counting wall-clock seconds is better, but it still leaves you translating between clocks by hand, and it still says nothing about which word is being spoken right now.
The deeper issue is that these are all different clocks, and none of them is the one you actually care about. You don't care about "two and a half seconds from now." You care about the instant the voice says this word. The per-word stream is that clock, handed to you directly. Everything else, the video playhead and wall-clock time, stops being something to estimate and becomes something you convert into the word clock.
Anchoring the word clock to the wall clock
A word's start is a position on the media timeline, the stream's own clock, not a wall-clock instant. To fire at the right real-world moment you need one anchor tying the two together: wall_t0 = Date.now() - currentTime * 1000, the wall-clock time at which media-position zero played. With that anchor the scheduler watches the player clock and fires a cue when media time crosses the word's start, plus a small lead so the gesture and the syllable land together.
The browser reports currentTime back about four times a second and the anchor is recomputed on each sample, because the two clocks drift apart slowly and re-sampling that often keeps the accumulated error negligible. The same freshness check doubles as a safety rule: if the newest anchor is more than about two seconds old, the scheduler refuses to fire against it. A cue that can't be placed on a current player clock is dropped, not guessed. That is a deliberate choice worth stating plainly, because it comes up again: the system would rather skip a gesture than fire it on the wrong word.
A 24/7 AI repo-roaster
To make this concrete, here is the system it was built for: a HeyGen photo avatar that hosts a continuous Twitch show. It drives a real, visible Chrome window (navigating, scrolling, highlighting, clicking through live pages) and narrates what it sees. Viewers drop GitHub repository URLs in chat; the avatar opens each one, reviews it on air, and posts a verdict to a leaderboard. Since June 5 it has run continuously at roughly 2,400 text pushes and 2,500 synchronized browser actions a day, all on a single Mac mini.
The author model never emits coordinates. It writes narration with inline cues like [CUE: {"op":"navigate","target":"https://github.com/foo/bar"}], where each cue is an intent ("highlight the pricing limits," "point at the star button") pinned to a phrase in the script, not a pixel or a CSS selector. A separate, deterministic grounder resolves that intent against the captured page. It is ordinary string matching, not vision: it searches the page's rendered text and structure for the phrase, falling back from an exact span to a shorter unique one to the containing section, and gives up rather than guess if nothing matches. The model decides what to point at; non-model code decides where it is and when the voice gets there.
The timing is handled entirely by the word stream. A component called word_tracker subscribes to the per-word feed and walks a cursor through the script as each timestamp arrives. When the cursor reaches a cue's position, the cue fires (a cursor move, a highlight, a scroll, a click) at the moment the avatar's voice reaches the word it was pinned to.
Matching the script to the incoming words is exact, not fuzzy. The tracker accepts a word only when the streamed token is an exact (case-insensitive) substring within a 200-character lookahead of the cursor, and it won't consider the script started until three tokens line up in order. The strictness is the point: a cursor that guesses eventually fires a click on the wrong word, and a wrong-word action looks far worse than a missing one. There is one escape hatch, since strictness that never acts is its own failure: if the avatar is clearly speaking but the words haven't confirmed within about eight seconds, the tracker fires the remaining cues best-effort rather than dropping them.
Two problems, one stream
The same per-word stream solves a second problem. A pre-scripted show wouldn't need any of what follows; you could render it ahead of time. This show fights to stay close to the live edge because that is what lets a viewer's chat message get on air. The tighter the gap between what has been written and what is being spoken, the sooner a freshly submitted repository can be slotted in. Pacing is how you keep that gap tight without starving the avatar into silence.
So the content brain has to know its runway: how many seconds of already-spoken-for audio sit buffered ahead of the viewer, which tells it when to write and push the next scene. Undershoot and the avatar runs dry and goes silent; overshoot and the queue balloons, the buffer grows, and every chat submission now waits behind minutes of committed script.
Measuring runway by estimate (count the words you've sent, divide by a speaking rate, subtract elapsed time) drifts for the same reasons the stopwatch did. The word stream already carries the answer. The last word's end is the media-time position of the final syllable you've committed; the player's currentTime is where the audience actually is. The runway is the gap between them:
runway_sec = latest_word_end - currentTime
Both terms live on the same media-time axis, so the gap is immune to wall-clock drift and reflects the voice's real pace, including any [pause] marks, with no words-per-second guessing. Through a rendering hiccup it stays honest, since the player clock and the last word's end both hold in place rather than one sliding past the other. The one thing the formula can't see on its own is the word feed itself going quiet: words arrive in bursts, so a gap in the feed can make the backlog read artificially low. That failure, and the fail-closed rule that keeps it from starving the show, is the subject of a later section. One stream, two answers: when to fire a cue, and how much runway is left.
How it's wired
Under the hood the system is four small Python components handing off over HTTP and SSE:
- chat_ingester reads Twitch chat, filters for GitHub URLs, and queues them, posting a quick templated reply in chat. It also pre-writes a spoken acknowledgement with Claude Haiku 4.5 as a fallback, though the line actually spoken on air is regenerated in context by the showrunner.
- showrunner is the content brain. Claude Opus 4.8 is the episode author: once per repository it receives the whole captured page and writes the scene (narration, browser plan, inline cues), and it also drives the in-scene reactions and spoken acknowledgements. Claude Sonnet 4.6 runs a lighter planner that fills airtime when nothing is queued. A Playwright executor drives Chrome while word_tracker fires the cues on the spoken word.
- stream_orchestrator ("M1") owns the HeyGen Realtime session and every surface OBS composites: the avatar page, the captions, the runway endpoint, and the proxy that fans the per-word stream out to everything that needs it.
- script_producer is a small library (the word_tracker and cue parsing) that the showrunner imports.
flowchart TD
viewer([Twitch viewers]) -->|drop GitHub URLs| chat[Twitch chat]
chat --> ingester[chat_ingester]
ingester -->|SQLite queue| showrunner[showrunner: author + planner + executor + word_tracker]
showrunner -->|scene: narration + browser plan + cues| m1[stream_orchestrator M1]
m1 <-->|text in / audio + HLS + per-word SSE| heygen[(HeyGen Avatar Realtime API)]
showrunner -->|navigate / scroll / click| chrome[Visible Chrome window]
m1 -->|avatar video + captions| obs[OBS Studio]
chrome -->|window capture| obs
obs -->|RTMP| twitch([Twitch])
A HeyGen Realtime session is capped at about an hour, so the show can't hold one forever. Rather than wait to be cut off mid-sentence, a scheduler restarts the whole stack about every 50 minutes, just under the cap, and the new session re-anchors and picks up from the editorial state (which repositories have been covered, recent phrasing, scores) that lives outside any single session. The seam is short by design, and the new session resumes from that carried-over state. If a session dies early, on a backend error or a dropped connection, the same cold-reopen path recovers it. There's no warm standby, just fast, boring recovery.
The pacing gate itself is deliberately dull, and one incident is why. The word stream went briefly stale; an earlier version of the gate fell back to a wall-clock estimate, which can drift negative on a long session, and a negative runway looks like starvation, so the pusher did what it was told and force-fed text. The buffer ballooned by about 94 seconds and dragged the sync back into the uncanny zone. The fix was a rule, not a patch: when the sync signal goes stale, fail closed to a number that can only drain, never inflate. On the stale path the runway is computed from the real buffered word_end minus a fresh currentTime, clamped non-negative, so it decays honestly to zero instead of lying upward. A more structured successor to this gate, a small state machine that labels each moment measured, repinning, blind, or dead, currently runs in shadow mode beside the live pusher. It is not in control yet; how we're deciding whether to trust it is the next section.
None of this is flawless. Cue firing depends on the script's words matching the timestamps coming back, and when they diverge the tracker can lose its place; the result is a dropped or mistimed cue, a visible blemish, never a wrong-word fire. The live runway still has no upper clamp, which is exactly the class of bug the shadow state machine is meant to close. A system that runs unattended for weeks is held together as much by monitoring and these fail-closed defaults as by any single clever mechanism.
Does it actually work?
The architecture makes a testable promise: an action lands on the spoken word, not near it. The recorded numbers say it holds. Over the first ten hours of per-cue logging (2,221 executed cues), a cue fired a median of 283 milliseconds from the word it was pinned to, 1.48 seconds at the 95th percentile, 4.28 seconds at the 99th. At the median that reads as part of the sentence. The tail is the honest cost of a hard real-time pipeline: a small fraction of cues land visibly late, on the right word but behind it, which is the tradeoff the strict matching takes on purpose, a late gesture over a wrong one.
- Sync accuracy: 283 ms median cue-to-word lag (1.48 s p95) over the first ten hours of per-cue instrumentation, across 2,221 executed cues. Every cue that fired executed cleanly across all five action types; separately, about 5.8% of authored cues never fired at all, dropped when the tracker lost the cursor rather than firing on the wrong word, the fail-closed default at work.
- Push acceptance: all 3,943 text pushes over roughly 37 hours passed the runway-and-clock gate, none rejected for a stale player clock or an over-full buffer.
- Pipeline speed: pushed text becomes tracked speech in a median of 2.85 seconds (13.18 s p95, n=1,805). This is not what the viewer waits for: the audience hears a word only after a deliberate 20 to 35 second pacing buffer, the show's designed liveness depth, not latency.
- Session discipline: the scheduler restarted the stack 22 times in a row at 50.3 to 50.4 minutes each (a standard deviation of about three seconds), a clean scheduled cadence with no crash restarts in the last twenty.
- Interactivity: a viewer's GitHub link reaches an on-air acknowledgement in a median of about two minutes, fastest observed 17.5 seconds. That figure is bounded by where the request lands in the current episode, not by pipeline speed.
What else this builds
The repo-roaster is one use of the underlying primitive: a photorealistic face driven by text that also exposes the exact timing of every word it speaks. The same two streams aim at plenty of other action sets, a sports co-caster whose replays cut on the words as it calls them, a support agent whose cursor lands on "Settings" as it says the word, a tutor whose on-screen highlights track the explanation. None of them need a new API; they need the video and the timestamps beside it, aimed at a different set of actions.
The code
The code is open source (github.com/heygen-com/live-streamer), MIT-licensed. The reusable core is small: the word_tracker, the anchor math, and the runway gate. Read it as the real thing rather than a cleaned-up sketch.
The general idea outlasts this particular show: find the clock that tells the truth about the thing you care about, here the spoken word, and make every other clock do the work of meeting it. The Realtime API hands you that clock directly. What you bind to it is open.