The problem
An audio podcast mostly has to agree with one thing: its timeline. Video has a harder job. Every time we cut away and come back, what we see has to agree with what was there before.
A clip can look perfectly fine on its own and still look wrong in the episode. The guest's mouth moves while the host is speaking. The room changes slightly between two shots. A small timing error becomes visible after enough cuts.
We call each uninterrupted stretch of dialogue a turn. Each turn has its own audio, its own video, and its own place in the final timeline.
Before rendering, those turns share the same context. They're written as one conversation, use the same characters, and draw from the same scene references. Once rendering starts, though, each turn is independent; it doesn't get to see what another turn produced.
That independence is useful for the pipeline, but it creates the continuity problem behind Video Podcast: making independently generated clips hold together over the length of an episode.
The basic approach
Video Podcast has two phases: Preview and Render. Preview creates the dialogue and assets the user can inspect, edit, and regenerate. They stay mutable until the user confirms the episode.
Render starts after confirmation. The clips are generated independently, then brought back together during final assembly.
For the parts of continuity we can measure, the pipeline follows a simple pattern:
generate → measure → accept | repair
Timing: audio as the master clock
Timing problems showed up early. If a rendered turn ends slightly later than the audio slot it was meant to fill, the error carries forward. In one prototype we ended up about 200 ms off over the course of an episode before aligning the pauses between turns.
Loudness caused a different problem. TTS could give us host turns around −15 LUFS and guest turns at −26 to −29 LUFS. LUFS is a standardized measure of perceived loudness, and a gap that large is immediately noticeable once those turns are cut together.
Our lip-sync service renders at 25 fps, or one frame every 40 ms. Every video duration has to land on that grid, so we calculate the frame-aligned duration once and use it throughout the rest of the pipeline.
We store it as visual_dur_s: the exact amount of time the turn has to occupy in the finished video. Lip-sync rendering, cutaway planning, and assembly all work from that same value rather than interpreting the duration independently.
The audio for each turn is padded with silence until it lands on a whole-frame boundary, and inter-turn pauses use the same grid. During assembly, clip boundaries are calculated from the running timeline rather than repeatedly rounding individual durations. Loudness normalization happens without changing the timing.
Audio ends up driving the timeline because silence is easy to add. Half a video frame isn't.
Camera consistency
An episode keeps returning to the same few camera positions. Those views need to agree with one another.
Generate them independently and small differences become obvious in the edit: the room changes, the lighting shifts, or a character looks slightly different. Each view can be good on its own and still fail once we cut between them.
Three separate generations
Each shot generated on its own
Wide shot
Host closeup
Guest closeupOne continuous generation
Three shots cut from a single generated scene
Wide shot
Host closeup
Guest closeupBoth rows use the same character references and the same short room description. Each image is the first result returned, with no reselection.
Instead, we generate the camera views from the same scene context so the model has one room and one cast to work with.
Once the generation comes back, we check that it has the structure we expected. A multimodal check then verifies that each view contains the right subject and is usable for that camera position.
We also avoid making conventional person detection the authority here because the cast isn't always human. Statues, robots, creatures, and other non-human characters can all be podcast hosts.
If a camera position doesn't produce anything good enough to use, we'd rather regenerate than pick the least bad option and ship it.
Generating the views together doesn't guarantee consistency. It does avoid asking multiple independent generations to reconstruct the same room and characters from scratch.
The opening shot: repairing generated audio
Most turns follow the same path: take a portrait, pair it with synthesized speech, and render a lip-synced clip.
The opening is different. It starts from the shared scene and is animated separately so the episode opens with more natural motion and both speakers in frame.
That generation also creates audio, which introduces two problems. The words are usually the ones we asked for, but the model invents its own voice instead of using the host the user selected. And because the generated opening extends beyond the scripted line, the model can sometimes finish our text and keep talking.
We fix the first problem with voice conversion. Rather than synthesizing the line again from text, we re-voice the generated performance using the selected host voice. That keeps the cadence and timing of the original performance.
We do rely on one assumption here: that voice conversion preserves timing closely enough for the new audio to remain aligned with the generated mouth motion. We don't independently verify that afterward.
The extra dialogue is easier to handle because we already know what the host was supposed to say. We align the expected text with the generated audio, find where the intended line ends, trim the extra material, and put the result back on the same frame grid used by the rest of the episode.
We could keep pushing on the prompt and ask the model to get the voice, script ending, and timing right in one shot. In practice, keeping a good generated performance and repairing the parts we can measure has been more reliable.
Listener cutaways: detecting unwanted speech
Some turns cut briefly to the listener while the speaker's audio continues underneath. The listener needs to look alive without looking like they're talking.
A cutaway from a generated episode. The speaker's audio runs underneath throughout, while the picture holds on the listener for four seconds before returning.
The safest version is easy to generate: produce a listener clip with no speech-driven mouth movement. The problem is that this can look stiff, so we refine it to add more natural motion.
That's where the risk comes in. The refinement step doesn't really understand that this person is listening and must not speak. It sees someone in a conversation and is asked to make the footage more lifelike. Sometimes “more lifelike” includes moving their mouth as if they're talking.
So we check for that after generation.
We sample the clip and measure how much the listener's mouth moves over time. If the movement crosses our threshold, the clip is flagged. We can replace it with another usable listener clip or fall back to the safer, less animated version.
The check has blind spots. If we can't reliably measure the mouth, the clip currently comes back unflagged. It also looks for changes in mouth movement, which means some unusual cases can still pass.
So unflagged doesn't mean correct. It only means this particular check didn't observe the failure mode it was designed to catch.
Confirmation is a commit point
vulnerable windowa regeneration here could rewrite a key mid-copyreconcileguardscopy portraits+ per-turn audiore-read+ comparedispatchrender12345step 4 re-reads and compares, closing the window1 reconcile: apply regenerations that finished but weren’t seen2 guards: reject if any regeneration is still running3 copy: into the run’s own immutable location, not session keys4 re-read row + compare snapshots5 dispatch render
Preview is designed to be mutable; Render can't be.
Say a user regenerates an asset and immediately hits Confirm. That regeneration may still be finishing while the render is preparing to start. If rendering begins against changing inputs, different parts of the pipeline can end up seeing different versions of the same episode.
Confirmation is where we turn that mutable preview into a fixed render snapshot.
First, we incorporate any regeneration work that has already finished and reject confirmation if something is still actively changing. Then we pin the portraits and audio that this render will use.
There is still a small window where something could change while that snapshot is being created, so once it's complete we compare it with the current preview state. If anything moved underneath us, confirmation fails rather than starting a render from inconsistent inputs.
The ordering is enough: settle finished work, reject active work, pin the inputs, then verify that they stayed stable while we did it.
Failure handling
Different failures get different responses.
If something outside the pipeline may recover on its own, we retry. If the input was valid but generation produced a bad sample, we regenerate. If the failed piece is optional and the episode still works without it, we drop it.
Listener cutaways are the clearest example. The speaker footage already fills the entire turn, so a cutaway can fail without breaking the episode. We simply leave it out.
Most other pieces don't have that property. A clip that can't fill its place in the timeline, a missing required portrait, or a malformed opening can't simply disappear. Those failures stop the render.
What survives better models
The models improved noticeably while we were building this, and several of these failure cases already happen less often than they did in the first versions of the pipeline.
That helps, but the underlying constraints don't change.
A 25 fps render still has 40 ms frames. Preview inputs can still change while someone is confirming them. Independently generated views can still disagree. And if we ask a model to make a silent listener look more natural, we still need some way to know whether “natural” crossed into “talking.”
If these checks end up firing less and less often, great.
We'd still keep them.
A cutaway from a generated episode. The speaker's audio runs underneath throughout, while the picture holds on the listener for four seconds before returning.
The safest version is easy to generate: produce a listener clip with no speech-driven mouth movement. The problem is that this can look stiff, so we refine it to add more natural motion.
That's where the risk comes in. The refinement step doesn't really understand that this person is listening and must not speak. It sees someone in a conversation and is asked to make the footage more lifelike. Sometimes “more lifelike” includes moving their mouth as if they're talking.
So we check for that after generation.
We sample the clip and measure how much the listener's mouth moves over time. If the movement crosses our threshold, the clip is flagged. We can replace it with another usable listener clip or fall back to the safer, less animated version.
The check has blind spots. If we can't reliably measure the mouth, the clip currently comes back unflagged. It also looks for changes in mouth movement, which means some unusual cases can still pass.
So unflagged doesn't mean correct. It only means this particular check didn't observe the failure mode it was designed to catch.
Confirmation is a commit point
Preview is designed to be mutable; Render can't be.
Say a user regenerates an asset and immediately hits Confirm. That regeneration may still be finishing while the render is preparing to start. If rendering begins against changing inputs, different parts of the pipeline can end up seeing different versions of the same episode.
Confirmation is where we turn that mutable preview into a fixed render snapshot.
First, we incorporate any regeneration work that has already finished and reject confirmation if something is still actively changing. Then we pin the portraits and audio that this render will use.
There is still a small window where something could change while that snapshot is being created, so once it's complete we compare it with the current preview state. If anything moved underneath us, confirmation fails rather than starting a render from inconsistent inputs.
The ordering is enough: settle finished work, reject active work, pin the inputs, then verify that they stayed stable while we did it.
Failure handling
Different failures get different responses.
If something outside the pipeline may recover on its own, we retry. If the input was valid but generation produced a bad sample, we regenerate. If the failed piece is optional and the episode still works without it, we drop it.
Listener cutaways are the clearest example. The speaker footage already fills the entire turn, so a cutaway can fail without breaking the episode. We simply leave it out.
Most other pieces don't have that property. A clip that can't fill its place in the timeline, a missing required portrait, or a malformed opening can't simply disappear. Those failures stop the render.
What survives better models
The models improved noticeably while we were building this, and several of these failure cases already happen less often than they did in the first versions of the pipeline.
That helps, but the underlying constraints don't change.
A 25 fps render still has 40 ms frames. Preview inputs can still change while someone is confirming them. Independently generated views can still disagree. And if we ask a model to make a silent listener look more natural, we still need some way to know whether “natural” crossed into “talking.”
If these checks end up firing less and less often, great.
We'd still keep them.