1. Introduction: The Rise of Code-to-Video
Programmatic video rendering (code-to-video) is rapidly becoming the standard representation for motion graphics generation. Instead of outputting heavy, non-editable pixel streams, modern AI architectures generate clean, structured code — such as HTML or React — which renders deterministically into high-definition videos.
The open-source community's enthusiasm confirms this shift — and HyperFrames is setting the pace. Our recently released open-source framework, built specifically for AI agents, pioneered HTML-to-video and has already climbed to 50k+ GitHub stars in just a few months.
What is the Code-to-Video Task?
The code-to-video task is straightforward: given a prompt or visual narrative, an LLM agent generates code that a rendering framework or engine (e.g., HyperFrames) can deterministically render into the final video.
While modern Large Language Models (LLMs) are exceptionally proficient at writing valid code, they struggle with motion graphics and visual intent. Writing code that executes without errors is fundamentally different from crafting motion design that looks compelling, balanced, and professionally animated.
Prompt
Figma product-launch HOOK — macro flower blossom pull-back that reveals a plant-guide web page, then the design tool building it, 16:9, ~6.8s. Palette: dark forest green (#1B3A28) page background, lime green (#C5ED76) card, pink (#F886B1) starburst, white petals, light-grey (#F5F5F5) tool chrome. Silent.
CRITICAL — camera: one continuous rapid zoom-out (0–4.5s) with exponential deceleration, then a locked static frame (4.5s–end) while only canvas elements animate.
Scene 1 (0.0–2.5s): extreme out-of-focus macro — a pale grey-green field with a blurred dark forest-green stem rising, a glowing white circle at its tip. As the camera pulls back over a lime-green field, more soft white circles pop in and expand outward, resolving into a flat vector flower: five-plus white circular petals around a dark green asterisk/cross stem center. Blur sharpens as the zoom recedes.
Scene 2 (2.5–3.7s): the pull-back reveals the flower lives in a lime-green rounded card, the center of a three-card row on a dark green web page. Serif header "The Latest" above the cards; top nav "Modern Garden" left, links "Explore Guides Collections Shop" right. Each card has a dark green lower band with a small circular ↗ button: left card — pink many-pointed starburst flower, "12 min" / "Plant Care 101"; center — the white-circle flower, "15 min" / "Light Guide"; right — white card with grey/green semicircle pattern and a pink-cored circle, "8 min" / "Seed Starting". Card titles serif white, durations small grey sans.
Scene 3 (3.7–4.5s): the zoom keeps pulling back and reveals the whole web page is an artboard on the canvas of a Figma-style design tool: light-grey chrome, left panel ("The Modern Garden" file, Pages, Layers list with Flower 1 / Flower 2 / Flower 3), right properties inspector (Motion, Animation style, Transform, Layout, Appearance, Fill), a blue selection box on the canvas, and a bottom animation timeline with layer tracks, colored keyframe/duration bars and a millisecond ruler. The zoom eases to a stop framing the full app window.
Scene 4 (4.5–6.8s): frame locked. A blue playhead scrubs left-to-right across the bottom timeline, and the card artwork on the canvas animates in sync: the center white-petal flower rotates about its hub, the left pink starburst spins in place, and the right card's circle motif shifts/changes (a lime-yellow form sweeping in). Ends mid-playback.
Motion summary: blurred macro stem → white petal circles bloom outward while one continuous zoom-out resolves flower → card → "The Latest" three-card web page → full design-tool workspace; camera locks, then timeline playhead scrubs and the three flower graphics animate live on canvas.
Prompt
Key feature — AI website-builder prompt, 16:9, ~7.5s. An AI orb + prompt bar generate a website. Silent.
CRITICAL — camera: no real camera. The orb + input bar scale/expand (element). Static frame.
Scene 1 (0.0–~1s): solid black; a neon purple/blue sparkle icon (two 4-point stars) inside a thin circular outline appears centered. The bg fades to a dark blurred gradient (deep orange/red on the left, dark blue on the right). The sparkle fills solid white; the ring becomes a dark sphere with a colorful glowing rim.
Scene 2 (~1–4s): the sphere shifts left and shrinks a little; a dark-grey (#2A2A2A) pill-shaped input bar expands to the right — "Ask AI Create anything" → the user types "Design a website for SpaceX starship".
Scene 3 (~4–7.5s): the prompt reads "Design a website for SpaceX starship" with a "Generating…" state.
Motion summary: neon sparkle orb forms on black → dark orange/blue gradient → "Ask AI" pill bar expands, type "Design a website for SpaceX starship" → "Generating…". Element-level scale/expand + typing; no camera move.
2. The Core Problem: Code Accuracy vs. Visual Intent
When an LLM writes the code for a video, the generated code often passes syntax checks and renders fine. Yet, the output frequently feels flat, amateurish, or visually jarring.
LLMs currently suffer from a lack of visual intent across several core design dimensions:
- Animation Timings & Easing: Models tend to use linear transitions or abrupt delays instead of organic easing curves (power2.out, cubic-beziers, staggered entrances).
- Spatial Hierarchy & Composition: Text overlays clash with background contrast, fonts lack typographic scale, and elements crowd the margins or overlap awkwardly.
- Motion Pacing & Rhythm: Animations often trigger simultaneously rather than building a narrative cadence (e.g., hook → main message → call-to-action).
- Visual Polish & Craft: Fine details — such as subtle shadow depth, micro-interactions, or cohesive color palettes — are frequently omitted.
In response, we — and the broader community — have built skills, agent harnesses, and other scaffolding on top of base LLMs: system prompts stuffed with design rules, multi-turn and/or multi-agent review loops, retrieval of reference compositions, style-guide injection. These wrappers do move the needle over the base model. But they come with real tradeoffs:
- Increased Token Cost: Stacking on more skills, rules, and retrieval — whether in a single agent or spread across a multi-agent harness — means more tokens, more model calls, and more latency spent per render.
- Engineering overhead: Each harness needs its own prompts, guardrails, and test suite to keep it from silently regressing.
- A steeper learning curve: The scaffolding that improves output also makes the tooling harder for new users and beginners to configure and reason about.
- A ceiling, not a fix: Harnesses reliably beat the base model, but getting a result that holds up beyond a demo still takes a lot of per-video human touch and design direction.
3. Evaluating Generated Motion Graphics
Why Evaluating Generated Video is Uniquely Hard
Evaluating motion design presents a unique challenge:
- Human Evaluator Inconsistency: While an average viewer can immediately sense that a video "looks bad," they usually lack the motion design vocabulary to explain why (e.g., "the animation timing feels off" or "the layout just isn't nice").
- Off-the-Shelf Vision-Language Model Failure Modes: General-purpose Vision-Language Models (VLMs) can inspect frame sequences, but their judgment of temporal flow, composition balance, and aesthetic polish is inconsistent. They routinely miss frame-level glitches or hallucinate visual flaws that do not exist. They can also carry an unintentional self-preference bias toward generations from their own model family — e.g., a Gemini VLM judging a Gemini-generated video — and a position bias favoring whichever video is placed first in their context.
- Traditional Video Metrics Don't Transfer: FVD (Fréchet Video Distance) scores a batch of generated clips against real ones distributionally, not one composition against its brief — and tends to track per-frame quality more than motion. CLIPScore checks whether a frame matches the prompt semantically, with no sense of timing, composition, or design polish; a jarring, badly-timed render can still score well.
| Which is better? | A | B |
|---|---|---|
| VLM1 | 1 | 3 |
| CLIPScore | 0.706 | 0.734 |
| FVD | 0.37 | 0.38 |
| Human | 5 | 0 |
A New Evaluation Metric
The 5-Axis Evaluation Framework
Rather than relying on a single overall score, every composition is evaluated across five distinct visual and design axes:
- Engagement — Visual impact, scroll-stopping quality, aesthetic appeal, and hook strength.
- Prompt-Intent — How accurately the generated video fulfills the prompt requirements, messaging, and branding constraints.
- Composition — Spatial layout balance, contrast, typography selection, hierarchy, and visual harmony.
- Temporal — Animation timing, easing curves, transition smoothness, cadence, and overall rhythm.
- Craft — Execution quality, absence of visual clipping, proper asset alignment, and clean rendering.
Evaluation Model (The Judge)
Identifying the gap in current metrics is the easy part. How to address it is more difficult. To tackle the gap, we designed and trained a new evaluation model — not a fine-tuned LLM/VLM, but our own model architecture focused on evaluating these axes in video. This new Judge model learns from human preference data for each axis and is trained to predict the likelihood of a video being preferred by humans. Annotators were trained and screened on motion-design fundamentals before rating, so the preference data itself stays high quality.
Give it the original prompt and two candidate videos, and it returns P(A): the probability a human would prefer Video A over Video B. It does this once per axis — five independent verdicts, not one blended score.
General-purpose VLMs can technically attempt this same comparison, and linters can confirm the code runs, but neither is built for this specific judgment call. Measured against both:
Evaluator comparison
| Evaluator | Per-axis A/B verdict | Human alignment | Latency | Cost per comparison |
|---|---|---|---|---|
| Code Linters | None | N/A | <1s | Negligible |
| VLMs | Inconsistent | 75% | Several seconds | High |
| Our Judge | High | 82% | <1s | Low |
Confidence and human agreement
| Judge confidence |P − 0.5| | Agreement with human raters |
|---|---|
| < 0.25 | 75.7% |
| 0.25 – 0.40 | 95.2% |
| ≥ 0.40 | 98.8% |
Together, this shows the judge model is well aligned with human evaluation (across 1,464 samples) — the more confident it is, the more closely it matches human judgment.
Judge Model Architecture
We will be following up with a technical blog diving into deeper detail on the judge model architecture and its training and evaluation processes.
Evaluation Comparison Examples
Three held-out test pairs, never seen in training, one per axis. In each, the VLMs get it wrong or cannot decide, while the judge is more tuned towards the human preference.
| Axis | VLM Vote1 | Our judge | Human |
|---|---|---|---|
| Engagement | 2-2 | 76% of people would vote B | 1-4 |
| Axis | VLM Vote1 | Our judge | Human |
|---|---|---|---|
| Composition | 2-2 | 75% of people would vote A | 4-1 |
| Axis | VLM Vote1 | Our judge | Human |
|---|---|---|---|
| Temporal | 2-2 | 94% of people would vote A | 5-0 |
1 VLM vote counts above are always even: because of the position bias noted above, each pair is judged twice with the video order swapped. The voting VLMs were Gemini 3.5 and GPT-5.5.
4. The Code2Video Benchmark Dataset
Along with our new evaluation metric and judge model, we created a benchmark of 168 curated, human-created motion graphic videos, each one a beat of a real product-launch video, across the categories that LLMs still struggle to produce via code:
- Hook
- Problem
- Product intro
- Key feature
- Benefits
- Social proof
- Call to action
- Brand outro
5. Benchmark Results and Model Leaderboard
We evaluated leading LLMs by giving them the HyperFrames skill, running them through the 168 benchmark prompts, rendering their code via HyperFrames, and then subjecting the results to round-robin pairwise evaluations against both human reference videos and competing models, using our judge model.
We report Elo rather than a score against the human reference. Measured against the reference, almost every model scores near zero and the few that do not sit only slightly above it, which tells model developers little. Pairwise comparison shows how a model stands against its competitors and against its own previous version. A release that moves from #14 to #10 is visible progress, even while every model still loses to the reference.
Below are the overall ELO rankings across the evaluation axes:
Three things surprised us:
- No clear winner. The top four sit within 12 Elo of each other and their intervals overlap. The newest OpenAI model did not pull away from its predecessor.
- Open-weight models are close. qwen3.8-max, deepseek-v4.1-flash, kimi-k3 and glm-5.3 all land within 50 Elo of first place.
- Following the brief is nearly solved; craft is not. Models are closest on Prompt-Intent and furthest apart on Composition and Craft.
Leaders by evaluation axis
Five-model comparison, ranked independently within each axis.
Engagement
gpt-5.5#11578Elo- gpt-6-astra#21576Elo
claude-opus-5#31571Elo- gpt-5.6-sol#41564Elo
muse-spark-1.3#51557Elo
Prompt-Intent
- gpt-6-astra#11577Elo
- gpt-5.5#21575Elo
- claude-opus-5#31570Elo
- gpt-5.6-sol#41562Elo
- muse-spark-1.3#51556Elo
Composition
- gpt-5.5#11585Elo
- gpt-6-astra#11585Elo
- claude-opus-5#31576Elo
- gpt-5.6-sol#41572Elo
- muse-spark-1.3#51561Elo
Temporal
- gpt-5.5#11579Elo
- gpt-6-astra#21578Elo
- gpt-5.6-sol#31568Elo
- claude-opus-5#41565Elo
- muse-spark-1.3#51555Elo
Craft
- gpt-6-astra#11586Elo
- gpt-5.5#21585Elo
- claude-opus-5#31574Elo
- gpt-5.6-sol#41572Elo
- muse-spark-1.3#51560Elo
Example Generations
Four pairs from the arena, each making one point: the spread between the best and worst model on a single brief (Example 7), how little separates the leaders (Example 8), an older model beating its successor (Example 9), and a slip by the leader (Example 10).
| Axis | Our judge | Human |
|---|---|---|
| Engagement | 98% of people would vote claude-opus-5 | 5-0 for claude-opus-5 |
| Prompt-Intent | 98% of people would vote claude-opus-5 | 5-0 for claude-opus-5 |
| Composition | 98% of people would vote claude-opus-5 | 5-0 for claude-opus-5 |
| Temporal | 97% of people would vote claude-opus-5 | 5-0 for claude-opus-5 |
| Craft | 98% of people would vote claude-opus-5 | 5-0 for claude-opus-5 |
| Axis | Our judge | Human |
|---|---|---|
| Engagement | 50% of people would vote A | 3-2 |
| Prompt-Intent | 50% of people would vote A | 3-2 |
| Composition | 51% of people would vote A | 3-2 |
| Temporal | 52% of people would vote A | 3-2 |
| Craft | 51% of people would vote A | 3-2 |
| Axis | Our judge | Human |
|---|---|---|
| Engagement | 57% of people would vote B | 2-3 |
| Prompt-Intent | 54% of people would vote B | 5-0 |
| Composition | 55% of people would vote B | 4-1 |
| Temporal | 52% of people would vote B | 5-0 |
| Craft | 55% of people would vote B | 2-3 |
Key Insights & Common Failure Modes
Our benchmark revealed distinct patterns in how frontier LLMs approach code-to-video authoring:
- Motion is the weak axis. Models hit every beat in the brief, but the timing between beats breaks.
- Text collides. Text is laid out for the final frame, not for the path it travels.
- Safe means flat. When unsure, models shrink everything: readable, but nothing has presence.
Examples 11 and 12 compare two models on the same brief, with rater votes and judge scores. Examples 13 and 14 set a model against the human-authored reference to show the pattern; the reference is not a leaderboard entry, so those pairs carry no scores.
Follows the brief, misses the timing
Following the brief means reproducing what the prompt asks for: the words, the colors, the order of beats. Most frontier models now do this well. What they miss is timing: how long each element takes to arrive, and whether it settles before the next one moves. The result performs every step of the brief and still feels rough.
Prompt
Google DeepMind product-launch PRODUCT_INTRO — Gemini Code Assist wordmark reveal, 16:9, ~4.7s. White (#FFFFFF) background for the wordmark beats, then a soft blurred pastel mesh; Google-gradient type (blue #4285F4 → purple → pink/magenta ~#E91E63); camera static — motion is letter assembly, gradient shifts, and one zoom-blur text transition. Silent.
Scene 1 (0.0–1.0s): on white, oversized gradient letters assemble into the word "Gemini" — the opening frame is dominated by a huge blue "G" with an "e" tucked low beside it; the remaining letters bounce in and settle onto the baseline while the whole word scales down. The result is a large, bold rounded sans "Gemini" with a playful stagger ("mini" letters still springing) and a left-to-right blue→pink gradient.
Scene 2 (1.0–2.5s): the word shrinks and "Code Assist" joins to its right, forming the centered title "Gemini Code Assist" in a smaller weight. The gradient fill animates across the phrase — hues sweep from pink/red through purple toward blue — and at ~2.5s the text settles to a solid near-black (#202124) on white.
Scene 3 (2.8–3.3s): transition — the dark "Gemini Code Assist" rapidly scales up and blurs out (zoom-blur dissolve) while the white background crossfades into a soft, out-of-focus animated gradient, first in hazy pale lime-green/yellow tones with a bright diamond-shaped glow at center.
Scene 4 (3.2–4.7s): over the drifting pastel background — which morphs from the pale yellow-green into misty pale purple-blue — the line "Hello, how can I help?" types on character by character, centered, in a medium-weight sans with the blue→pink gradient mapped across it ("Hello," blue, "help?" pink/red). The finished line holds over the gently undulating glow as the clip ends.
Motion summary: scattered oversized gradient letters snap into "Gemini" and scale down → "Code Assist" appends, gradient hue-sweeps then snaps to near-black → dark title zoom-blurs away as a pastel aurora fades in → "Hello, how can I help?" types on in gradient and holds. No camera movement; all element-level.
Prompt
Google DeepMind product-launch PRODUCT_INTRO — Gemini Code Assist wordmark reveal, 16:9, ~4.7s. White (#FFFFFF) background for the wordmark beats, then a soft blurred pastel mesh; Google-gradient type (blue #4285F4 → purple → pink/magenta ~#E91E63); camera static — motion is letter assembly, gradient shifts, and one zoom-blur text transition. Silent.
Scene 1 (0.0–1.0s): on white, oversized gradient letters assemble into the word "Gemini" — the opening frame is dominated by a huge blue "G" with an "e" tucked low beside it; the remaining letters bounce in and settle onto the baseline while the whole word scales down. The result is a large, bold rounded sans "Gemini" with a playful stagger ("mini" letters still springing) and a left-to-right blue→pink gradient.
Scene 2 (1.0–2.5s): the word shrinks and "Code Assist" joins to its right, forming the centered title "Gemini Code Assist" in a smaller weight. The gradient fill animates across the phrase — hues sweep from pink/red through purple toward blue — and at ~2.5s the text settles to a solid near-black (#202124) on white.
Scene 3 (2.8–3.3s): transition — the dark "Gemini Code Assist" rapidly scales up and blurs out (zoom-blur dissolve) while the white background crossfades into a soft, out-of-focus animated gradient, first in hazy pale lime-green/yellow tones with a bright diamond-shaped glow at center.
Scene 4 (3.2–4.7s): over the drifting pastel background — which morphs from the pale yellow-green into misty pale purple-blue — the line "Hello, how can I help?" types on character by character, centered, in a medium-weight sans with the blue→pink gradient mapped across it ("Hello," blue, "help?" pink/red). The finished line holds over the gently undulating glow as the clip ends.
Motion summary: scattered oversized gradient letters snap into "Gemini" and scale down → "Code Assist" appends, gradient hue-sweeps then snaps to near-black → dark title zoom-blurs away as a pastel aurora fades in → "Hello, how can I help?" types on in gradient and holds. No camera movement; all element-level.
| Axis | Our judge | Human |
|---|---|---|
| Engagement | 70% of people would vote B | 0-5 |
| Prompt-Intent | 67% of people would vote B | 1-4 |
| Composition | 59% of people would vote B | 3-2 |
| Temporal | 51% of people would vote A | 5-0 |
| Craft | 61% of people would vote B | 2-3 |
Motion without easing
Easing is the change in speed that makes motion feel natural: a quick start that slows into place, a pause before the next move. Without it, motion is either mechanical or unstable: elements move at constant speed, snap into position, or twitch from frame to frame instead of gliding.
| Axis | Our judge | Human |
|---|---|---|
| Engagement | 52% of people would vote A | 4-1 |
| Prompt-Intent | 55% of people would vote A | 2-3 |
| Composition | 59% of people would vote A | 3-2 |
| Temporal | 58% of people would vote A | 5-0 |
Text that collides as it moves
Text length changes with the copy, and models lay text out for its final position rather than for the path it travels. When words animate in from different directions, they overlap mid-motion before they separate.
Prompt
revid.ai product-launch CTA — kinetic typography, 16:9, ~3.6s. Dark-charcoal #161A1A background, bold centered sans-serif; neon-green #4ADE80 + red #F43F5E accents.
Scene 1 (0.0–1.5s): "Boost your engagement" — "Boost your" is white #FFFFFF; "engagement" starts offset to the right in mid-grey #888888 and springs left to snap beside it while its color tweens grey→white. Behind the text, a neon-green rising line-graph stroke draws on (trim-path) from the bottom-left, swooping up to the right.
Scene 2 (1.5–2.0s): the whole scene scales down + pans up out of the top of frame; a red #F43F5E low-battery outline icon (one charge bar) drops in from the top with a spring/overshoot into the upper-center.
Scene 3 (2.0–3.6s): "& turbocharge" then "your social media" reveal via bottom-up masked slides; the final word "media" starts grey offset-right and springs left, tweening grey→white, completing the lockup. Holds.
Motion summary: kinetic text + green graph trim-path → zoom-out/pan transition + battery-icon drop → masked-reveal CTA lines. Silent. Pure 2D motion graphics — reproducible in HyperFrames.
Prompt
revid.ai product-launch CTA — kinetic typography, 16:9, ~3.6s. Dark-charcoal #161A1A background, bold centered sans-serif; neon-green #4ADE80 + red #F43F5E accents.
Scene 1 (0.0–1.5s): "Boost your engagement" — "Boost your" is white #FFFFFF; "engagement" starts offset to the right in mid-grey #888888 and springs left to snap beside it while its color tweens grey→white. Behind the text, a neon-green rising line-graph stroke draws on (trim-path) from the bottom-left, swooping up to the right.
Scene 2 (1.5–2.0s): the whole scene scales down + pans up out of the top of frame; a red #F43F5E low-battery outline icon (one charge bar) drops in from the top with a spring/overshoot into the upper-center.
Scene 3 (2.0–3.6s): "& turbocharge" then "your social media" reveal via bottom-up masked slides; the final word "media" starts grey offset-right and springs left, tweening grey→white, completing the lockup. Holds.
Motion summary: kinetic text + green graph trim-path → zoom-out/pan transition + battery-icon drop → masked-reveal CTA lines. Silent. Pure 2D motion graphics — reproducible in HyperFrames.
Legible, but flat
When a model is unsure how to fill a frame, it plays safe: small type, thin lines, faint backgrounds, wide margins. Everything stays legible and nothing has presence. The brief may ask for a map that fills the frame or cards that carry weight; the model delivers a smaller, quieter version of each.
Prompt
Finally CRM product-launch KEY_FEATURE — real-time campaign tracking map, 16:9, ~4.2s. Clean white background, a light-gray dotted world map in curved perspective filling the lower two-thirds, dark bold sans-serif title, white contact cards with colored status pills; camera static throughout. Silent.
Scene 1 (0.0–1.0s): on white, a world map rendered entirely from small light-gray dots sits in an arched 3D-like perspective across the lower frame, dotted with five colored marker dots that pulse softly (expanding fading rings): yellow mid-left, black bottom-left, blue top-center, red mid-right, magenta/pink bottom-right. The bold dark title "Real-time campaign tracking" fades in centered at the top. Five white contact cards spring in one after another from their markers (bouncy scale-up, each tethered to its dot by a thin gray line). Each card holds a small circular avatar photo, a bold name, and a rounded status pill: "Justin Trudeau — Delivered" (yellow dot), "Lula Da Silva — Delivered" (black dot), "Emmanuel Macron — Delivered" (blue dot), "Vladimir Putin — Bounced" (red dot; light-red pill with dark-red text), "Olaf Scholz — Delivered" (pink dot). "Delivered" pills are light gray with dark-gray text.
Scene 2 (1.0–3.0s): status pills update live with quick snappy swaps: Lula Da Silva flips "Delivered" → "Opened" (~1.3s, light-blue pill, dark-blue text); Olaf Scholz flips to "Opened" (~2.4s); then in quick succession Emmanuel Macron flips to "Opened" and Lula Da Silva updates again to "Replied" (~2.9s, light-green pill, dark-green text with tiny accent dots beside the word). Vladimir Putin stays "Bounced" throughout.
Scene 3 (3.0–4.2s): hold on the final state — Trudeau Delivered, Lula Replied, Macron Opened, Putin Bounced, Scholz Opened — while the map markers keep pulsing.
Motion summary: static camera; motion is the pulsing rings on the five map dots, the spring pop-in of the five tethered cards, and three-plus quick status-pill swaps (gray Delivered → blue Opened → green Replied).
Prompt
Finally CRM product-launch KEY_FEATURE — real-time campaign tracking map, 16:9, ~4.2s. Clean white background, a light-gray dotted world map in curved perspective filling the lower two-thirds, dark bold sans-serif title, white contact cards with colored status pills; camera static throughout. Silent.
Scene 1 (0.0–1.0s): on white, a world map rendered entirely from small light-gray dots sits in an arched 3D-like perspective across the lower frame, dotted with five colored marker dots that pulse softly (expanding fading rings): yellow mid-left, black bottom-left, blue top-center, red mid-right, magenta/pink bottom-right. The bold dark title "Real-time campaign tracking" fades in centered at the top. Five white contact cards spring in one after another from their markers (bouncy scale-up, each tethered to its dot by a thin gray line). Each card holds a small circular avatar photo, a bold name, and a rounded status pill: "Justin Trudeau — Delivered" (yellow dot), "Lula Da Silva — Delivered" (black dot), "Emmanuel Macron — Delivered" (blue dot), "Vladimir Putin — Bounced" (red dot; light-red pill with dark-red text), "Olaf Scholz — Delivered" (pink dot). "Delivered" pills are light gray with dark-gray text.
Scene 2 (1.0–3.0s): status pills update live with quick snappy swaps: Lula Da Silva flips "Delivered" → "Opened" (~1.3s, light-blue pill, dark-blue text); Olaf Scholz flips to "Opened" (~2.4s); then in quick succession Emmanuel Macron flips to "Opened" and Lula Da Silva updates again to "Replied" (~2.9s, light-green pill, dark-green text with tiny accent dots beside the word). Vladimir Putin stays "Bounced" throughout.
Scene 3 (3.0–4.2s): hold on the final state — Trudeau Delivered, Lula Replied, Macron Opened, Putin Bounced, Scholz Opened — while the map markers keep pulsing.
Motion summary: static camera; motion is the pulsing rings on the five map dots, the spring pop-in of the five tethered cards, and three-plus quick status-pill swaps (gray Delivered → blue Opened → green Replied).
6. A Training Signal for Models — and a Feedback Loop for Agents
An additional reason we created the judge evaluation model is to give model developers a path to RLAIF (Reinforcement Learning from AI Feedback) that is closely aligned with human judgment. Our judge model's cost, latency, and human alignment make it easy and reliable to help frontier models push the boundaries of visual intent and understanding.
This feedback loop can also be applied to agentic workflows on existing models as a feedback loop for agent developers or a self-refinement loop for the agents themselves.
7. Released in Partnership with Kaggle
We partnered with Kaggle to release the full Code2Video Benchmark, including all reference compositions, design briefs, and our evaluation judge API. Every model run in this report executed directly on the Kaggle platform, which provides the execution environment and maintains an auditable history of raw traces and logs. Kaggle hosts and audits these runs so the results stay open and verifiable. Additionally, Kaggle will maintain the leaderboard, adding new models as they launch.
Today, Kaggle gives us a clean way to ship Harbor-compatible datasets and custom evaluation harnesses across an initial set of models. Eventually, you will be able to bring your own models and run them against the exact same judge used in this benchmark. We hope this gives builders a shared home to test how well models write HyperFrames code and get inspired to create benchmarks of their own.
- See the leaderboard here: kaggle.com/benchmarks/heygen/code2video