One Model, Multiple Controllability: Composable Behavior LoRAs for Video Avatars

The hard part of giving an avatar model many controllable behaviors — expressiveness, gaze, camera motion, a composed professional register — is not the model. It is the data: a corpus where all of those behaviors co-occur, correctly labeled, at studio quality is rare, expensive, and in practice does not exist. This post shows how we skip that dataset entirely — train each behavior as a separate LoRA on cheap, abundant single-attribute data, compose the adapters at inference, and close the small train/inference gap with a short fine-tune on only a little real overlap.

Behavior control, by example

Before the how, here is the what — the same source identity and audio, changed only by which behavior adapter is switched on.

Expressive vs. professional

These two sit at opposite ends of the same axis, so they are never composed together — composition is for behaviors that are orthogonal, not antagonistic.

Gaze control — a functional adapter, composed

Gaze is directional, so here it is by example: the same identity and audio generated with the expressive LoRA alone vs. the expressive + gaze LoRAs composed.

The point is not just that gaze control works — gaze is a functional (control) LoRA, and it keeps working while composed on top of the expressive style LoRA. Composition does not blunt the functional behavior: the two adapters run together and each still does its job. How we get that — and why it is harder than it looks — is the rest of this post.

The problem: behaviors are cheap alone, expensive together

A production avatar model has to do many things at once. It should look expressive when the script is energetic, or carry a professional, composed demeanor — the serious, restrained counterpart to expressive — when the moment calls for it; follow a commanded gaze direction (straight at the camera, or a natural side glance); produce gestures on cue from the prompt; move the camera on command or hold a static shot; convey a requested emotion; and stay convincing against a dynamic background rather than only a static one. Each of these is a behavior we want to control on demand:

The naive way to get them is to collect one large dataset in which every behavior is present, correctly labeled, and simultaneously high quality — and train on it. In practice that dataset does not exist, and it is expensive to build. The behaviors are correlated with the source of the data:

• Data rich in one attribute is usually abundant and cheap in isolation — plenty of footage exhibits a specific gaze pattern, or strong expressive motion, or a particular camera behavior.
• Data that is simultaneously strong in several attributes and high quality is rare and expensive. The intersection is thin.
• Worse, the abundant single-attribute sources are often lower visual quality than our studio footage. Training the whole model on them directly would inject the behavior and drag down fidelity.

So we face a classic coverage-vs-quality tension: the data that teaches a behavior is not the data that preserves quality, and the data that has everything at once barely exists.

Why not just fine-tune one model on the union

The obvious fallback — pool every source and fine-tune the base model on the union — fails in two ways.

Entanglement. A single set of weights fine-tuned on mixed sources learns the behaviors as one inseparable bundle. You cannot later ask for “expressive but static camera” if those two attributes were never disentangled during training; you get whatever correlation the data happened to contain.

Quality contamination. Because the low-quality single-attribute sources are also the largest, they dominate the gradient. The model regresses toward their fidelity, and the studio-quality behavior we started from erodes.

We want the opposite: behaviors that are isolated (each can be dialed in independently), a base fidelity that is firewalled from the quality of any single behavior’s data, and the ability to add or retire a behavior without retraining the model.

The recipe at a glance

Our approach has three stages:

1. Base model. Train (or reuse) a strong, high-fidelity base avatar model. This defines the quality floor and is never degraded by the behavior data.
2. Per-attribute LoRA SFT. Freeze the base. For each behavior, attach a dedicated low-rank adapter (LoRA) and train it only on that behavior’s data. One dataset → one LoRA.
3. Compose at inference. At generation time, activate exactly the adapters whose behaviors the request calls for, each with its own strength.

Per-attribute LoRAs: one behavior, one adapter

A LoRA adapter replaces a frozen weight matrix’s contribution W·x with W·x + α·B(A(x)), where A and B are small low-rank matrices and α is a scalar strength. Only A and B train; W — the base model — stays frozen. We attach adapters to the attention projections (query / key / value / output of self-, text-cross-, and audio-cross-attention), the feed-forward layers, and the output heads, so an adapter has enough reach to change behavior without touching base capability.

Training each adapter on a single behavior’s data gives us exactly the properties the naive approach lacked:

• Isolation. Each adapter “owns” one behavior. Because the base is frozen, the adapter is a pure delta — the behavior it encodes is separable by construction.
• A quality firewall. Low-quality behavior data can only move the adapter, never the base weights. Base fidelity is preserved no matter how noisy a behavior’s source is.
• Cheap iteration. Adding a behavior is training one small adapter; retiring one is deleting a file. No behavior’s data can regress another, because they never share trainable parameters.
• Composability 🌟 Adapters are additive deltas on a shared frozen base, so in principle they can be combined at inference.

That last property is the goal — and the source of the problem we spend the rest of this post on.

Composing behaviors at inference

At inference we hand the model a small map of which behaviors to activate and how strongly — for example {expressive: 1.0, gaze‑control: 0.8}. The model applies the selected adapters simultaneously. Concretely, for every adapted layer the output is the base contribution plus the sum of the active adapters’ deltas:

This is elegant: one request can mix expressiveness with a specific gaze and studio polish, each independently dialed, all on top of one frozen base. Empirically, it works — the composed avatar does exhibit the requested combination of behaviors.

But look closely at that equation.

The train / inference gap

During per-attribute SFT, each training sample activates exactly one adapter — a clip from the expressiveness set trains the expressiveness adapter with every other adapter switched off, and so on. The model therefore only ever sees the single-adapter regime:

At inference we instead ask for the sum of several deltas that were each learned in isolation. Nothing during training ever exposed the model to the joint regime. The composed output is, quite literally, a linear superposition of independently-trained adapters — a first-order extrapolation into a region of behavior space the model was never trained on.

The core issue. Single-adapter training makes multi-adapter inference an out-of-distribution regime. The composition works because the deltas are small and roughly orthogonal, not because the model was ever taught to combine them. Push several strengths up together and you drift off the trained manifold — deltas can over- or under-shoot and adapters can interfere where their behaviors overlap. That risk grows precisely when several behaviors are requested at once, the common case in production — which is why we address it head-on.

Closing the gap: overlap co-activation fine-tuning

The fix follows directly from the diagnosis. If the failure is that the model never saw multiple adapters active together, then we should show it that regime during training — turn multi-adapter activation from an extrapolation into something observed and optimized.

The obstacle is the same one we started with: data where several behaviors co-occur is rare. But we do not need much of it, and — crucially — we do not need it to be labeled as multi-attribute. We can mine the overlap that already exists in our per-behavior subsets:

1. Find the intersection. Many clips legitimately belong to more than one behavior subset — an expressive clip that is also studio-quality, a gaze-controlled clip that is also expressive. We tag every clip with the full set of behaviors it satisfies, so a clip in the overlap carries several adapter labels rather than one.
2. Co-activate during training. When such a clip is sampled, we activate all of its adapters together in a single forward pass — the exact additive-sum path used at inference — and train them jointly on that clip.
3. Learn the composition, don’t extrapolate it. Now the joint regime W·x + Σi αi·Bi(Ai(x)) appears in the training objective. The adapters are optimized to work together on real footage, so inference-time composition reproduces a seen behavior instead of extrapolating a first-order guess.

Why it is cheap. The base stays frozen and only the adapters move, and the stage needs only a small amount of real overlap data — clips that genuinely exhibit several behaviors at once — far less than a fully co-labeled everything-at-once corpus. Single-behavior clips keep training their single adapter; overlap clips additionally teach the model how the adapters coexist. Single-adapter behaviors are preserved while the multi-adapter regime is brought in-distribution.

The effect is to align training with the way the model is actually used. Requests in production co-activate several behaviors; the overlap stage ensures the model has been optimized for exactly those co-activations rather than trusting a linear approximation to hold.

A note on strengths. The overlap fine-tune co-activates adapters at full strength (α = 1). At inference you can still dial an individual behavior down — the switchboard in Figure 1 shows gaze at 0.8 — but strengths near 1 stay in the regime the model was trained on, and turning one down mostly reduces that behavior's intensity rather than requesting a combination the model has never seen. Very low strengths are the one place a smaller version of the same train/inference gap can reappear.

Results: behaviors stack, each metric lights up, quality holds

We evaluate on a fixed 50-case benchmark (n = 50) and build the behavior set up one adapter at a time. Four automatic metrics, each the target of one behavior:

• Body movement (expressive) — DWPose motion-intensity score (upper-body keypoint velocity); higher = more animated.
• Gaze-control success (gaze) — the fraction of the 50 cases whose generated eye direction matches the command (straight / side, including cross-control), scored by a gaze estimator and reported as a count (e.g. 48/50).
• Camera movement (dynamic-camera) — mean P(camera-motion) from a VideoMAE motion classifier, 0–1; higher = more camera motion. The dynamic-camera request targets ≈0.4.
• Lip-sync — SyncNet score (audio–visual sync confidence [SyncNet]; higher = better); the quality guard that should hold no matter how many adapters are on.

Each adapter lights up its own metric — and only its own. Adding expressive lifts body movement (3.2 → 4.7); adding gaze raises gaze-control success from 80% to 96% (48/50); adding camera-movement raises camera motion to the requested level (0.15 → 0.42). Through all of it lip-sync holds (~8.5–8.6): the behaviors stack without breaking quality.

Why this matters

Treating behaviors as composable adapters over a frozen base turns a data problem into an engineering one. We no longer need a mythical dataset in which every attribute co-occurs at high quality; we need abundant single-attribute data (which exists) plus only a small amount of real overlap data where a few behaviors co-occur. Behaviors become modular — added, tuned, retired, and combined independently — while base fidelity is firewalled from the quality of any one behavior’s data.

The one subtlety that makes it actually work is recognizing that naive composition is linear extrapolation, and that a short fine-tuning stage on a small amount of real overlap data converts that extrapolation into a trained regime. It is a cheap fix — cheap because it needs so little overlap data, not because it needs none — for a problem that is easy to miss precisely because the naive version appears to work.

Related work

Composing LoRAs is an active area, but most methods combine independently trained adapters after the fact: ZipLoRA learns to merge a subject LoRA with a style LoRA; LoraHub dynamically assembles task LoRAs for unseen tasks; TIES-Merging and DARE resolve interference when merging many fine-tuned deltas. Our setting differs in what it asks of the data. Instead of merging adapters post-hoc, we co-activate them during a short fine-tune on real overlap mined from single-attribute subsets — so the composition is trained, not approximated, and we never need a co-labeled dataset in which the behaviors appear together.

References

1. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
2. Team Wan (2025). Wan: Open and Advanced Large-Scale Video Generative Models. arXiv:2503.20314.
3. Peebles, W., & Xie, S. (2023). Scalable Diffusion Models with Transformers. Proceedings of ICCV. arXiv:2212.09748.
4. Shah, V., et al. (2023). ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs. arXiv:2311.13600.
5. Huang, C., et al. (2023). LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. arXiv:2307.13269.
6. Yadav, P., et al. (2023). TIES-Merging: Resolving Interference When Merging Models. NeurIPS. arXiv:2306.01708.
7. Yu, L., et al. (2023). Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (DARE). arXiv:2311.03099.
8. Chung, J. S., & Zisserman, A. (2016). Out of Time: Automated Lip Sync in the Wild (SyncNet). ACCV Workshop.