A product architecture guide to combining pre-rendered video states, TTS, lip-sync and real-time generation for expressive AI companions without runaway latency or cost.
Why full generation is not always the best answer
Real-time generative video is compelling, but a companion product is judged every second. Latency, identity drift and cost can become more important than maximum visual novelty. A hybrid architecture can deliver a more stable experience by using a small library of high-frequency states for common moments and reserving expensive generation for situations where it creates visible value.
The core state set
A practical baseline can include attentive idle, speaking, happy or amused, concerned or disappointed, surprised and a short transition state. The exact number depends on the character, but each state should cover a broad emotional range through facial intensity and timing. This keeps the asset library manageable while preventing every response from looking identical.
Separate speech from emotion
Lip movement and emotional expression are related but should not be one hard-coded clip. TTS determines timing, while the visual system chooses a compatible state and applies lip-sync or a speaking overlay. This makes the same emotional state reusable across many utterances. It also allows the system to switch from listening to speaking without regenerating the entire scene.
Transitions create realism
Users notice abrupt cuts more than they notice small imperfections inside a clip. Short transition clips, pose-compatible states and consistent camera framing reduce visual discontinuity. The system can wait for natural sentence or breath boundaries before changing state. A stable face, hairstyle, outfit and background often matter more to perceived quality than constant motion.
When real-time generation earns its cost
Generate when the user explicitly asks for a new scene, outfit, action or selfie; when a premium interaction requires a unique visual; or when the conversation reaches a moment that cannot be represented by the state library. Routine listening and talking can stay on cheaper paths. This turns generative video into an event rather than a tax on every second of interaction.
Latency budget
A video companion has multiple delays: speech recognition, reasoning, safety checks, TTS, lip-sync and rendering. These should be orchestrated in parallel where possible. The interface can immediately acknowledge input with an attentive animation while the answer is prepared. Tuikor’s article on real-time video companion trade-offs discusses why latency and cost must be modeled together.
Identity consistency
Every asset should be generated or captured against a strict identity reference. Face geometry, hair, body proportions, lighting direction and camera distance should stay within a controlled range. Real-time generation needs the same constraints. A visually impressive clip that looks like a different person breaks the social illusion faster than a simpler but consistent animation.
Emotion routing
The language model should not directly choose arbitrary animation names. A lightweight classifier can map response context into a small emotion taxonomy with confidence and intensity. Rules can prevent rapid oscillation. For example, a mildly disappointing topic should not trigger a dramatic sadness state. Designers can then tune the mapping independently from the dialogue model.
A sensible rollout
Start with a small, high-quality state library and instrument how often each state is used. Add states only when real conversations reveal repeated gaps. Keep expensive generation user-triggered or premium until unit economics support broader use. This architecture gives teams a path from reliable video companionship today toward more generative interaction as models improve.
Practical takeaway
For product teams, the useful next step is to test this framework against real conversations and real user controls. A companion experience becomes durable when identity, memory, multimodal interaction and monetization reinforce one another rather than operating as separate features.