AI Character Video Engines Compared: Unreal, Unity, Browser 3D

Unreal and Unity: What Capabilities Do They Provide?

Game engines are programmable real-time scene systems: they load models and materials, compute skeletons and physics, control cameras and lighting, and generate frames. Editors organize these assets. For games, player input drives scene changes; for video, a pre-written timeline decides when a character speaks, turns, raises a hand, or cuts cameras. The camera output becomes the film.

Unreal Engine (UE) , developed by Epic Games, holds a major position in high-end 3D games, realistic scenes, and virtual production. For character video, key tools are MetaHuman for digital humans, Control Rig for character control, Sequencer for shot timelines, and Movie Render Pipeline for output. Core programming uses C++; Blueprint provides visual scripting; Python mainly serves editor automation, not a general runtime Python interface.

Unity , from Unity Technologies, is widely used for mobile, indie, and cross-platform interactive apps, and can produce high-quality 3D. For programmers, C# components, the Animator animation system, Timeline, and Cinemachine camera tools form a script-driven pipeline. Character models and facial driving usually require separate assets or plugins.

These are ecosystem tendencies, not quality ceilings. Unity can do photorealism; UE can do stylized work. Final results depend on assets, render settings, and artists. Market position also depends on measurement. GDC's 2026 industry survey of over 2,300 professionals showed 42% chose Unreal and 30% chose Unity as primary engine among respondents, though not all answered the engine question. Established indie studios still show high Unity usage. These shares indicate both are mainstream options, but they do not represent global game revenue share, install base, or character-video market share. For the author's needs, ready-made characters, motion assets, and automated output matter more than leaderboard rankings.

A Speaking Character Requires at Least Four Layers of Work

Imagine a tutorial clip: the character says "Look at the chart on the left" while turning and pointing. Four problems must be solved:

Character assets. The model needs a skeleton to bend arms; the face needs controllable morph targets (blendshapes) for visemes and expressions. Blendshapes are pre-baked shapes like "mouth open" or "smile" blended by weight.

Performance data. Speech needs matching mouth shapes (visemes), body needs motion, gaze needs targets. One viseme can map to multiple phonemes. Audio-to-viseme covers only part of the performance and does not automatically generate natural pointing or finger contact.

Scene and rendering. Apply motion to the character, set lights, cameras, backgrounds, and output frames at specified times.

Video compositing. Combine character footage with subtitles, charts, voice-over, and music on a single timeline, then encode to MP4.

The engine mainly handles the character rendering segment (shown in blue in the diagram). Upstream still needs character and performance data; downstream still needs video assembly.

Coding agents accelerate data conversion, scene setup, and batch calls. They can write arm-rotation code, but if the model lacks proper bones, code cannot invent joints; if the motion lacks weight shift, the code runs correctly but the performance looks stiff. This explains why a "chat-capable digital human demo" is far from a sustainable video system: it may only show a front-facing bust, real-time playback, one language, and miss full-body walking, Chinese visemes, camera cuts, and repeatable export.

HyperFrames Boundaries: Has a Render Engine, Can Also Carry Browser 3D

HyperFrames includes @hyperframes/engine, but this engine is a page-capture and video-generation engine based on Puppeteer and FFmpeg, not a character and scene editing system like Unreal or Unity. Simultaneously, HyperFrames supports Three.js-style browser 3D. Three.js is a JavaScript 3D library that loads models, plays skeletal animations, and renders scenes in a web page. Therefore, "no game engine backend" does not mean it can only do 2D text animation.

Two integration paths exist:

Drive the character inside the browser. Load the character with Three.js, sample motion and facial weights by time, let HyperFrames capture frames. This reuses the JavaScript toolchain, but character assets, motion control, and rendering determinism must be handled yourself.

Let an external engine render character clips first. UE, Unity, or Blender output video or image sequences, then hand them to HyperFrames for subtitle and chart composition. This architectural boundary is clearer; the cost is that any performance change requires regenerating source material. It is a designed integration, not a claim that HyperFrames provides ready-made adapters for those engines.

An easily overlooked issue: smooth real-time playback does not guarantee correct frame-by-frame export. HyperFrames seeks by time. If the character relies on the real-time clock, network responses, or ever-accumulating physics state, jumping to 10 seconds may differ from playing sequentially to 10 seconds. A better offline-video approach fixes audio, visemes, and motion data first, then derives character state from video time. Hair, cloth, and spring-bone simulations with historical state need baking, deterministic replay, or special handling. Simply setting the animation player's time may not solve everything.

What Parts Do Open-Source Projects Already Provide?

Maturity is judged by clear boundaries, documentation, examples, version maintenance, and traceable issues — not just stars. Below, "reusable components" and "complete production systems" are assessed separately; conclusions come from repo and doc inspection, not deployment acceptance.

Browser Characters: three-vrm and TalkingHead

pixiv/three-vrm is the open-source implementation of VRM in Three.js (MIT license). VRM is a humanoid character format based on glTF, adding humanoid skeleton and expression conventions. For developers driving stylized or cartoon characters with JavaScript, this is a priority base component to verify. It provides loading and control foundations for the character format, but does not automatically turn a script into a full performance. The character file's usage license is independent of the library's MIT license.

met4citizen/TalkingHead is closer to an application component that "makes a 3D character speak" (MIT license). Built on Three.js, it offers speech viseme and character motion interfaces, with examples of connecting external speech services. Two boundaries are critical: it requires specific model skeleton and facial morph targets — not every VRM file works out of the box; its README's built-in viseme languages do not include Chinese. Connecting a service that provides viseme or facial data can extend capability, but the specific service, language, and character mapping must be verified.

Assessment: three-vrm suits as a character format foundation; TalkingHead suits as a speaking-character reference implementation. To integrate into deterministic offline video, time control must be verified or reworked — a real-time demo cannot be treated as a turnkey output solution.

Unity Characters: UniVRM, Plus Independent Motion and Output Pipelines

vrm-c/UniVRM is the standard VRM implementation for Unity (MIT license). It is a format integration component, suitable for loading and using VRM characters in Unity. The advantage: character control, cameras, and business logic go into a familiar C# project. Gaps are clear: still need viseme source, motion library, skeleton retargeting, and export scheduling. Retargeting maps motion from one skeleton to another; a model may import but not guarantee correct motion proportions or foot contact.

Unity's official Recorder package needs careful checking. Recorder 5.1's documented limitation: when launching the editor with -batchmode, Recorder cannot start recording normally; Recorder is also an editor tool and cannot be dropped into a standalone Player as a general capture library. This does not mean Unity cannot auto-output, but that "command-line engine launch" and "unattended video capture" still have work between them. Evaluate editor workflows with a graphical environment, or implement custom render-texture readback and encoding, but validate in the prototype phase.

Realistic Characters: MetaHuman and Audio2Face-3D

UE's MetaHuman is a mature commercial tool ecosystem, not an open-source project. It places high-quality characters and facial authoring tools in one system; MetaHuman Animator supports generating facial animation from audio, video, or depth data, giving a more complete starting point for realistic speaking characters. Body motion still requires separate planning. Even if a tool supports some capture method, check the current version, OS, input conditions, and output skeleton; "audio drives face" does not imply "dialogue auto-generates full-body performance." Final shots can be organized in Sequencer and output via Movie Render Pipeline.

An open-source direction worth watching is NVIDIA/Audio2Face-3D . It is the official component entry, containing SDK, training framework, models, and plugins; its core task is generating facial animation data from audio, not an entire film. Licenses are not uniformly MIT: code, models, containers, and data have separate terms; the SDK uses MIT, models use other terms. The inspected SDK requires Windows or Linux and an NVIDIA GPU with matching CUDA. For developers on Apple Silicon Macs only, this is not a directly runnable native SDK path; remote inference and data transfer must be evaluated separately. It fits facial-driving technical validation, but model download, inference environment, character morph mapping, and film output all need further integration.

Agent-Controlled Editors: Unity MCP and Unreal MCP

CoplayDev/unity-mcp lets agents manage Unity scenes, assets, and scripts. The repo has documentation and continuous releases, suitable for the candidate tool list. chongdashu/unreal-mcp provides UE editor control and Blueprint operations, but the repo explicitly marks experimental status and does not recommend production use. Same name (MCP) does not imply equal coverage or stability. MCP is a protocol for agents to call tools; it reduces repetitive editor operations but does not add missing facial morphs or guarantee natural motion. When selecting, test whether the agent can read scene state, detect errors, and fix asset references — not just whether it supports "create an object."

Lighter Lip-Sync Tools and Fully Open-Source Engines

DanielSWolf/rhubarb-lip-sync generates timestamped mouth-shape sequences from recordings. Originally for 2D mouth animation, it can serve as input for custom 3D viseme mapping, but the mapping layer must be written. Non-English recordings can use language-agnostic phonetic recognizers; the project notes results are usually less accurate than English recognizers; Chinese effectiveness needs listen-and-watch verification.

If "engine itself must be open source" is a hard requirement, Godot (MIT license) should enter candidates. It has 2D/3D scene and animation capabilities. Whether it has plugins for the target art style, lip-sync, and motion requires separate investigation; open-source engine alone does not mean the digital-human pipeline is complete.

If real-time interaction is not needed, Blender also deserves verification. It is open-source 3D creation software; Python automation suits organizing offline scenes and render tasks. For film, real-time frame rate is not the only goal; model cleanup, animation tweaking, and stable output sometimes matter more. It equally needs character and performance resources.

The research conclusion: Mature base components and commercial production tools exist; there is insufficient evidence to designate any open-source repo as a generally mature solution covering "script, natural full-body performance, Chinese lip-sync, camera, unattended MP4." This does not assert nothing exists globally, but that the candidates' capability boundaries do not yet cover this requirement set.

Same Character Video, How to Choose a Starting Point?

First distinguish the character's role in the video.

Explainer character — mainly stands or sits, speaks, occasionally nods and points at charts: start with a stylized character, use Three.js + three-vrm for minimal verification, then decide whether to borrow TalkingHead's speech and motion organization. Both sit in the browser stack, easier for coding agents to modify and inspect. Cost: you must build a reliable viseme and time-sampling layer yourself.

Realistic digital human — face close-ups and realistic lighting directly determine video value: prioritize UE + MetaHuman. Accept larger editor, asset management, and GPU costs for a more complete realistic production toolset. First confirm machine compatibility with needed MetaHuman features, then plan automation.

Existing C# or Unity background — planning to extend the character into a long-running interactive product: Unity + UniVRM deserves focused consideration. For someone who just wants the first explainer video quickly, first prove the extra engineering work is necessary.

Offline short film needing polished motion — put Blender in the same verification round. If the scene involves two-person dialogue, walking, handing objects, or hugging, the real difficulty often becomes motion, contact, and camera design; merely swapping engines hardly eliminates that production work.

Costs must be separated. UE allows source viewing but is not open source in the usual sense. Current terms classify rendered video as royalty-free non-engine products, not subject to game revenue sharing; qualifying non-game commercial use may still require seat subscriptions. MetaHuman's current license permits use in other engines or creative software, but export, material, and skeleton adaptation remain technical work. Unity is also commercial-license software. It has cancelled the Runtime Fee; Personal eligibility and Industry conditions for non-game, non-entertainment projects must be checked per actual entity and use case. Engine license, character models, motion packs, and speech services should be costed separately — "engine is free" cannot cover the whole production chain.

Do a 30-Second Verification First, Then Decide Whether to Scale Investment

Use a deliberately constrained prototype: one character, one fixed camera, one Chinese voice-over, only idle, nod, and one pointing action, output 1080p 30 fps 30-second MP4. These duration and parameters are suggested test conditions, not measured performance results.

Pick a model with clear license, inspectable skeleton and mouth shapes. First confirm the character can close mouth, open mouth, turn head, and raise hand — then write automation. Don't let the agent generate a human model from scratch.

Fix a voice file, generate or prepare timestamped viseme data. Chinese test includes bilabial stops, pauses, numbers, and mixed Chinese-English reading; actually observe sound-to-mouth relationship. Volume-driven mouth opening alone cannot prove viseme accuracy.

Bind idle and gestures to explicit motion clips, handle blend transitions. First observe shoulders, wrists, and feet for pops; then consider more motions. Random blinks use fixed seeds or pre-generated timetables.

Establish a unified timeline. Let the agent generate orchestration data like the example below, then interpreted by browser or engine adapters. This is a custom interface sketch, not a ready-made API of HyperFrames, UE, or Unity; fields like fps are prototype conventions.

{
  "fps": 30,
  "duration": 30,
  "audio": "speech.wav",
  "actor": "presenter",
  "gesture": {"clip": "point_left", "start": 8.2},
  "gaze": {"target": "chart_left", "start": 8.0, "end": 10.5}
}

For 30 fps video, frame n time is n/30 seconds. Visemes, motion, cameras, and subtitles should share the same time base. When stateful simulations are involved, additional baking or replay schemes must be defined.

Verify the true output flow: from clean start, no manual export clicks, get complete MP4; then change one line, regenerate the corresponding segment. Record completion time, peak memory, output frame count, and manual fix steps, then decide whether to continue.

Four results are specially checked: character state consistency across repeated exports; natural mouth closure during pauses; pointing action actually reaching the chart; after dialogue change, viseme, motion, and subtitles all update together. Image comparison helps spot differences, but final pixels across different GPUs need not be identical.

If the prototype already re-renders stably, the next step adds a second character and camera cuts. If only viseme or motion fails, fix that one link first — avoid swapping engines, models, and speech services simultaneously, ending up not knowing which change caused what. For the current need, start verification with "stylized character + browser 3D + fixed performance data + HyperFrames composition"; when realistic facial close-up becomes a core requirement, then invest in UE + MetaHuman. Selection ultimately answers: within affordable production cost, which route can sustainably output character performances I'm willing to publish.