Douyu’s Live2D Virtual Avatar Plugin: Unity Architecture & Key Tech

1. Project Background

After the first popular generic virtual streamer on YouTube at the end of 2016, many virtual streamers appeared in 2017. In recent years platforms such as Bilibili, Huya, YY, and AcFun have launched virtual‑streamer zones or tools, with dozens to hundreds of streams simultaneously.

Numerous market tools (Huya PC, YY streaming, AcFun face‑capture helper, Steam FaceRig) are based on Live2D technology, which is widely adopted. Professional services can create a customized Live2D avatar for 2,000–5,000 CNY.

To leverage this market and Douyu’s user base, a Douyu virtual‑avatar tool was developed, compatible with existing Live2D ecosystem, allowing streamers to create personalized avatars and build an internal content ecosystem.

2. Core Functions

The Douyu virtual‑avatar plugin, built on the Unity engine, integrates the following features:

Display Live2D models, preview motions and expressions.

Change model parts such as hairstyles and clothing.

Adjust part colors (hair, eye color, etc.).

Drive facial motions from webcam key points.

Drive facial motions from iPhone X+ face‑capture data.

Detect facial expressions and trigger corresponding model expressions.

Load and showcase model‑provided motions via hotkeys.

3. Technical Architecture

3.1 Business Capability Overview

The virtual avatar consists of three main components:

Virtual‑avatar plugin: a Qt‑based live‑companion plugin.

Vtuber Manager: launches and manages the avatar display process.

Shared Memory Reader: reads frames from the display process.

PluginElement: outputs avatar frames to the streaming preview area.

Avatar display process: a Unity‑based rendering process.

Within the display process, modules include UI panel, process manager, model manager, model renderer, scene manager, scene renderer, output manager, data manager (face‑capture handling), shared memory writer, and IPC client.

The avatar‑setting process contains modules such as Motion Trigger, Camera Preview, Shared Memory Reader, and IPC Client.

3.2 Avatar Process Architecture

The display process is divided into four layers:

UI layer – renders UI and handles interaction.

Framework layer – provides interaction context between business logic and UI.

Business layer – implements specific features based on requirements.

Engine layer – core rendering engine that loads, renders, and drives characters and scenes.

4. Key Technical Points

4.1 Unity Rendering

In Unity, each GameObject can attach scripts derived from MonoBehaviour. Unity automatically creates instances of these classes and invokes lifecycle methods such as Awake, Start, and Update. Execution order across multiple MonoBehaviours follows the lifecycle diagram; custom order can be set via Player Settings → Execute Order.

4.2 Unity UI with FairyGUI

Unity offers several UI solutions (IMGUI, UGUI, UIElements) and third‑party libraries (NGUI, FairyGUI). UGUI was found cumbersome: many controls require manual assembly, prefabs and scripts intertwine, and debugging is not intuitive. FairyGUI provides an independent editor, supports many engines, and greatly speeds up UI development. UI functionality is implemented by attaching custom scripts to the UIPanel GameObject.

4.3 Inter‑Process Communication (IPC)

A lightweight TCP protocol over localhost is used. The virtual‑avatar plugin acts as a TCP server and launches the display process as a client, passing the server port via command‑line arguments. Conversely, the display process can act as a server for the setting process. Named pipes were tested but failed to provide full‑duplex communication in Unity, so they were abandoned.

4.4 Live2D Integration

Live2D works by applying 2‑D texture transformations (translation, rotation, deformation). The SDK consists of a Core module that parses .moc3 files and a Framework module that handles physics, motion preview, expression playback, and rendering.

4.5 Live2D Model Dressing

Although the official SDK does not expose a dressing API, models are composed of multiple parts with separate textures. By exporting optional parts as separate textures, the application can swap textures at runtime to achieve dressing effects. A *.personality.json file stores available dress‑up options and texture paths.

4.6 Live2D Model Recoloring

Each model part becomes an ArtMesh (a Unity GameObject) with Mesh, Material, and Texture components. While Mesh and Texture are fixed, the Material’s color can be altered via Unity shaders, enabling runtime recoloring of the avatar.

4.7 Face Data Processing

4.7.1 Webcam Face Data

Webcam data provides 106 facial landmark coordinates. By analyzing relationships among these points, expressions such as blink, mouth movement, eyebrow motion, and head rotation are derived. A reference aspect ratio is computed as:

distance_ref_ = (ouler_distance(1,33) + ouler_distance(17,52)) / 2

All subsequent landmark parameters are normalized against this reference.

4.7.2 ARKit Face Data

ARKit on iPhone X+ devices outputs BlendShape data. The system receives this data via UDP, filters relevant BlendShape coefficients, converts them to Live2D parameters, and drives the avatar to synchronize facial motions.

5. Product Interface

6. Future Plans

Support loading and driving of 3D models (e.g., MMD, FBX).

Add full‑body pose recognition.

Enable custom facial sculpting.

7. Project Summary

The Douyu virtual‑avatar project leveraged Unity and Live2D to implement model dressing and recoloring capabilities not found in competing products. Simply using the official SDK would not achieve the same results; understanding underlying mechanisms allowed the team to unlock new industry potentials and discover unexpected use cases.

References

[1]

blendshapelocation: https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation