Deploy MiniMax H3 Locally: Character Replacement, Lip-Sync & Audio Preservation Guide

Introduction

The author demonstrates replacing both performers in the "APT." music video with Jensen Huang and Sam Altman (in a maid outfit) using MiniMax H3. The process separates character replacement, original vocal preservation, and lip-sync correction into independent steps so each can be fixed individually.

What Gets Replaced

The demo shows a 77.33-second continuous segment (not the full MV). Simple face-swap tools suffice for face-only changes, but swapping hair, clothing, and body shape requires video character replacement with masks to define editable regions. The author splits the pipeline: character replacement → original vocal → lip-sync → editing.

Local Deployment of MiniMax H3

1. Install ComfyUI and Verify GPU

Environment: Ubuntu, RTX 4090 24GB, ComfyUI. Install NVIDIA driver, Git, Python 3.12. Verify GPU detection: nvidia-smi Clone ComfyUI and create a virtual environment:

git clone https://github.com/Comfy-Org/ComfyUI.git
cd ComfyUI
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install torch torchvision torchaudio \
  --extra-index-url https://download.pytorch.org/whl/cu130
python -m pip install -r requirements.txt

Follow ComfyUI's CUDA 13.0 installation path; ensure driver compatibility. Backup existing environments before updating. Official manual: https://docs.comfy.org/installation/manual_install

Check CUDA and launch:

python -c "import torch; print(torch.__version__, torch.version.cuda); print(torch.cuda.is_available())"
python main.py

Last line should output True. Open http://127.0.0.1:8188 to confirm ComfyUI UI.

2. Download Models to Correct Directories

Character replacement uses H3 Ref2VA base model + Character Swap LoRA . LoRA alone cannot run; base model, text encoder, and two VAEs (video & audio) are required. Files go under ComfyUI/models/: diffusion_models/ →

minimax_h3_ref2va_pruned_int8_convrot.safetensors

text_encoders/

→

qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors

vae/

→

minimax_h3_video_vae_int8_convrot.safetensors

vae/

→

minimax_h3_audio_vae_fp32.safetensors

loras/

→ h3_character_swap_pro4500_1000.safetensors First four from https://huggingface.co/Comfy-Org/MiniMax-H3; LoRA from https://huggingface.co/akatz-ai/MiniMax-H3-Character-Swap-LoRA. Total weights ~40GB; reserve 100GB SSD for environment, intermediate videos, cache. Download commands in the provided package avoid cloning entire repos.

3. Import Workflow and Run a Short Clip

Drag 01-author-character-swap-v1.gui.json into ComfyUI, refresh models or restart, select files in loaders. Workflow uses built-in nodes, validated on ComfyUI 0.37.0; missing core nodes (e.g., MiniMaxH3ReferenceToVideo) indicate version/log issues.

Feed reference image to Reference Image input, 24fps source clip to Source Video, adjust character description. GUI defaults: V1 LoRA strength 1.0, 20 steps, res_multistep / simple sampler, Turbo disabled (optional Turbo loader bypassed).

Reference images need clear face/clothing, empty hands, simple background — avoid unwanted props (microphones) being copied. Source video must be clean.

Default 0.4MP is for debugging. Production: set to 0.98MP (1344×768) per H3 guide; input/output dimensions must be multiples of 32. Exporting to 1080p does not automatically add character detail.

Check terminal for model load errors, CUDA OOM, and ComfyUI/output/ for new video. If OOM: shorten clip, reduce reference materials, lower generation resolution — do not downscale source to save VRAM. Basics: https://docs.comfy.org/tutorials/video/minimax/minimax-h3

Advanced pipeline (per-person crop, mask, local compositing) uses different config than GUI defaults:

Source Material: 1920×1080, 24fps

Character Adapter: Character Swap V1, strength 1.0

Sampling: 20 steps, beta schedule, Turbo off

Character Region: Mask-cropped, ~1MP minimum

Audio: Re-wrapped from original video

Lip-sync: Original-frame-driven LivePortrait correction

As of 2026-10-17, author recommends V1.5; this demo retains V1 for reproducibility. V1.5 not yet validated in this MV pipeline — cannot just swap filename and reuse all params. Advanced API graph needs SAM3, MaskVidExperiments, VEDA, and preprocessing nodes; retains legacy FP16 video VAE. First run GUI, then add dependencies per MODELS_AND_NODES.md when ready for per-person masking.

Short Shots First: Scene Breakdown, Masking, Per-Person Replacement

1. Find Continuous Shots

Use clean, watermark-free source (watermarks persist if "keep original background" is on). Pick a 3-5 second clip with frontal faces, minimal occlusion. MV cuts fast (e.g., 30-35s spans three shots) — split by cut points, process separately, reassemble.

H3 output frame count limited by workflow frame cap. Used 90 and 124 frames. At 24fps, 5s = 120 frames; 124 frames ≈ 5.17s. Pad tail frames by generating extra then trimming to original frame count — do not stretch video.

2. Verify Masks Select Correct Person

Advanced flow uses SAM3 to track persons, exports binary masks: white = modify, black = keep. Foreground singer masked; background drummer separate.

Check first, middle, last frames: hair covered? clothing stays in mask after turn? microphone accidentally excluded when hand raises? Multi-person crossing — verify tracking IDs match correct person.

3. One Person Per Pass, Then Composite

Initial attempt with two reference images caused identity bleed. Revised: same source clip, process Sam's mask alone, then Jensen's alone, composite each via its mask.

Reference image defines identity and outfit; motion follows source. Singer keeps original microphone, drummer keeps sticks — do not prompt "singing" for all.

Adjustable prompt (replace character/prop descriptions):

PROMPT Replace only the masked performer in <Video 1> with the adult character in <Picture 1>. Use the reference identity, hair and outfit. Follow the source performer's body position, gestures, gaze and mouth state on each frame. Keep closed lips closed. Retain the existing microphone, cable and instruments, including their contact with the hands. Keep unmasked people and the scene unchanged. Do not introduce new props or a new performance.

After generation, verify identity, clothing, person count, props, motion. Wrong face → fix character replacement; missing mic → fix props. Do not proceed to lip-sync or upscaling on flawed base.

Original Vocal and Lip-Sync Must Be Preserved Separately

Easiest: keep audio track outside model, re-wrap after video processing. Input audio / output with sound ≠ lossless original vocal ≠ lip-sync.

When processed video aligns with original timeline and original audio suits MP4, use FFmpeg to copy streams:

ffmpeg -i replaced-silent.mp4 -i original.mp4 \
  -map 0:v:0 -map 1:a:0 -c:v copy -c:a copy \
  -movflags +faststart final.mp4

Example for full segment. For a 5-second excerpt, extract corresponding 5-second audio — do not attach full song from start; AAC slicing involves encoder frame boundaries, precise sample verification separate.

Lip-sync is harder. H3 references original performance but may open closed mouths or drift on profile, fast turns, mic occlusion. Solution: drive LivePortrait with original corresponding frames, composite only mouth region. Some shots used gentle hybrid mode, blend weight 0.4, then re-check lip open/close and mic occlusion. LivePortrait repo: https://github.com/KlingAIResearch/LivePortrait

Side-by-side comparison at same timestamp shows mouth movement, hand-mic connection.

Still motion transfer, not pixel-perfect copy. Large profile, blur, occlusion, crossing cause errors. Package includes Linux lip-sync reference script requiring separate LivePortrait install and weights.

How to Check Upscaling and Ghosting

1080p file ≠ every generated face detail at 1080p. Background kept at 1080p; characters generated at mask-cropped resolution then pasted back. Character region resolution, source sharpness, face size in frame all affect result.

Blurry face → check source, crop region, generation resolution, compression; super-resolution later. This clean version skipped full-segment upscaling; character sharpness determined by earlier generation.

"Ghosting" diagnosis: wall projections of head/hands existed in original. Background retained, old projection outlines unchanged → looks like extra person layer.

Fix locally, no need to redo whole segment. Another common double-image: composited video blended again with old character mask, original hair/hat reappears → fix compositing chain.

Final per-shot review: identity, clothing, count, props, edges; continuous playback for lip-sync and cuts. Original vocal decoded to PCM, sample-by-sample comparison for corresponding time range. This 77.33s version passed prefix sample-level consistency check; watermark-over-speaker area shows minor repair texture.

Resources and Closing Advice

Model weights, original MV, and song not included — prepare per project licenses. Package images for flow understanding, not pre-paired inputs. Demo is AI character replacement; public sharing must be clearly labeled; model usage per original project license.

Get one short shot's face, clothing, props, and lip-sync right first, then move to next. Batch queuing is easy; finding what went wrong per shot is the most time-consuming part of this MV workflow.