How JD’s Open‑Source JoyAI‑Echo Tackles the Three Big Challenges of Long‑Form Video Generation

JD recently released JoyAI‑Echo, an open‑source framework for long audio‑video generation that directly tackles the industry’s three longstanding challenges: inconsistent character appearance across shots, fluctuating speaker timbre, and painfully slow generation speed.

The framework introduces four concrete technical innovations: (1) a cross‑modal memory bank that persistently stores and reuses character visual features and voice characteristics across multiple shots, ensuring high identity consistency; (2) a memory‑driven post‑training pipeline that combines Supervised Fine‑Tuning (SFT), cross‑modal Reinforcement Learning with Human Feedback (RLHF), and Distribution Matching Distillation (DMD), which alone yields roughly a 7.5× speedup; (3) a Director Agent that enables conversational editing—users describe desired changes in natural language, the system parses them into script, characters, scenes, and shots, and only regenerates the affected segments; (4) a lightweight real‑time super‑resolution module offering two up‑scaling modes (736×1280 → 1152×1920 and 736×1280 → 1472×2560) while maintaining low latency.

To evaluate performance, the team built a benchmark set of 100 stories and 3,000 shots, measuring cross‑shot consistency, video quality, text‑to‑video alignment, and speech‑content accuracy. JoyAI‑Echo leads on all core metrics, achieving a speech‑content accuracy of 0.8646.

A user‑preference survey showed strong favorability: 81.7% preferred its audio quality, 80.6% its adherence to prompts, 63.6% its visual aesthetics, and 59.4% its IP consistency. Even in portrait short‑video tasks, JoyAI‑Echo received higher user recognition for visual aesthetics and prompt compliance.

The framework opens new possibilities for virtual story creation, digital‑human content production, rapid brand‑marketing video iteration, pre‑visualization and storyboard generation, interactive educational material, and game cutscene creation. All code and model weights are publicly available on GitHub (https://github.com/jd-opensource/JoyAI-Echo) and the project homepage.