AI MediaKit: Multimodal Understanding & ReFM for Seamless Subtitle Removal

Challenge: Subtitle Removal in Real-World Videos

As short dramas, comics, e-commerce videos, and ads go global, subtitle removal is no longer just erasing a line of dialogue at the bottom. Real videos contain complex text: vertical names beside characters, stylized chapter titles with calligraphy strokes, outlines, shadows, and dynamic effects; semi-transparent watermarks floating over complex backgrounds. Meanwhile, signage, posters, product packaging, clothing text, and on-screen displays are native scene information that must not be deleted. This turns subtitle removal into two harder problems: (1) which text is post-production overlay to erase, and (2) which text belongs to the real scene and must stay, while ensuring the background remains realistic, stable, and flicker-free after removal.

Multimodal Semantic Understanding: Classifying Text Roles

Traditional OCR-based approaches only detect where text is, not whether it is an overlay layer. In real frames, dialogue subtitles, character introductions, program logos, handheld posters, and product packaging can appear simultaneously. Erasing all text damages brand info and scene details; processing only the bottom misses vertical stylized text, transition titles, and corner watermarks. AI MediaKit introduces multimodal understanding to first judge the role of text in the frame, then decide removal.

Stable Text Trajectories Across Frames

The same subtitle across consecutive frames suffers from character motion, camera changes, and compression noise, causing detection boxes to jitter, break, or briefly disappear. AI MediaKit does not judge each frame in isolation. It organizes detection results into stable text trajectories using spatial overlap and temporal continuity, selecting multiple representative frames for joint analysis. This leverages context to judge text attributes and fills short-term detection gaps, reducing flicker where one frame erases text and the next shows it again.

Classification Categories

To erase: dialogue subtitles, character names and identity introductions, chapter titles, decorative prompt text, promotional watermarks.

To preserve: signage, posters, books, clothing prints, product packaging, on-site screens, brand cards, and fixed logos.

The decision is based not merely on text content or position, but on the relationship between text and the people, objects, and scene carriers. The same large text floating beside a character (identity intro) vs. printed on a poster (title) receives completely different treatment. For semantically ambiguous regions, a conservative strategy prioritizes preserving the original frame, lowering the risk of accidentally erasing high-value content like brand exposures, product info, stage screens, and plot props.

ReFM: Residual Flow Matching for Natural Inpainting

Accurate judgment solves "where to erase"; natural repair determines whether the result is truly seamless. Traditional generative inpainting starts from random noise to rebuild the target region. For subtitle removal, this path is too long: the model must not only remove text but also regenerate large amounts of already-correct content, leading to texture drift, character distortion, and extra hallucinations.

Residual Editing from Original Latents

AI MediaKit transforms the video generation model into a residual editing model for removal tasks — ReFM. Instead of starting from Gaussian noise, it starts from the latent variables of the subtitled video and learns the residual between that and the clean frame. The objective shifts from "repaint the whole video" to "make only the necessary modification." This shorter, more deterministic editing path better inherits the original video's characters, lighting, textures, and motion, achieving high-quality repair in 4 sampling steps .

Dual Guidance for Detail and Consistency

To recover details occluded by subtitles, the model incorporates two complementary guides in the video generation backbone:

Cross-frame context aggregation for occluded regions: searches adjacent frames for reference structures and textures, providing information for locally occluded areas.

Global video representation guidance: continuously injects the original video's overall content and style into the editing process, reducing discontinuity between repaired regions and surrounding frames.

Local information handles "restore correct details"; global information ensures "the whole clip still looks like the same visual world."

Latent Precise Repasting for Boundary Control

On this basis, Latent precise repasting further guards generation boundaries: in latent space, the edited region is precisely constrained, preserving regions by repasting original latents as much as possible, letting the model generate only the necessary removal and completion range. Combined with stable temporal masks, this reduces color shifts, sharpness loss, and texture drift in non-target areas. Effectively, it draws a boundary for generative repair: generate fully where needed, keep original maximally where not.

Handling Real-World Text Varieties

Real production challenges are not regular subtitles but freely positioned, complex-styled, highly fused text. AI MediaKit enhances training data and inference strategies for high-frequency scenarios:

Character introduction fancy text: large font, free position, with outlines, shadows, vertical layout. System removes both character intros and dialogue subtitles, using neighbor frames to restore covered hair, clothing, and background structures.

Artistic and dynamic titles: often exceed conventional subtitle areas, covering large frame portions. Specialized data for calligraphy, variety-show stylized text, chapter titles enables handling irregular strokes, colored outlines, moving text, and large-area occlusion.

Semi-transparent subtitles: blend with background textures, hard to detect and prone to stroke residuals. Watermark-specific training and region-enhanced supervision improve handling of low-contrast text, mirror watermarks, and repeated promotional text.

Scene-native text: not removal targets. For posters, book pages, brand cards, stage screens, product packaging, the model judges them as scene-native via carrier relationships and preserves original information. Removal quality depends not only on how much text is erased, but also on how much original content is kept.

Production-Ready Pipeline: Stability and Efficiency

Model performance must serve content globalization via stable, scalable production pipelines. AI MediaKit builds a complete pipeline covering text detection, multimodal judgment, video repair, and result verification:

Few-step generation: ReFM backbone uses 4-step sampling, balancing repair quality and processing efficiency.

Long video continuous processing: overlapping windows and temporal context stitching reduce segment-boundary flicker.

Original-spec delivery: restores source video resolution after inference and preserves original audio.

Distributed batch processing: supports multi-GPU parallelism, checkpoint restart, staged caching, and per-video verification.

Full-chain traceability: retains detection, semantic judgment, repair, and status logs for quality auditing and rapid issue localization.

AI MediaKit also adapts to GPU and NPU compute environments, improving compatibility and portability across infrastructures, leaving more flexibility for compute selection and large-scale deployment. These capabilities move subtitle removal from single demos into batch production workflows.

Multi-Language Support for Globalization

For short-drama globalization, subtitle removal is a key step before translation, localized dubbing, and new subtitle creation. For e-commerce, advertising, and copyrighted content, it enables removal of old copy, character tags, and promotional watermarks, providing clean plates for secondary creation. The new version supports subtitle removal for 50+ languages , smoothing overseas content creation and localization production.

Conclusion

This upgrade is not just about erasing one subtitle line more cleanly. It lets the system understand the role of text in video and perform controlled local editing with generative capabilities: what should be erased is erased completely; what should stay is preserved accurately; the post-removal frame is restored realistically and stably. From "seeing text" to "understanding the scene," AI MediaKit subtitle removal brings seamless erasure to more complex real production scenarios, giving video localization, asset reuse, and secondary creation a more reliable starting point.