Code-Rendered MV: 20 Scenes, Word-Synced Lyrics, 4K Deterministic Frames

What It Does

The pdoom-video repository is a web application (TypeScript + three.js, run with bun + Vite) that maps any moment t to a single frame. Every frame is a deterministic function of t — preview and offline renders produce identical pixels. There is no editing software, no timeline, no keyframe tracks. All visuals are generated at runtime: lyrics written by a spark, loss curves drawn in a coordinate system, paperclip geometry generated in the scene. The "edit" is a table in timeline.ts that assigns each scene a start and end second, aligned to word timestamps and the musical beat grid.

Each file under app/src/scenes/ is a "plate" (scene module). timeline.ts contains about twenty lines like E('open', 'open', 0, b.loss) — one entry per plate, with window times computed from aligned lyric data rather than hand-coded timecodes. The whole piece can be previewed live in the browser: bunx vite serves localhost:5173, spacebar plays/pauses, arrow keys step frames, [ / ] jump scenes, ?t=23 seeks to 23 seconds. What you see is the final render output.

Repository Components

Rendering Engine ( app/src/engine/ )

engine.ts

: timeline playback, per-frame rendering, adaptive sampling. post.ts: post-processing — bloom, glare, chromatic aberration, film grain, vignette. type.ts: typesetting system with four font families. lines.ts: GPU line-segment batching, supports 10k–200k segments. glsl/common.ts: shader library — palette constants, noise, SDF, copperplate hatching ( hatch / engrave). stroke.ts: single-stroke font (pen plotter / engraving style). hud.ts: global HUD and P(doom) readout. lyrics.ts / audio.ts: lyric and audio data interfaces.

Scenes ( app/src/scenes/ )

19 plates: open, loss, prompt, hook, room, shoggoth, spacetime, ascent, bureau, leftturn, paperclips, fuse, stack, dense, loom, ilya, outro, plus shared motifs in _motifs.ts. Several plates split into sub-modules (e.g., dense-press.ts, dense-gpu.ts, dense-askew.ts).

Audio Analysis Pipeline ( analysis/ , Python + uv)

Demucs stem separation ( htdemucs_ft) plus a mel-band-roformer karaoke model for vocals.

CTC forced alignment + Whisper cross-validation, producing word-level (partially syllable-level) timestamps.

Beat, downbeat, onset, section, loudness envelope analysis.

Outputs data/lyrics.json and data/audio.json (beat 132.007 BPM).

Offline Renderer ( app/scripts/render.ts )

Five modes: stills (single frames), sheet (contact sheet), plates (regenerate scene stills), perf (performance test), video (full video).

Documentation

docs/TREATMENT.md

(24k chars): complete storyboard bible and style spec. docs/ENGINE.md (16k chars): engine and scene API guide.

How It Was Built

The entire piece was created in collaboration with Claude inside Claude Code — concept, style bible, shot-by-shot storyboard, lyric alignment, audio analysis, renderer, every scene, every render — all via conversation. But "AI-generated" here does not mean one prompt and wait. The workflow:

Step 1: Define the Style Spec

TREATMENT.md

locks the palette: background #0A0A0B, paper/text #EEE9DF, danger orange #FF4D12 (signal color — spark, P(doom) value, sung words), hot core #FF8A3D. Only one rare accent is allowed: #D8FF3C acid green for the "shrooms" 2 seconds. The spec explicitly forbids "purple/cyan neon cyberpunk, glowing brains, Matrix code rain, lens flare soup, generic particle nebulae" — in short, nothing that looks like AI-generated imagery.

Step 2: Produce the Data

Run Demucs to split vocals and accompaniment, use two models (CTC + Whisper) to cross-validate word-level alignment. Why so rigorous? Because every lyric in the piece is word-synced karaoke — each word highlights the instant it is sung, early by at most 0.4 s but never before the sound.

Step 3: Write Scenes Plate by Plate

One file per plate. Hard rule: output must be a pure function of time t; Math.random() forbidden (only seeded PRNG mulberry32 allowed). Verification: render still PNGs, then "use the Read tool to look at these PNGs".

Step 4: Render

Headless Chrome drives the app, streams frames to ffmpeg.

Beat-Syncing Cuts

Scene boundaries are not hand-coded timecodes but "find a word in the lyrics, cut on the nearest beat before it". Code pattern:

const cut = (q: string, nth = 0, tol = 0.02) => { const s = ly.get(q, nth).words[0]!.start; return au.timeOfBeat(Math.floor(au.beatAt(s + tol))); };

Change a lyric timestamp and the whole edit moves with it.

Where the Real Effort Went

Motion Blur Is a Sampling Problem

On export, each frame is not rendered once but averaged over many sub-frames within the shutter time ( --shutter 0.2, i.e., 1/5 frame duration). Fast motion leaves continuous trails instead of stepped ghosts.

The key is that sub-frame count is adaptive per 32×32 pixel tile : static areas 12, normal camera motion 36, whip pans / hard pushes / fast zooms 108 or even 324. The engine inserts new sub-frames symmetrically around existing ones (keeping uniform distribution centered on the frame moment), compares the new and old tile averages, and stops when tile change falls below a threshold (default 3 levels in 255). Converged trails or static frames stop evolving; already-converged tiles are culled early via depth buffer on the GPU, so they never enter shading. A fast object crossing a static background costs mainly the object itself.

4K Is True Rendering, Not Upscaling

Add --scale 2 and every layer, line, and shader renders at 3840×2160 physical resolution, while scenes still layout at 1920×1080 logical pixels — so 4K frames look identical to 1080p but sharper. Cost is GPU-bound: one frame from 40 ms (static) to several seconds (108–324 sub-frames of ray-marched room). Full song on M5 Pro ≈ 57 minutes, peak memory ≈ 23 GB. Film grain rendered per-pixel at 4K; encoding expensive — default CRF 16 yields ~670 Mbit/s, ~13 GB total.

Two Renderers Feed One ffmpeg

Default --workers 2: two headless Chrome instances alternate rendering 8-frame chunks, streamed in order to a single ffmpeg. Result matches single-renderer output. One renderer saturates GPU on heavy scenes; the second fills gaps on light scenes. Frames waiting for the encoder buffer in RAM (default 1 GB) then spill to disk (default 20 GB), so renderers never block on x264.

Typesetting Has Rules

Four font families with distinct roles: Archivo (width axis 62–125, weight 300–900) is the voice of lyrics, strokes stretch with singing; IBM Plex Mono is the machine — tokens, labels, HUD, forms; Cormorant Garamond (especially italic) is the sacred/prophetic register, used sparingly; single-stroke font for "written by pen" text. Every proportional glyph line gets kerning adjustments, even gaps between separately drawn segments are tuned by eye. Spec even dictates punctuation: typographic quotes ' " " … for prose, but mono UI voice keeps typewriter quotes because it shows "user-typed input".

"Don't Look AI-Generated" Has Concrete Execution

shoggoth

is the project's own design , not the famous meme; "Death Note eyes" borrow only the concept of labels (name, lifespan) floating above things, zero anime characters; model names (ChatGPT, Sydney, Gato) appear only as lyric text, no logos, no imitation of real product UIs.

Limitations

Designed for team collaboration, not a one-person day. ENGINE.md states "do not edit files outside your scene (engine changes ask lead)" and "do not edit src/timeline.ts " — an engineering convention for parallel multi-author work.

Stateful scenes lose adaptive sampling. If a scene needs simulated state (particles, feedback buffers), it must flag stateful = true; export then uses fixed --samples, losing per-tile adaptation.

Performance target ≤ 25 ms/frame. Each Canvas2D layer upload costs 2–4 ms; a scene cannot exceed 2–3 layers.

Regenerating timing data needs extra resources. Committed data/*.json are all the renderer needs; but regenerating them requires stem files and intermediate artifacts (not in repo) plus ~4 GB model weights. Full reproduction of the alignment pipeline means assembling those yourself.

The song isn't the project's own. Code is MIT, fonts keep their licenses, audio, lyrics, data/lyrics.json are not MIT — lyric copyright belongs to the author; this version uses a Suno-generated "Claude-Pop" rendition. Reuse with caution.

Worth Studying?

Who should read it: People doing procedural visuals, creative coding, generative art. This repo is a rare high-completion reference — not just "how to draw a pretty effect with three.js" but "how to organize a 20-scene, music-synced, 4K-export, multi-author video engineering project." ENGINE.md and TREATMENT.md are themselves worth reading: the former teaches how to write a deterministic scene module, the latter teaches how a music video's visual consistency is maintained by a single spec.

One technique to steal: Turn "editing" into data. Scene boundaries bound to lyric words and beat grid; change one timestamp and the whole edit follows. This pattern ports directly to any audio-visual sync generation project.

Not for: People wanting a drop-in template for their own MV. Song rights, 4 GB models, multi-scene engineering conventions, performance budgets — these make "cloning" cost far higher than "referencing." It's better read as a methodology than hacked as a template.

A telling detail: The repo's latest commit message reads "Revision 6: burning-line cut into the left turn, no white washes, cleaner type." Sixth revision, still tweaking "no white washes" and "cleaner type." A code-rendered MV ends up obsessing over the same things as a hand-crafted film — aesthetics, not technology.

GitHub: github.com/mexicat/pdoom-video

Project: pdoom-video

Stars: 2.7k+

Language: TypeScript (three.js) / Python (analysis pipeline)

License: MIT (code); audio/lyrics copyright original author

Updated: 2026-10-13