EM²Mem: Event-Centric Multimodal Memory for LLMs Boosts Accuracy & Efficiency
EM²Mem introduces an event-centric multimodal memory system that organizes experiences by events, aligns multimodal cues during memory construction, and enables efficient retrieval, outperforming baselines on long-video QA benchmarks while reducing inference time and token usage.
Problem: Retrieve-Then-Align vs Align-Then-Retrieve
Long-video understanding requires connecting scattered, cross-modal evidence (names in dialogue, objects in frames, task completion across segments). Existing methods store descriptions, transcripts, keyframes, or graph facts separately and retrieve relevant pieces at query time, then ask the model to align them — a retrieve-then-align pipeline. This risks filtering out weakly relevant evidence that only becomes useful when combined with other cues.
EM²Mem flips the order: align-then-retrieve . During memory construction, it binds multimodal observations to shared event anchors, so later queries can directly reason over pre-organized events.
EM²Mem Architecture: Three-Layer Event Memory
The system divides long video into 30-second base segments, each becoming an event anchor with a unique ID and time span. Evidence (keyframes, visual descriptions, transcripts, structured fields for actions, objects, topics, scenes, entities) is attached to the same anchor.
Layer 1 – What happened: Stores keyframes, visual descriptions, dialogue transcripts, and structured fields (actions, objects, topics, scenes, entities).
Layer 2 – Contextual narrative: Builds multi-scale summaries (3 min, 10 min, 1 hour) linked back to base events, preserving local detail within longer narratives.
Layer 3 – Long-term regularities: Plot graphs connect persons, objects, locations, topics, and temporal relations across events; semantic graphs organize habits, preferences, routines, and stable relationships, all traceable to supporting events.
At query time, the system locates relevant events, expands along temporal or entity links, selects necessary evidence, and forms a compact context. For visual verification, keyframes can be included.
Case Study: Who Plans to Grow Flowers?
A multi-person discussion about flowers, vases, and crafts is used to contrast EM²Mem with WorldMM. The question: "Who plans to grow flowers?" WorldMM retrieves flower-related segments but misattributes the plan to Tasha because her segment contains "planning" semantically close to "plans." EM²Mem, having preserved Katrina's association with the planting discussion and later "we can also plant" statements, correctly answers Katrina .
Experimental Results
Evaluated on three long-video QA benchmarks (EgoLifeQA, Ego-R1 Bench, Video-MME Long). EM²Mem achieves higher accuracy than WorldMM and other baselines. On EgoLifeQA (first 250 questions), structured event fields + build-time alignment reach 71.2% accuracy vs 68.0% for structured fields with retrieval-time fusion.
Event-level evidence recall (strict 30-second segment): EM²Mem Top-5 recall 30.8% vs WorldMM's 23.8% (5 iterative retrieval rounds), a 7.0 percentage-point gain. Top-1 recall reaches 23.0%.
Efficiency Gains
Single-query latency: 98.21 s vs WorldMM's 459.00 s ( 4.67× speedup ).
Reasoning tokens: 15.27M vs 42.03M ( 63.66% reduction ).
End-to-end (memory build + 500 queries): 105,160 s vs 318,210 s ( 3.03× speedup ).
Break-even: ~24 queries to amortize build time; ~536 queries for total token parity (equal-weight input/output counting).
Gains come from pre-organized event memory: queries read and filter ready-made evidence instead of assembling multimodal clues on the fly. Memory built once is reused across many queries.
LightMem-Ego: Applied System
Built on EM²Mem, LightMem-Ego demonstrates two applications:
Personal life assistant: Records daily experiences via phone/smart glasses; helps retrieve misplaced items, past conversations, and organize life notes.
Scientific lab assistant: Continuously records experiment video, operations, and dialogue; supports step retrospection, key operation/phenomenon lookup, and experiment log organization.
Code and technical report: https://github.com/zjunlp/LightMem-Ego (technical report: arXiv:2607.11487). Paper: EMNLP 2026 Findings, arXiv:2609.00551.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
