Skip to content
EM²Mem: Event-Centric Multimodal Memory for LLMs Boosts Accuracy & Efficiency

EM²Mem: Event-Centric Multimodal Memory for LLMs Boosts Accuracy & Efficiency

EM²Mem introduces an event-centric multimodal memory system that organizes experiences by events, aligns multimodal cues during memory construction, and enables efficient retrieval, outperforming baselines on long-video QA benchmarks while reducing inference time and token usage.

Machine Heart
Machine Heart
Machine Heart

Problem: Retrieve-Then-Align vs Align-Then-Retrieve

Long-video understanding requires connecting scattered, cross-modal evidence (names in dialogue, objects in frames, task completion across segments). Existing methods store descriptions, transcripts, keyframes, or graph facts separately and retrieve relevant pieces at query time, then ask the model to align them — a retrieve-then-align pipeline. This risks filtering out weakly relevant evidence that only becomes useful when combined with other cues.

EM²Mem flips the order: align-then-retrieve . During memory construction, it binds multimodal observations to shared event anchors, so later queries can directly reason over pre-organized events.

EM²Mem Architecture: Three-Layer Event Memory

The system divides long video into 30-second base segments, each becoming an event anchor with a unique ID and time span. Evidence (keyframes, visual descriptions, transcripts, structured fields for actions, objects, topics, scenes, entities) is attached to the same anchor.

Layer 1 – What happened: Stores keyframes, visual descriptions, dialogue transcripts, and structured fields (actions, objects, topics, scenes, entities).

Layer 2 – Contextual narrative: Builds multi-scale summaries (3 min, 10 min, 1 hour) linked back to base events, preserving local detail within longer narratives.

Layer 3 – Long-term regularities: Plot graphs connect persons, objects, locations, topics, and temporal relations across events; semantic graphs organize habits, preferences, routines, and stable relationships, all traceable to supporting events.

At query time, the system locates relevant events, expands along temporal or entity links, selects necessary evidence, and forms a compact context. For visual verification, keyframes can be included.

Case Study: Who Plans to Grow Flowers?

A multi-person discussion about flowers, vases, and crafts is used to contrast EM²Mem with WorldMM. The question: "Who plans to grow flowers?" WorldMM retrieves flower-related segments but misattributes the plan to Tasha because her segment contains "planning" semantically close to "plans." EM²Mem, having preserved Katrina's association with the planting discussion and later "we can also plant" statements, correctly answers Katrina .

Experimental Results

Evaluated on three long-video QA benchmarks (EgoLifeQA, Ego-R1 Bench, Video-MME Long). EM²Mem achieves higher accuracy than WorldMM and other baselines. On EgoLifeQA (first 250 questions), structured event fields + build-time alignment reach 71.2% accuracy vs 68.0% for structured fields with retrieval-time fusion.

Event-level evidence recall (strict 30-second segment): EM²Mem Top-5 recall 30.8% vs WorldMM's 23.8% (5 iterative retrieval rounds), a 7.0 percentage-point gain. Top-1 recall reaches 23.0%.

Efficiency Gains

Single-query latency: 98.21 s vs WorldMM's 459.00 s ( 4.67× speedup ).

Reasoning tokens: 15.27M vs 42.03M ( 63.66% reduction ).

End-to-end (memory build + 500 queries): 105,160 s vs 318,210 s ( 3.03× speedup ).

Break-even: ~24 queries to amortize build time; ~536 queries for total token parity (equal-weight input/output counting).

Gains come from pre-organized event memory: queries read and filter ready-made evidence instead of assembling multimodal clues on the fly. Memory built once is reused across many queries.

LightMem-Ego: Applied System

Built on EM²Mem, LightMem-Ego demonstrates two applications:

Personal life assistant: Records daily experiences via phone/smart glasses; helps retrieve misplaced items, past conversations, and organize life notes.

Scientific lab assistant: Continuously records experiment video, operations, and dialogue; supports step retrospection, key operation/phenomenon lookup, and experiment log organization.

Code and technical report: https://github.com/zjunlp/LightMem-Ego (technical report: arXiv:2607.11487). Paper: EMNLP 2026 Findings, arXiv:2609.00551.

EM²Mem overview: event-centric multimodal long-term memory
EM²Mem overview: event-centric multimodal long-term memory
EM²Mem pipeline: build event units, connect via plot and semantic graphs, retrieve-expand-select for queries
EM²Mem pipeline: build event units, connect via plot and semantic graphs, retrieve-expand-select for queries
Qualitative case: WorldMM misattributes plan to Tasha; EM²Mem correctly identifies Katrina via preserved associations
Qualitative case: WorldMM misattributes plan to Tasha; EM²Mem correctly identifies Katrina via preserved associations
Main benchmark results: EM²Mem outperforms WorldMM on EgoLifeQA, Ego-R1 Bench, Video-MME Long
Main benchmark results: EM²Mem outperforms WorldMM on EgoLifeQA, Ego-R1 Bench, Video-MME Long
Ablation: structured event fields + build-time alignment yields 71.2% accuracy on EgoLifeQA subset
Ablation: structured event fields + build-time alignment yields 71.2% accuracy on EgoLifeQA subset
Cumulative cost: EM²Mem crosses WorldMM at ~24 queries (time) and ~536 queries (tokens), then stays lower
Cumulative cost: EM²Mem crosses WorldMM at ~24 queries (time) and ~536 queries (tokens), then stays lower
Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

LLM memorymultimodal memoryEMNLP 2026align-then-retrieveEM²Memevent-centric memoryLightMem-Egolong-video QA
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.