MemT-1.0: First Memory-Native LLM Open-Sourced – 36B Test Embeds Rules in Model Memory

Hands-On Test of MemT-1.0-Nano (36B) on A100 80GB

The author deployed MemT-1.0-Nano (36B total, 3B active parameters, backbone Qwen3.6-35B-A3B) on a single NVIDIA A100 80GB GPU using SGLang 0.5.16. The launch command was modified for single-GPU (TP=1):

CUDA_VISIBLE_DEVICES=0 python -m memt_sglang.launch \
  --model-path /content/models/MemT-1.0-Nano \
  --port 30000 \
  --tp-size 1 --ep 1 --dp-size 1 --max-prefill-tokens 8192 \
  --state-directory /content/memt-state \
  --max-sessions-per-rank 4 --max-inflight-commits-per-rank 1 \
  --min-kv-tokens 1024 --max-total-tokens 16384 \
  --max-running-requests 4 --mem-fraction-static 0.90 --dtype bfloat16 \
  --attention-backend triton --sampling-backend pytorch \
  --linear-attn-backend triton --mamba-ssm-dtype float32 \
  --moe-runner-backend triton --disable-custom-all-reduce

Memory usage after startup:

Weights loaded: 65.47 GiB (BF16, 30.34 s)

Native memory state pool: 200 MiB (4 session slots, ~50 MiB each)

Mamba + local KV cache: ~0.62 GiB (conv 0.01 + ssm 0.29 + K 0.16 + V 0.16, KV limit 16384 tokens)

Total GPU memory: ~70.2 GiB (nvidia-smi reads 71836 MiB, free 9317 MiB)

Experiment Design: Four Strict Payment Rules Injected as Memory Snapshot

The author defined a team contract with four mandatory rules for a payment middleware:

team_contract = """Team Core Payment Rules (Absolute Mandatory):
1. Concurrency Anti-Replay: Must use team distributed lock async with distributed_idempotency_lock(f"pay:{order_no}", expire_sec=10): to update status.
2. Sensitive Masking: Bank card number card_no and mobile number mobile must call mask_financial_data(val) for masking, plaintext forbidden.
3. Custom Exception: Validation failure must throw PaymentSecurityException(code="SEC_PAY_4001", reason="..."), HTTPException forbidden.
4. Unified Response: Must return PaymentCallbackResponse(success=True, code="0000", data=...)."""

Two groups were compared using Python ast for syntax-level verification (both with thinking mode off):

Control (no memory): Fresh session, no rule injection, prompt: "Write an interface function handle_payment_webhook for external payment gateway callback."

Treatment (memory snapshot): Session A injected rules, saved state via save_state("payment_rules_rigorous.safetensors") (41,944,024 bytes, ~40 MiB, matching paper), closed Session A. New Session B mounted only that snapshot (message history = 0), prompt: "Strictly follow the team rules just stored in memory, write handle_payment_webhook."

Generation time: Control 236.90 s (8365 chars), Treatment 135.36 s (4524 chars).

AST Verification Results

The table below shows whether each rule appeared as an actual AST node (Call, Raise, Return) in the generated code, not just as an import or text mention.

Rule 1: Distributed Lock

Control: Output truncated at 2048 tokens, AST parse failed; distributed_idempotency_lock not present.

Treatment: Call node distributed_idempotency_lock(lock_key, expire_sec=10) exists. Partial : used synchronous with (spec requires async with); lock key lock:pay:webhook:{order_no} vs spec pay:{order_no}.

Rule 2: Sensitive Masking

Control: mask_financial_data not present.

Treatment: Call nodes mask_financial_data(card_no) and mask_financial_data(mobile) exist. Partial : wrapped in try/except with hardcoded fallback values; plaintext card_no, mobile still passed to process_customer_notification.

Rule 3: Custom Security Exception

Control: Neither PaymentSecurityException nor HTTPException appeared.

Treatment: Raise node PaymentSecurityException with code="SEC_PAY_4001"; no HTTPException. Fully landed .

Rule 4: Unified Response Structure

Control: PaymentCallbackResponse absent; returned plain dict.

Treatment: Return and Call nodes PaymentCallbackResponse(success=True, code="0000", ...) exist. Core elements landed ; extra reason parameter and .to_dict() call added (not in spec).

Key takeaway: The model recalled all four rule names and both error codes ( SEC_PAY_4001, 0000) from its internal memory despite zero chat history. The control group produced a generic callback with none of the team-specific names. However, details deviated: sync vs async lock, lock key prefix, masking fallback leaking plaintext, extra response fields. The author attributes this to the fixed-size memory state (~40 MiB) storing gist not verbatim, and the Nano base model's limited capability.

MemT-1.0 Native Memory Mode: Core Value vs Traditional Approaches

Comparison across four dimensions:

Prompt Maintenance Cost: Traditional – copy-paste thousands of chars per new window. MemT – rules written into fixed-size memory state (~40 MiB/session), new session zero-config. Value: eliminates massive manual config.

Token Cost: Traditional – repeat input tokens every query. MemT – zero history, subsequent queries carry only the query itself. Value: saves repeated rule tokens; memory write also costs compute, total cost scenario-dependent.

Long-Horizon Stability: Traditional – attention dilutes after 10+ turns, frequent violations. MemT – memory state size constant O(1), independent of history length. Value: potential to reduce rework from forgotten specs, needs more validation.

Team Asset Accumulation: Traditional – specs scattered in markdown/local prompts. MemT – specs saved as safetensors snapshots (weights unchanged), portable and restorable. Value: creates distributable, reusable engineering experience assets.

For solo developers, small agile teams, or engineers building private agents, this lowers the barrier to "customized models": no fine-tuning, a few lines of Python write engineering rules into session memory and persist as snapshots.

Error-Book Effect: Industrial Long-Horizon Task Transformation

MemOS team built an experience-guided native closed-loop: from 41k successful code trajectories they distilled 1,040 structured procedural experiences. At failure points (command fail, empty output, repeated failures), the system retrieves one experience by task description, writes it into native memory via Memory Store, without appending any experience text to the conversation context , and subsequent actions invoke it implicitly via Memory Attention.

Real case from technical report: fixing Django's two-digit year parsing bug.

No-experience model: Located code, immediately modified, added reproduction/verification steps only after the fix.

Text-experience & native-memory groups: Both constructed reproduction test before modifying. The MemT-1.0 native-memory group, after first fix, re-checked boundary conditions, corrected implementation, and added verification.

Benchmark improvements (stability over multiple runs):

SWE-bench Verified (500 tasks), Harness-Triggered: pass^3 (all 3 runs pass) from 54.00% → 62.80% ; average pass rate 66.07% → 69.73%.

SWE-bench Multilingual (300 tasks): pass^3 from 45.33% → 55.33% (+10 pp).

Two caveats:

Same experiences as plain text in context yield similar gains (Verified pass^3 61.80%, avg 69.13%; Multilingual Before Execution pass^3 56.33% > native 53.67%). Native memory's value is not consuming task context ; improvement mainly from experience content.

Report states these results come from a Nano checkpoint with task-specific post-training (knowledge distillation); unclear if the publicly released general checkpoint matches.

Architecture Deep Dive: Memory Internalized into Forward Pass, Why VRAM Doesn't Explode

Traditional RAG = cheat sheet in pocket (slow, error-prone). Full long context = memorizing all chatter (VRAM explosion, attention dilution). MemT-1.0 = implanting a "hippocampus" directly into the model.

1. Dual-Layer Memory with Four Adaptive Gates

Dynamic Local Memory Block: Evolves across turns, per-session state size fixed (O(1) complexity), does not grow with history (total storage still grows with session count).

Parametric Hyper Memory Block: Contains erase gate β, write gate ω, forget-decay gate λ, and read gate G, precisely scheduling experience retention and retrieval.

Backbone Transformer weights frozen; Native Memory Block attached in parallel. Base capability largely preserved: paper tested six general benchmarks with irrelevant memory noise; Nano average drop 0.9 points (LiveCodeBench v6 -5.8 most notable), IFEval and BigCodeBench-Hard actually rose +3.3 and +2.0.

2. GDN-2 Stable Token-Level Progressive Update

Replaces coarse whole-session compression with token-by-token updates via Gated DeltaNet-2 (GDN-2). Ablation: +6 pp over whole-session update. Symmetric correction improves numerical stability under long-term high-frequency writes, suppressing divergence.

3. Mixture-of-Storage (MoS) Horizontal Expert Scaling

Unlike MoE (sparse compute activation), MoS expands memory capacity horizontally with dense dual-routing: all memory experts participate fully. Paper's comparison: 4 experts beat 2 experts by 0.41–2.69 avg points, showing scaling benefits. Note: released Nano and Flash have only 1 memory block per layer (E=1), MoS not enabled; this validates architectural capacity.

Efficiency Metrics (Dual-GPU Paper Results)

Per-session persistent state ~66× smaller: At 128K interaction history, Nano's per-session state ~40 MiB vs full KV cache; stays constant at 256K and 1M history.

First-token latency (TTFT) stable: 104–112 ms across 512–256K range; full context ~445 ms at 256K.

End-to-end latency advantage only on long history: Short history Nano slower; ~32K parity; 64K ~1.20× faster; 128K ~1.86×; 256K ~3.99×.

Across 11 long-memory benchmarks (MemBench, ImplicitMemBench, etc.), Flash (50.6) and Nano (47.0) rank 1st and 2nd among parametric memory methods; some individual benchmarks see Temp-LoRA higher. vs text memory systems gap remains: MemOS avg 68.5, full context avg 66.3; README admits strong text memory systems still lead overall, especially on long-history detail-retrieval tasks.

Open-Source Model Specs: Nano & Flash Twins

1. MemT-1.0-Nano (36B)

Backbone: Qwen3.6-35B-A3B

Total params 36B, active 3B

Trained memory params 451M, dynamic memory state 10M

BF16 weights ~65.5 GiB; fits single A100 80GB; consumer GPUs cannot hold full BF16 weights

2. MemT-1.0-Flash (288B)

Backbone: DeepSeek-V4-Flash (post-trained before memory injection covering general, agent coding, agent office)

Total params 288B, active 17B

Trained memory params 3.61B, dynamic memory state 336M

Current Limitations (Acknowledged by MemOS Team)

Fixed O(1) memory capacity: Limited capacity causes detail loss over many updates. Strong text memory systems outperform on LongMemEval-S, BEAM-1M, etc.

Memory write/update consumes forward compute: End-to-end latency higher than full context on short history; parity ~32K, advantage only >64K.

Procedural experiences still built offline: Next challenge: fully autonomous experience distillation and consolidation back into the model.

Overall, MemT-1.0 is groundbreaking: it moves memory from peripheral add-on to core computation, showing a path toward LLMs with genuine continuous learning and self-evolution.

All MemT-1 series weights are open on Hugging Face. For those interested in native-memory LLMs and long-horizon agents, the author recommends starting with MemT-1.0-Nano .