IQuest-Q1 Technical Deep Dive: 320B MoE Agent Model with 512K Context

Model Architecture and Efficiency Innovations

IQuest-Q1 adopts a sparse Mixture-of-Experts (MoE) architecture with 320B total parameters and 15B active parameters per token. Each MoE layer contains 256 routed experts, with 8 experts activated per token. The model comprises 88 Transformer layers with a hidden dimension of 3072.

Attention Mechanism

The model combines Sliding Window Attention (SWA) and Full Attention (FA) in a repeating pattern: three SWA layers followed by one FA layer. The SWA window size is 4096 tokens, reducing attention computation and KV Cache overhead for long sequences. Grouped Query Attention (GQA) is used with 48 query heads and 8 key/value heads (6 query heads share one KV head), further shrinking KV Cache. Partial RoPE applies rotary position embeddings to only 32 of 128 dimensions per head.

Long Context and Recursive Multi-Token Prediction

IQuest-Q1 supports a 512K token context window, targeting coding agents that need to ingest large repositories, terminal logs, and multi-turn histories. For faster generation, the model introduces Recursive Multi-Token Prediction (Recursive MTP). During pretraining, two MTP layers provide additional future-token supervision. After main training, one MTP layer is converted into a Recursive Drafter for inference. The drafter's sliding window is limited to 512 tokens (vs. 4096 for the main model) to control prediction cost in long-context scenarios.

Training Stability and Scaling Metrics

The team employs Stable SplitNorm, the Muon optimizer, and μP (maximal update parametrization) to stabilize large-scale training. They propose a Critical Model Size (CMS) metric that analyzes the relationship between safe and optimal learning rates to assess scaling boundaries. Experiments show IQuest-Q1 retains training stability margin at its current scale.

Data and Environment for Agent Training

Pretraining and mid-training consume approximately 29T tokens. The code data ratio increases dynamically from 40% to 70% during pretraining. Mid-training adds reasoning QA and agent execution trajectories while extending context to 512K tokens.

To teach agents how to execute tasks in real environments, the researchers built over 1 million containerized, executable task instances sourced from code repositories and agent trajectories. For post-training, they constructed executable environments covering software engineering, terminal workflows, frontend applications, cybersecurity, and professional tasks, each equipped with verifiers that judge success by actual execution results.

Reward Hacking Mitigation

A three-layer defense prevents reward hacking: (1) Environment preparation removes answer-leaking history, reference solutions, and isolates tasks. (2) Execution-time checks block unauthorized access to hidden tests or external answers. (3) Post-collection trajectory audits combine rule-based and model-assisted review to discard cheating trajectories.

Post-Training: Multi-Environment RL and Multi-Teacher Distillation

Post-training targets software engineering, long-horizon tasks, and general agent abilities via Supervised Fine-Tuning (SFT), multi-environment Reinforcement Learning (RL), and Multi-Teacher Online Policy Distillation (MOPD). RL spans multiple agent frameworks (Claude Code, Codex), requiring the model to interact with diverse environments and adjust strategies based on feedback. Independent verifiers assess task completion, while isolation and auditing mechanisms filter invalid rewards.

MOPD introduces four specialist teacher models covering: terminal interaction & user experience, multi-framework adaptation, long-code & security tasks, and general agent capabilities. Model merging integrates these specialized skills into a single unified model.

Benchmark Results Across 30 Evaluations

IQuest-Q1 is evaluated against top-tier models including GLM-5.3 Flash, DeepSeek-V4 Flash-0731, Tencent Hy3, MiniMax M3, Claude Opus-5, and GPT-5.6 Sol. Key results:

AutomationBench (cross-app workflows): 52.7 vs. Claude Opus-5 50.3, GPT-5.6 Sol 45.8.

τ³-Banking (multi-step tasks): 51.8 vs. Claude Opus-5 44.7, GPT-5.6 Sol 44.3.

SkillsBench (tool use): 74.1 vs. GLM-5.3 Flash 56.2, DeepSeek-V4 Flash-0731 72.0.

MCP-Atlas (skill use): 78.4 vs. DeepSeek-V4 Flash-0731 72.8, MiniMax M3 74.2.

Claw-Eval (continuous execution): 77.1 vs. GPT-5.6 Sol 70.2, GLM-5.3 Flash 74.2, Claude Opus-5 79.6.

IQuest-CLIBench (CLI agent simulation): 53.7, exceeding GLM-5.3 Flash and DeepSeek-V4 Flash-0731.

Code benchmarks: NL2Repo-Bench 63.0, CyberGym 84.5, both above GPT-5.6 Sol.

General reasoning: GPQA Diamond 93.3 (near Claude Opus-5), IMOAnswerBench-Verified 93.0 (highest reported).

The model still trails leaders on some high-difficulty software engineering tasks but shows strong potential across code and agent benchmarks.

Self-Improvement Experiment

To demonstrate recursive development, the team tasked an IQuest-Q1-based agent with analyzing 35 tool-call errors from long-horizon tasks. The agent inspected 6,054 tool-call records, identifying root causes: training data over-concentrated on a single tool-call format and mismatches between tool definitions and call parameters. The agent then proposed data cleaning and format diversification, which were human-reviewed and applied to subsequent training. Verified improvements are reused in later iterations, reducing redundant R&D effort.

Significance: A From-Scratch Pretraining Path

Unlike many open models that fine-tune existing bases (Qwen, DeepSeek), IQuest-Q1 is independently pretrained from scratch. For IQuest Research, this establishes a complete, reproducible research pipeline spanning pretraining, mid-training, agent-oriented post-training, and self-improving evaluation—laying groundwork for future model iterations.

Technical report: https://arxiv.org/pdf/2610.30949; Blog: https://iquestlab.github.io; Hugging Face: https://huggingface.co/IQuestLab/IQuest-Q1; GitHub: https://github.com/IQuestLab/IQuest-Q1.