MiniMax Open-Source MSA: High‑Performance Attention Kernels Optimized for NVIDIA SM100

1. Architecture Design

MSA combines two runtime compilation stacks—csrc JIT (built on Jinja templates) and CuTe‑DSL (a Python DSL for CUTLASS)—to enable coordinated sparse and dense attention acceleration on NVIDIA SM100.

1.1 Dense Attention and Index Stack (csrc JIT)

Provides a Dense FMHA operator that can run full‑precision attention or act as a “Proxy Pass” in sparse pipelines to quickly compute block‑level max_score. The accompanying sparse_topk_select operator extracts Top‑K block indices.

1.2 Block Sparse Attention Stack (CuTe‑DSL)

Implements end‑to‑end sparse acceleration. In the Prefill stage it supports extreme low‑precision quantization such as NVFP4/FP4. In the Decode stage it offers Paged FP8, BF16, and FP4 wrappers that fit large‑model long‑text paging memory management.

1.3 Minimal Migration (Bridge)

Provides an adaptation layer that lets existing dense fmha_sm100 code switch to the sparse prefill path with a single line change.

2. Environment Requirements

Hardware

GPU: NVIDIA SM100 (Compute Capability 10.0)

Software

CUDA Toolkit (nvcc in PATH, version ≥12.x)

Python ≥3.10

Toolchain: CUTLASS submodule (auto‑fetched)

Environment Check

nvcc --version          # expect ≥12.x
nvidia-smi --query-gpu=compute_cap --format=csv | grep "10.0"   # confirm SM100
python -c "import sys; print(sys.version_info[:2])"   # expect (3, 10)

3. Installation Guide

# Clone the repository with submodules
git clone --recursive https://github.com/MiniMax-AI/MSA.git msa
cd msa
# Standard install
pip install .
# Editable install for development
pip install -e .

The repository also includes usage examples and benchmark code.

4. Reference Resources

Github: https://github.com/MiniMax-AI/MSA
Algorithm documentation: https://github.com/MiniMax-AI/MSA/blob/main/docs/MiniMaxSparseAttention.pdf