NVIDIA: Long Context Windows Are Unnecessary — Short Context with State Compression Saves 2-230x Cost
NVIDIA research shows long LLM context windows cost 3.6-7x more and reduce throughput similarly, while short-context agents with semantic state compression (SSSM) match or exceed accuracy on long-horizon and long-input tasks at 2-230x lower cost, challenging the need for massive context windows.
