Machine Heart

Oct 25, 2026 · Artificial Intelligence

Dust: Zeroth-Order Optimization Trains Transformers Without Backpropagation

Q Labs introduces Dust, a zeroth-order optimization method that trains Transformers by adding Gaussian noise to per-token activations instead of using backpropagation, achieving competitive results at 243M parameters and 20M tokens while challenging the necessity of differentiability in neural network training.