Why Hugging Face’s New Rust‑Based Candle Framework Could Redefine AI Inference

Recently Hugging Face quietly open‑sourced a heavyweight ML framework called Candle. Unlike the usual Python‑centric approach, Candle is written in Rust and focuses on performance (including GPU support) and ease of use.

Candle’s core goal is to make serverless inference possible. Large frameworks like PyTorch are bulky, slowing instance creation on clusters. Candle enables deployment of lightweight binaries and removes the Python runtime, whose overhead and GIL can hurt performance.

Can Rust Really Deliver?

PyTorch is Python‑based, offering fast development and a simple API, but Python can introduce performance bottlenecks, especially due to the Global Interpreter Lock and runtime overhead. Deploying PyTorch models often requires extra steps compared to compiled languages.

Hugging Face aims to address these issues by rewriting an ML framework in Rust, a language already used in parts of its ecosystem (safetensors, tokenizer). However, Rust’s steep learning curve deters some developers.

Some developers point out that PyTorch already provides Python‑free deployment paths such as TorchScript, libtorch, ONNX export, and that C++ can be used for training and inference.

Others argue that while Python remains convenient for preprocessing and business logic, non‑Python languages like Rust can simplify production deployment and improve inference efficiency.

Comparison with PyTorch

Candle now supports cutting‑edge models like Llama 2 and can run them in containers or even browsers. Its structure includes:

Candle‑core: core operations, device and Tensor definitions.

Candle‑nn: tools for building real models.

Candle‑examples: usage examples.

Candle‑kernels: custom CUDA kernels.

Candle‑datasets: datasets and loaders.

Candle‑Transformers: utilities for Transformers.

Candle‑flash‑attn: FlashAttention v2 layer.

Key features of Candle compared with PyTorch:

Simple syntax, PyTorch‑like style.

CPU and CUDA backends (m1, f16, bf16).

Supports serverless, small and fast deployment.

WASM support for browser inference.

Model training capabilities.

Distributed computing via NCCL.

Out‑of‑the‑box models: Llama, Whisper, Falcon, StarCoder, etc.

Embedding user‑defined ops/kernels such as flash‑attention v2.

The article invites readers to share their thoughts on Hugging Face’s new framework.