How to Fit Large Language Models into Cars and Robots: A Hardware‑Aware Scaling Law

Deploying large language models (LLMs) on embodied systems such as autonomous vehicles and mobile robots faces a fundamental dilemma: low latency and high precision cannot be achieved simultaneously with conventional cloud‑centric designs.

The research described here introduces a hardware‑aware scaling law that jointly optimizes model perplexity and inference speed for edge devices, reducing perplexity by 19.42% while cutting architecture search time from months to a few days under identical latency constraints.

Why Edge‑Specific Model Design Is Needed

Edge devices are limited by strict memory, bandwidth, power, and latency budgets, which reshapes the rules for AI model construction. Models that thrive in data‑center environments become inefficient when transferred to the edge, leading to either excessive latency for large, accurate models or poor accuracy for tiny, fast models.

To address this, a hardware‑software co‑design approach is required so that every neuron aligns with the underlying chip characteristics.

Roofline Model Analysis

The study applies the Roofline Model to classify model execution states into compute‑bound and bandwidth‑bound regions. Increasing model depth linearly raises both compute and memory reads, while widening the model causes exponential growth in parameter volume and memory traffic.

In batch‑1 inference scenarios typical of edge devices, parameters are rarely reused across tokens, and on‑chip caches cannot hold massive weights, causing the processor to idle while waiting for off‑chip memory.

These observations motivate a shift from blind parameter scaling to a hardware‑aware architecture search.

Pareto‑Optimal Architecture Search (PLAS)

By combining training‑loss prediction with hardware latency models, the authors formulate a Pareto optimization problem that seeks a set of architectures forming a performance frontier where no model can improve accuracy without increasing latency.

Using Latin hypercube sampling to initialize the search space, the algorithm iteratively refines candidates near sparse regions until the frontier cannot be pushed further left‑downward.

Figures (a)‑(c) illustrate the Pareto front, the impact of FP16 vs INT8 precision, and the distribution of optimal architectures across latency budgets.

Scaling Law Derivation

The framework binds training loss to architectural parameters (layers, width, KV‑cache dimension, expert activation rate) through a set of polynomial equations derived from extensive empirical data (thousands of models trained on 100 B tokens each).

Training on a diverse corpus (general text, math reasoning, code) with consistent optimizers yields a fit with R² = 0.975 on the training set and 0.952 on a held‑out validation set.

These equations enable rapid prediction of both perplexity and latency for any candidate architecture.

Evaluation on Edge Hardware

The method is evaluated on NVIDIA Jetson Orin, testing over 5 × 10⁴ model configurations in under 20 minutes using purely analytical estimates.

Two inference phases are distinguished:

Prefill : processing the long input prompt, compute‑bound.

Decode : generating tokens one‑by‑one, bandwidth‑bound.

Different applications tolerate different latencies (e.g., 20 ms for real‑time robotics, 500 ms for smart home assistants), guiding the selection of the appropriate point on the Pareto frontier.

Key Architectural Insights

Optimal edge LLMs tend to be wide and shallow rather than deep, because increasing width yields higher accuracy per latency unit than adding layers.

Sparse Mixture‑of‑Experts (MoE) architectures dominate the frontier; the best designs often activate a single expert during decode to avoid bandwidth bottlenecks while using few experts during prefill to limit compute pressure.

Traditional Transformers with a 4× feed‑forward expansion are sub‑optimal; edge‑optimal models use expansion ratios below 1, reallocating parameters to width or expert count.

Physical Constraints as Design Equations

The authors derive closed‑form expressions for the optimal number of layers, width, and expert activation rate based on three hardware constants: peak compute, memory bandwidth, and on‑chip memory capacity.

When memory is the limiting factor, wider models require higher sparsity to stay within the memory budget.

In dual‑constraint regimes (both compute and bandwidth limited), the optimal solution depends on which phase dominates, leading to distinct algebraic or quadratic solutions.

Practical Usage

Given a target device, engineers can measure its compute‑to‑memory ratio, plug the values into the derived formulas, and instantly obtain the ideal model dimensions without exhaustive search.

After determining the architecture, a short fine‑tuning run (a few billion tokens) yields a state‑of‑the‑art edge LLM.

Conclusion

The presented hardware‑aware scaling law and Pareto‑optimal search framework provide a systematic way to design, evaluate, and deploy large language models on resource‑constrained edge platforms, turning hardware specifications into concrete model design parameters.