Deconstructing Scaling Laws: The Interplay of Optimization, Architecture, and Data

Introduction

Training a large neural network is affected by many factors: optimizer, model architecture, and training data. The author calls the empirical tuning of these factors “炼丹” (alchemy) and seeks a quantitative description using Scaling Laws.

Preparation: Heterogeneous Power Inequality

The analysis relies on a frequently used inequality: for positive numbers a,b and exponents p,q with p+q=0, the sum a^p + b^q attains its minimum at a specific ratio. The equality condition is derived via weighted AM‑GM (see citation [2]).

Three‑Way Decomposition of Loss

Given data, architecture, and optimizer, the training objective is to minimize the loss on the ideal data distribution.

The total gap between the current state and the ideal loss is split into three non‑negative terms:

Data error – the loss on the training set versus the ideal distribution (generalization error).

Optimization error – the gap between the best possible loss on the training set (with an ideal optimizer) and the loss achieved by the actual optimizer.

Architecture error – the gap between the best possible loss on the training set (with an ideal architecture and optimizer) and the loss of the current architecture.

This decomposition allows each factor to be studied separately while keeping the overall relationship explicit.

Optimization Chapter

The author focuses on three hyper‑parameters: learning rate, batch size, and training steps. Assuming the “more training is better” hypothesis, the loss contribution is modeled as a sum of a monotonic decreasing term (training progress) and a noise term that grows with larger batch size or learning rate.

Using the heterogeneous power inequality, the optimal learning rate is shown to be inversely proportional to a power of the batch size, and the optimal batch size is proportional to a power of the total number of training samples. The derived forms match the decoupled expressions in works such as 2503.12645 [3] and 2603.15958 [4], and simplify to four‑parameter formulas comparable to 2607.01487 [5] and 2605.09154 [6].

Architecture Chapter

Model size is the primary architectural variable. The classic Kaplan Law [1] predicts loss ~ N^‑α, while the Chinchilla Law [11] refines the exponent for large‑scale training. The author discusses whether to count embeddings in the parameter count (see 2406.12907 [7]) and explores width‑vs‑depth trade‑offs using results from 2606.25008 [8], 2505.10465 [9], and 2602.05970 [10].

Sparse architectures such as Mixture‑of‑Experts (MoE) are introduced. The effective parameter count separates total parameters from activated parameters, following the “Effective Parameter Count” concept of 2202.01169 [16] and Ling Law [17]. The scaling law for MoE includes a sparsity factor that influences the coefficient but not the exponent, preserving the heterogeneous power inequality structure.

Memory‑based sparsity (PKM, UltraMem, Over‑Encoding, Engram) is treated similarly, showing how activation‑only parameters reduce compute while keeping loss low, and how optimal allocation between MoE and memory layers can be derived by the same inequality.

Data Chapter

Data error depends on training‑set size and the number of epochs. The author distinguishes single‑epoch and multi‑epoch regimes, noting that multi‑epoch training can cause over‑fitting. Recent works such as “Prescriptive Scaling Laws for Data‑Constrained Training” [28] and “Scaling Data‑Constrained Language Models” [29] are cited, and the optimal number of epochs is derived by balancing the decreasing term (more data) against the increasing over‑fit term.

Power‑Law Discussion

The article argues that power‑law forms are natural for monotonic decreasing relationships because they are scale‑invariant and linear in log‑log space, making them easy to fit and analyze. Exponential forms decay too quickly for realistic scaling, while keeping exponents fixed across architectural or optimizer changes preserves the analytical tractability of the heterogeneous power inequality.

Conclusion

The author acknowledges that the work is heuristic rather than a formal proof, but the three‑way decomposition and the heterogeneous power inequality provide a unified framework to relate optimizer, architecture, and data scaling. The derived optimal settings align qualitatively with known laws (Kaplan, Chinchilla, Step, Microsoft, DeepSeek) and suggest directions for future, more precise analysis.