Can a 2M‑Parameter Model Outperform XGBoost? Inside LimiX‑2M’s Tabular AI Breakthrough

Background and Problem

Large language models (LLMs) have demonstrated impressive capabilities on unstructured data, yet they often fail to handle structured tabular data effectively. In many critical domains—such as power‑grid scheduling, user modeling, and communication logs—key information resides in tables, and conventional deep learning approaches struggle to outperform classic gradient‑boosting methods like XGBoost and CatBoost.

LimiX‑2M Overview

The Tsinghua University team led by Cui Peng introduced LimiX, a family of transformer‑based models designed for general‑purpose tabular learning. LimiX‑2M, with only 1.94 M parameters, supports classification, regression, and missing‑value imputation in a single model, offering an “open‑box” solution that requires no task‑specific fine‑tuning.

Performance Evaluation

Benchmarks on 11 authoritative datasets show that LimiX‑2M consistently ranks second only to the larger LimiX‑16M, surpassing AutoGluon, TabPFN, and all traditional tree‑based models. In classification tasks (e.g., BCCO‑CLS) it captures the top‑two positions, while in regression (e.g., CTR23) it finishes third, all without any task‑specific training.

When fine‑tuned on the analcatdata_apnea2 dataset, LimiX‑2M improves AUC by 11.4 % while requiring only 60 % of the compute time of PFN‑V2.5, and it can be fine‑tuned on a consumer‑grade RTX 4090, unlike PFN‑V2.5 which demands larger GPU memory.

Training Data Generation

LimiX is pretrained entirely on synthetic data generated via a Structural Causal Graph (SCG) pipeline. The SCG creates directed acyclic graphs that model causal dependencies among features, ensuring diverse and controllable training distributions that enhance generalization.

Core Innovation: Radial Basis Embedding Layer (RaBEL)

Traditional tabular transformers embed numeric features using linear projections, which leads to a “low‑rank collapse”: shallow activations become highly correlated, limiting the model’s ability to capture non‑linear relationships. LimiX‑2M replaces this linear embedding with a Radial Basis Function (RBF) based embedding.

Each numeric value is mapped to a set of locally responsive RBF units, each defined by a learnable center and bandwidth per column. This creates a rich, non‑linear representation that captures local patterns directly at the embedding stage. A shared linear projection then maps these RBF responses into the model dimension, providing high effective rank from the first layer onward.

The result is a model that can represent complex, piecewise, or multimodal feature interactions early in the network, dramatically improving gradient utilization and overall expressiveness.

Conclusion

LimiX‑2M demonstrates that a carefully designed lightweight transformer can rival or exceed heavyweight models and classic tree ensembles on tabular tasks, while remaining computationally efficient, easy to deploy, and privacy‑friendly. Its open‑source release invites the community to adopt a high‑performance, low‑resource solution for a wide range of structured‑data applications.

Key Visuals