The UC Berkeley 2025 PhD thesis by Linyuan Gong demonstrates that large language models for code must learn program structure—legal positions, complete units, and semantic constraints—through a series of code‑structure‑aware techniques, improving translation, completion, repair, and generation tasks.
A recent study shows that pre‑training Transformers on synthetic, non‑language data generated by Neural Cellular Automata can boost language‑model performance by up to 6%, accelerate convergence by 40%, and improve downstream reasoning, even outperforming models trained on massive natural‑text corpora.
The article explains how pre‑training gives LLMs general language abilities, how fine‑tuning and alignment turn them into usable assistants, why this creates a surge of applications across domains, and what inherent limitations and engineering safeguards are needed.
This article explains the three evolutionary stages of AI—from large language models that generate text, through workflow‑enhanced systems using retrieval‑augmented generation, to fully autonomous agents capable of self‑directed decision‑making—while detailing the four core technologies that power each stage.
The article breaks down the three core stages of large‑language‑model training—pre‑training, supervised fine‑tuning, and RLHF—explaining their purpose, methods, and concrete examples while noting DeepSeek‑R1’s recent breakthrough and its implications for AI development.
The article critically examines whether the pre‑training Scaling Law still applies to Grok 3, compares its compute usage and model size with DeepSeek and OpenAI models, evaluates the cost‑effectiveness of pre‑training, RL and test‑time scaling, and explores how these insights shape future large‑language‑model development strategies.
The article examines Ilya Sutskever’s claim that pre‑training will end, argues that scaling laws still hold and data is not yet a bottleneck, highlights the scarcity of high‑quality frontier data, and explains why the industry is shifting toward inference‑time compute (o1) as a more sustainable path for large language models.
This article provides a comprehensive overview of how large language models (LLMs) are applied in recommendation systems, covering two main paradigms—LLM+RS as a component and LLM as a standalone recommender—detailing their impact on pre‑training, fine‑tuning, prompting, and future research challenges.
This article examines the conditions under which graph neural network pre‑training is beneficial, proposes a data‑centric generator framework to assess transferability, introduces a data‑active pre‑training strategy that selects informative graphs, and presents experimental results showing that using less, well‑chosen data can outperform full‑scale pre‑training.
This article provides a detailed overview of large language models (LLMs), tracing their evolution from statistical and neural language models to modern pre‑trained transformers, discussing scaling, training, adaptation, utilization, evaluation methods, available resources, and outlining current challenges and future research directions.
Tencent Robotics X unveiled how its robot dog Max combines pre‑trained AI models with reinforcement learning across three learning stages, enabling it to acquire, store, and apply skills for autonomous decision‑making in complex tasks such as the World Chase Tag competition.
The article introduces WuDaoMM and WuDaoCorpora 2.0, massive Chinese‑English multimodal datasets—including 650 million image‑text pairs, 3 TB of text, 93 TB of images, and 181 GB of dialogue—detailing their composition, formats, access options, and potential research applications.
This article provides an in‑depth overview of BERT, covering its NLP background, GLUE benchmark achievements, Transformer‑based architecture, pre‑training strategies (MLM and NSP), downstream fine‑tuning methods, and includes detailed PyTorch code implementations of its core components.
