Tagged articles
LLM training
51 articles · Page 1 of 1
This guide introduces the scaling ladder method for PhD students to predict model performance across compute scales, optimize MFU, and efficiently allocate limited academic GPU resources through structured experiments and practical configuration tips.
GPU optimizationLLM trainingMFU
0 likes · 13 min read

Soft-MTP decomposes the cross-entropy gradient in multi-token prediction into a distribution-alignment term and a residual term, then cancels the residual from gradients flowing to MTP heads and the shared backbone while preserving cross-entropy training on the shared output projection, improving language modeling and speculative decoding without extra parameters or full KL computation.
Cross-Entropy LossGradient DecompositionKL Divergence
0 likes · 17 min read

Researchers from Tianjin University and Alibaba propose MIPU, a reinforcement learning method that addresses training-inference mismatch in LLMs by directly optimizing inference-time performance, achieving higher accuracy and more stable training on Qwen3 models across math benchmarks.
LLM trainingMIPI PrincipleMIPU
0 likes · 11 min read

Datawhale community launches Diy-LLM, a Chinese-language tutorial built on Stanford CS336 covering tokenizers, Transformer architecture, MoE, distributed training, scaling laws, inference, alignment, and multimodal models through 16 chapters and 6 practical assignments with code.
DatawhaleLLM trainingMixture of Experts
0 likes · 9 min read

The article analyzes Xiaomi's MiMo model progress, clarifying its autonomous training foundation, distinguishing open-source reuse types, examining RL training costs, and evaluating talent impact versus commercial returns, arguing that technical output and organizational capability matter more than hype.
AI model evaluationLLM trainingMiMo
0 likes · 13 min read

Researchers from Huawei, CUHK, and HKUST open-source LegoFlow, a framework that automates the entire code data pipeline—from mining 12M GitHub PRs to 5,000 verified tasks, generating 2,780 trajectories, and training models achieving 70.2% on SWE-bench Verified—while revealing that task quality outweighs quantity, stronger models cheat more, and reasoning depth matters more than coverage.
LLM trainingLegoFlowRecursive Self-Improvement
0 likes · 18 min read

A curated roundup of 14 trending GitHub repositories covering scientific research agents, virtual iPhone testing, SEO automation, AI refusal modification, patent workflows, voice cloning, coding agents, time-series forecasting, LLM training internals, MCP servers, screenshot-to-code, App Store tooling, voice input, and architecture diagram generation.
AI AgentsGitHubLLM training
0 likes · 14 min read


Geek LabsSep 7, 2026 · Artificial Intelligence
The open-source project ai-engineering-from-scratch offers a 523-lesson, 20-phase curriculum that teaches AI engineering by building reusable tools from scratch, integrating coding agents as personalized tutors to bridge the gap between using AI tools and understanding their internals.
AI AgentsAI EngineeringLLM training
0 likes · 12 min read

This practical guide details seven core data sources for training a cybersecurity LLM—including academic papers, standards like MITRE ATT&CK, CVE/CWE databases, textbooks, community blogs, open datasets, and model distillation via EasyDataset—providing specific acquisition channels and reproducible Python code for each.
CVEEasyDatasetLLM training
0 likes · 28 min read

The article explains what an AI workstation is, its key components such as GPUs or NPUs, large memory, high bandwidth and efficient cooling, how it bridges desktop PCs and rack servers, and classifies workstations by form factor and compute level while referencing a 2026 global AI compute report.
AI acceleratorAI workstationGPU
0 likes · 3 min read

DeepSeek released the V4‑Flash model with the same architecture as the preview but a revamped post‑training pipeline—SFT, reinforcement learning with GRPO, and distillation—yielding dramatic benchmark jumps and illustrating how post‑training now defines the model's real‑world capabilities.
DeepSeekGRPOLLM training
0 likes · 11 min read

The DIVE study shows that expanding the diversity of tool pools and task structures, rather than merely increasing the amount of homogeneous training data, dramatically improves LLM agents' ability to generalize to unseen tools, as demonstrated by a 12k‑vs‑48k experiment and reinforced by a four‑stage synthesis pipeline and RL fine‑tuning.
AI AgentsDIVELLM training
0 likes · 14 min read

This guide breaks down the complete workflow for building a large language model—from tokenization and pre‑training to data curation, scaling laws, alignment via RLHF/DPO, and robust evaluation—showing why architecture is less critical than data, scaling, and engineering.
AI EngineeringLLM trainingModel Evaluation
0 likes · 12 min read

Unsloth and NVIDIA identified three low‑level bottlenecks in LLM fine‑tuning on consumer GPUs—repeated packed‑sequence metadata construction, serialized copy‑and‑compute during gradient checkpointing, and per‑expert routing overhead in MoE—and applied targeted patches that together deliver roughly a 25% speedup without changing hardware, code, or frameworks.
GPU optimizationGradient CheckpointingLLM training
0 likes · 12 min read

The article analyzes DeepSeek V4’s post‑training pipeline, explains how multi‑expert on‑policy distillation (OPD) differs from traditional teacher‑forcing, compares reverse‑KL and forward‑KL objectives, and uses analogies to human learning to illustrate the benefits and limits of OPD.
LLM trainingMulti-Expert ModelsOn-Policy Distillation
0 likes · 11 min read

DeepSeek has released TileKernels, a GPU kernel library written in the TileLang DSL, that targets H100/H200/B200 GPUs and claims to approach hardware limits in compute intensity and memory bandwidth, offering MoE routing, FP8/FP4 quantization, and dual‑language PyTorch references for deep‑learning engineers.
FP8 quantizationGPU optimizationLLM training
0 likes · 9 min read

Standard QA datasets fall short for deep research tasks because they lack the multi‑step, dynamic reasoning required; this article explains why, outlines four data‑construction techniques—SailorFog‑QA, WebFrontier, WebShaper, E2HQA—details trajectory sampling, filtering, scale considerations, and interview‑ready explanations.
AI AgentsData ConstructionLLM training
0 likes · 16 min read

Karpathy’s open‑source “autoresearch” project demonstrates how a compact LLM training environment on a single GPU can let an AI agent autonomously modify code, run five‑minute training experiments, evaluate improvements, and iteratively produce better models, illustrating a new research paradigm where AI conducts experiments while humans design the system.
AI research automationAutoResearchKarpathy
0 likes · 6 min read

A recent study reveals that the widely used KL regularization in LLM reinforcement learning (RLVR) is mathematically biased, leading to unstable training and poorer generalization, and shows that moving the KL term back to the reward with a simple K1 estimator can boost out‑of‑domain performance by up to 20%.
AI researchKL regularizationLLM training
0 likes · 10 min read

DeepSeek’s new paper “mHC: Manifold‑Constrained Hyper‑Connections” proposes a novel architecture that replaces traditional residual connections with mathematically constrained hyper‑connections, showing on a 27B model a modest 6.7 % training‑time increase but significant stability gains and superior performance on BBH, DROP and GSM8K benchmarks.
DeepSeekHyper-ConnectionsLLM training
0 likes · 8 min read

This article explains why function call capabilities in large language model agents require dedicated training, outlines the four core abilities to teach, describes the structure and sources of effective training data, and compares lightweight LoRA fine‑tuning with full supervised fine‑tuning approaches.
Agent SystemsData GenerationLLM training
0 likes · 11 min read

The article details the SmolLM3 3B‑parameter model, its architecture, dual‑mode inference, a three‑stage data‑curation strategy, rigorous ablation methods, preference optimisation (APO/DPO), model merging, and practical training‑stability tricks, offering a comprehensive guide for building high‑performing large language models.
APOContext ScalingLLM training
0 likes · 13 min read

This article explains the GRPO algorithm, an improvement over PPO for large language model training that eliminates the value network, uses group‑relative advantage estimation, and offers flexible supervision, resulting in higher efficiency, stability, and performance on tasks such as mathematical reasoning.
GRPOLLM trainingPPO
0 likes · 16 min read

This article breaks down the hardware bottlenecks of large‑scale LLM training, explains the Roofline performance model, arithmetic intensity, and how computation and communication costs interact on GPUs and TPUs, offering concrete formulas and examples for efficient scaling.
Arithmetic intensityGPULLM training
0 likes · 12 min read

In the era of data‑driven AI, Ray offers an open‑source unified compute framework that abstracts distributed system complexity, enabling developers to seamlessly scale Python code from a laptop to large GPU clusters, and provides the Ray AI Runtime (AIR) with libraries such as Ray Data, Train, Tune, and Serve to accelerate LLM training, hyper‑parameter tuning, and model serving.
AI RuntimeLLM trainingPython
0 likes · 10 min read

The paper "Efficient Long Context Fine-tuning with Chunk Flow" introduces ChunkFlow, a training framework that reorganizes variable‑length sequences into fixed‑size chunks, achieving up to 4.53× speedup and more stable GPU memory usage for large language models.
ChunkFlowGPU optimizationLLM training
0 likes · 7 min read

The article introduces QK-Clip, a lightweight per‑head weight‑clipping technique that uses the MaxLogit signal to prevent uncontrolled logit growth in massive LLMs, explains its design, compares it with prior methods, and shows that it stabilizes training without harming model performance.
Attention stabilityLLM trainingMaxLogit
0 likes · 15 min read

This step‑by‑step tutorial shows how to set up CUDA 12.4, install required packages, prepare a JSON dataset and custom reward, troubleshoot out‑of‑memory errors, and launch DeepSeek R1 training on an 8‑GPU A100 cluster using Accelerate, Deepspeed zero‑3 and vLLM configurations.
A100CUDADeepSeek
0 likes · 9 min read


ArchitectMar 16, 2025 · Artificial Intelligence
This article walks through the complete lifecycle of building a small large‑language model, covering token‑level inference, pre‑training, post‑training steps such as supervised fine‑tuning, reward‑model creation, and reinforcement‑learning methods like DPO, PPO and GRPO, culminating in a practical 0.5B model fine‑tuned for chain‑of‑thought reasoning.
GRPOLLM trainingReinforcement Learning
0 likes · 22 min read


ArchitectFeb 25, 2025 · Artificial Intelligence
DeepSeek R1 builds on the DeepSeek V3 base model using a multi‑stage reinforcement learning pipeline—including GRPO optimization, rule‑based reward modeling, supervised fine‑tuning, language‑consistency rewards, rejection sampling, and distillation—to produce a high‑performing, aligned LLM capable of accurate reasoning.
DeepSeekLLM trainingReward Modeling
0 likes · 24 min read


ArchitectFeb 24, 2025 · Artificial Intelligence
The article details the development, architectural evolution, and practical challenges of MoBA—a sparse attention framework inspired by Mixture‑of‑Experts that scales LLM context length to 10 M tokens, supports seamless switching between full and sparse attention, and is now released as a minimal open‑source solution.
AI ArchitectureContext ParallelLLM training
0 likes · 13 min read

This article provides a step‑by‑step, code‑first guide to reproducing DeepSeek‑R1 from the ground up, covering model selection, dataset preparation, custom reward functions, GRPO reinforcement‑learning training, supervised fine‑tuning, reasoning‑oriented RL, rejection sampling, and model distillation.
DeepSeek-R1LLM trainingPython
0 likes · 48 min read

This article explains the challenges of dynamic‑shape deep learning workloads and introduces BladeDISC++, an AI compiler that uses symbolic shape graphs, operation scheduling, and just‑in‑time auto‑rematerialization to dramatically reduce GPU memory peaks while maintaining training throughput.
AI CompilerBladeDISC++LLM training
0 likes · 16 min read

This article systematically reviews GPU monitoring for large‑scale AI training, covering MFU/HFU definitions, key DCGM metrics, NVLink bandwidth, common failure codes such as Xid and SXid, experimental insights on T4 and H100 GPUs, and practical case studies for diagnosing and mitigating performance drops.
DCGMGPU failuresGPU monitoring
0 likes · 26 min read

A recently discovered bug in popular LLM libraries shows that gradient accumulation can introduce significant accuracy loss compared to true large‑batch training, especially when sequence lengths vary, and the issue can be fixed by correcting the loss denominator scaling.
HuggingFaceLLM trainingdeep learning
0 likes · 6 min read

This article provides a thorough, yet concise, overview of Llama 3’s training pipeline, data handling, model architecture, scaling laws, post‑training techniques like SFT and DPO, and inference optimizations such as KV‑Cache, GQA, PagedAttention, and FP8 quantization, highlighting practical insights and benchmark results.
DPOGQAKV cache
0 likes · 32 min read

This article introduces Pai‑Megatron‑Patch, a suite of tools built on Nvidia Megatron‑LM that accelerates large language model training through dense and MoE model support, high‑precision HuggingFace↔MCore weight conversion, CPU offloading for optimizers and activations, FlashAttention‑3, and communication‑compute overlapping, and provides detailed experimental results and command‑line usage examples.
CPU OffloadingCommunication OverlapDistributed Optimizer
0 likes · 22 min read


NewBeeNLPJul 31, 2024 · Artificial Intelligence
The article shares hands‑on experience training 7‑ and 13‑billion‑parameter language models, covering essential hyper‑parameters, hardware requirements, data quality considerations, open dataset resources, and the systemic difficulties that arise when scaling to trillion‑parameter models.
LLM traininghyperparameterslarge language models
0 likes · 8 min read

This article translates and analyzes the MegaScale system—co‑developed by ByteDance and Peking University—that enables efficient, stable training of massive language models on clusters of more than 10,000 GPUs, achieving 55.2% MFU and a 1.34× speedup over Megatron‑LM.
GPU scalingLLM trainingMegaScale
0 likes · 15 min read


NewBeeNLPJun 12, 2024 · Artificial Intelligence
The article analyzes why the traditional cosine decay learning‑rate schedule hinders continued training of large language models and shows that fixed‑learning‑rate strategies such as Warmup‑Stable‑Decay, Cooldown, SWA, and Schedule‑Free Optimizer can match or surpass cosine performance while being more friendly to fine‑tuning.
CooldownLLM trainingSFO
0 likes · 7 min read

This article analyzes the main GPU/TPU cluster networking options—NVLink, InfiniBand, RoCE Ethernet, and DDC full‑schedule fabrics—examining latency, lossless transmission, congestion control, cost, power, and scalability to determine their suitability for large‑scale LLM training.
DDCData center fabricsGPU networking
0 likes · 18 min read


NewBeeNLPApr 11, 2024 · Artificial Intelligence
Andrej Karpathy released LLM.C, a pure C/CUDA implementation that trains GPT‑2‑style models in about 1,000 lines of code, detailing manual forward/backward passes, memory allocation tricks, SIMD CPU acceleration, CUDA porting, and migration tutorials, while comparing it to PyTorch and discussing broader LLM OS implications.
CUDAGPTLLM training
0 likes · 6 min read

The article analyzes ByteDance and Peking University's MegaScale system that enables efficient, stable training of large language models on clusters exceeding ten thousand GPUs, detailing algorithmic tweaks, 3D parallel communication overlap, operator optimizations, data‑pipeline improvements, network tuning, and fault‑tolerance mechanisms that together achieve a 55.2% MFU on a 175B model.
GPU clustersLLM trainingMegaScale
0 likes · 15 min read

This tutorial walks through fine‑tuning Shanghai AI Lab’s open‑source InternLM models with XTuner, explaining chat‑format conventions, loading and inference (including multimodal InternLM‑XComposer), dataset preparation, configuration sections, DeepSpeed acceleration, and memory‑efficient QLoRA details for 7‑B‑parameter chat models.
Chat FormatDeepSpeedHuggingFace
0 likes · 22 min read

This article explains how the CaR (Clustering and Ranking) approach evaluates data quality with a scoring model and selects diverse samples via PCA‑reduced sentence embeddings and K‑Means clustering, achieving comparable or superior large‑model performance while using just 1.96% of the original dataset.
CaR methodLLM trainingclustering
0 likes · 8 min read

PAI‑TorchAcc, Alibaba Cloud’s PyTorch accelerator, integrates the open‑source OLMo large language model and delivers up to 1.64× faster training on OLMo‑1B and 1.52× on OLMo‑7B by leveraging graph capture, distributed, compute, communication, and memory optimizations, with detailed usage steps and performance analysis.
LLM trainingOLMoPAI‑TorchAcc
0 likes · 7 min read

This article examines the main GPU/TPU cluster networking options—including NVLink, InfiniBand, RoCE Ethernet Fabric, and DDC full‑schedule networks—explaining their latency, loss‑less transmission, congestion control, cost, scalability, and suitability for large‑scale LLM training workloads.
GPU networkingInfiniBandLLM training
0 likes · 18 min read

This article introduces Pai‑Megatron‑Patch, an open‑source tool from Alibaba Cloud that streamlines large language model (LLM) training, weight conversion, FP8 mixed‑precision acceleration, and reinforcement‑learning workflows, providing detailed architecture, key features, code examples, and step‑by‑step usage instructions.
FP8LLM trainingMegatron
0 likes · 19 min read

This article provides a comprehensive, step‑by‑step guide to building large language models, covering tokenizer creation, vocabulary expansion, pre‑training strategies, dataset cleaning, instruction‑tuning techniques, and evaluation metrics such as C‑Eval and GPT‑4 based scoring.
LLM training
0 likes · 20 min read

The article examines why rising RM scores in large‑model training don't ensure superior LLM performance and presents seven practical RLHF tricks—ranging from KL‑penalty to global gradient clipping—that improve PPO stability and reduce resource overhead.
LLM trainingPPORLHF
0 likes · 7 min read


21CTOApr 13, 2023 · Artificial Intelligence
Microsoft has open‑sourced DeepSpeed‑Chat, a DeepSpeed‑based framework that simplifies end‑to‑end training and inference of ChatGPT‑style large language models, offering RL‑HF support, up to 15× speed‑up, massive cost reductions, and scalable performance on Azure for models ranging from billions to hundreds of billions of parameters.
AIDeepSpeedLLM training
0 likes · 7 min read
