Official platform of Tsinghua Big Data Research Center, sharing the team's latest research, teaching updates, and big data news.
This article breaks down the three core stages of large language model training—pretraining, supervised fine‑tuning, and alignment—detailing their objectives, typical data formats, scale requirements, and the latest techniques such as RLHF and DPO.
Analyzing recent DeepSeek flashmla repository commits, the article uncovers that the mysterious Model1 likely corresponds to DeepSeek‑V4, detailing architectural shifts to a 512‑dimensional head, full support for NVIDIA Blackwell GPUs, token‑level sparse MLA, and new mechanisms such as Value Vector Position Awareness and Engram.
The article introduces Huawei's VersatileFFN, an adaptive wide‑and‑deep feed‑forward design for large language models that reuses parameters to slash memory consumption while delivering stronger inference, detailing its dual‑system inspiration, technical mechanisms, experimental gains, and implications for efficient LLM deployment.
This article explains the core principles of Neural Radiance Fields, detailing how a fully‑connected network maps 5‑D coordinates to color and density, the role of positional encoding and hierarchical sampling, and provides a complete PyTorch implementation with training and rendering examples.
OptScale introduces a probabilistic framework that determines the optimal number of inference samples needed to meet a target accuracy with a confidence guarantee, dramatically reducing token usage while maintaining or improving performance across various large‑language‑model benchmarks.
The article analyzes how Andrej Karpathy's viral tweet sparked a seismic shift in software engineering, detailing the rapid rise of AI‑generated code, the emergence of AI agents as new programming abstractions, and practical steps developers and managers must take to stay relevant.
DeepSeek’s newly open‑sourced Engram module introduces a scalable lookup‑based memory that separates knowledge retrieval from computation, enabling O(1) deterministic access and significantly improving large language model performance on knowledge‑heavy, reasoning, code, and math tasks without extra FLOPs.
Ben Tossell, a non‑programmer turned AI‑driven creator, spent four months consuming 3 billion tokens to launch multiple CLI‑based projects, illustrating a new programming paradigm where prompt engineering and system orchestration replace traditional coding, and sharing practical lessons, tools, and insights from his experiments.
Learn how Python’s context manager protocol works, explore both class‑based and generator‑based implementations, and see practical examples—from file handling and database transactions to async operations—so you can prevent resource leaks, ensure exception safety, and write more maintainable code.
A recent study reveals that the widely used KL regularization in LLM reinforcement learning (RLVR) is mathematically biased, leading to unstable training and poorer generalization, and shows that moving the KL term back to the reward with a simple K1 estimator can boost out‑of‑domain performance by up to 20%.
