Always insightful, always fun
The Qwen2.5 Technical Report introduces a new series of large language models with up to 72 B parameters, expanded pre‑training data to 18 trillion tokens, advanced supervised fine‑tuning and reinforcement learning pipelines, and demonstrates strong performance across comprehension, reasoning, coding, and long‑context tasks.
This article examines Tencent's large language model deployments across content generation, intelligent customer service, and role‑playing scenarios, detailing the principles and practical implementations of Retrieval‑Augmented Generation (RAG), GraphRAG, and Agent techniques, and discusses challenges, optimization strategies, and real‑world use cases.
The article reflects on open‑source LLMs like Qwen2 and Llama 3.1, questioning whether models should self‑review answers, how hidden states might signal uncertainty, the role of loss‑function design, scaling laws, and the trade‑offs between PPO and DPO in alignment.
This article surveys current unified generation-and-understanding multimodal large-model architectures, compares LLM-centric and LLM-plus-diffusion designs, extracts common insights, details large-scale training tricks from models like Emu3, Chameleon and Janus, and outlines open research directions for visual encoders.
This article reviews the principles of absolute and relative positional encodings, explains why window extrapolation is crucial for large language models, analyzes current extrapolation methods, evaluates their performance, and answers common questions about extending LLM context windows.
This article presents a Kuaishou team's non‑autoregressive generative approach for recommendation reranking, detailing its architecture, loss design, experimental validation on Avito and Kuaishou datasets, and online A/B results that earned acceptance at KDD 2024.
This article examines various techniques for compressing and accelerating the KV cache in transformer models—including MQA, GQA, MLA, sliding‑window and linear attention, flash attention, page and ring attention, as well as mixed‑precision training and ZeRO parallelism—providing code snippets, implementation details, and practical trade‑offs.
The 30th SIGKDD conference in Barcelona featured 2,046 research papers with a 20% acceptance rate, and this article compiles the 59 recommendation‑system papers—covering large‑model recommenders, graph‑based methods, sequential models, fairness, privacy, advertising, debiasing, reinforcement learning and more—for researchers to explore the latest academic advances.
This article surveys ten recent multimodal large language model papers, covering vision representation laws, a stricter instruction benchmark, safety impacts of visual adaptation, the Mini‑Gemini architecture, automatic pruning, vision capability boosting, long‑context transfer, efficient token sparsification, math reasoning, and hallucination mitigation.
MIT’s 6.S978 Deep Generative Models seminar, taught by Associate Professor He Kaiming, offers graduate students a 15‑week deep dive into VAEs, autoregressive models, GANs, diffusion techniques, and cross‑disciplinary applications, with detailed weekly topics, required assignments, and publicly available lecture PDFs.
