The article reviews the evolution of large language models from the pre‑GPT‑3 era to the present, analyzes the conceptual and technical gaps between Chinese and global research, and outlines key future research directions such as scaling laws, prompting techniques, multimodal training, and efficient model architectures.
This article examines the use of Mixture‑of‑Experts (MoE) sparse training for GPT models, detailing the architecture, training and inference efficiency gains, experimental comparisons with dense models, custom routing algorithms, and step‑by‑step deployment on Alibaba Cloud AI platforms.
During Xiaohongshu’s REDtech livestream, AI researchers examined ChatGPT’s rapid adoption, versatile task performance, and underlying large‑scale pre‑training with in‑context learning, while highlighting persistent hallucinations, weak reasoning, high costs, and limited search‑engine replacement potential, and emphasized the importance of RLHF‑driven human feedback for future multimodal AI research.
In a detailed conversation, LinkedIn co‑founder Reid Hoffman and OpenAI CEO Sam Altman explore the commercial potential of large language models, the emerging AI‑plus era for science and the metaverse, and their vision for the next decade of AI breakthroughs, societal impact, and governance.
In a candid conversation, Sam Altman and Reid Hoffman explore the next stage of AI, discussing commercial opportunities of large language models, the rise of AI‑plus applications in science and the metaverse, future directions such as multimodal and continuously learning models, and the societal challenges of AGI, wealth distribution and universal basic income.
The article reviews how the Transformer breakthrough, the rapid scaling of large language models such as GPT‑3, Switch Transformer, and Alibaba's AliceMind and M6, together with multimodal research, are shaping the next phase of artificial intelligence toward more general, collaborative, and open AI systems.
The DataFun 2022 NLP Summit, held on July 30, brings together top researchers and industry leaders from Alibaba, Baidu, Microsoft, Amazon, and more to present the latest advances in large‑scale pre‑training, multimodal perception, information extraction, dialogue interaction, machine translation, and practical AI deployments, with live streaming and free registration via QR code.
The article introduces a token‑level pipeline parallelism strategy that splits the sequence‑length dimension of Transformer‑based language models, explains why this approach is feasible, presents a dynamic‑programming formulation for optimal slicing, discusses engineering challenges, and evaluates its performance on large GPT models.
Microsoft's Jigsaw tool leverages post‑processing and user feedback to improve large language model code synthesis, especially for Python Pandas, achieving up to 80% accuracy and aiming to automate code verification and debugging for developers.
The Tsinghua‑Meituan Digital Life Joint Research Institute’s Academic Salon on March 23 featured Associate Professor Liu Zhiyuan presenting the latest advances and ten key challenges in large‑model technologies, aiming to foster industry‑academia collaboration and drive innovation in representation learning, knowledge graphs, and social computing.
The article introduces Volcano Engine's veGiantModel, a high‑performance large‑model training framework built on PyTorch, Megatron and DeepSpeed, details its distributed parallel strategies, hardware setups, benchmark results showing up to 6.9× speedup over Megatron and DeepSpeed, and provides open‑source links for further use.
This article presents the PLUG 270‑billion‑parameter Chinese language model and introduces a parameter‑efficient sparsity training (PST) framework that combines unstructured and structured pruning with low‑rank decomposition to dramatically reduce model size while preserving downstream performance.
The article reviews the 2021 AI landscape, highlighting the race for ever‑larger pretrained models, the dominance of Transformers across modalities, the promise and limits of large models, the rise of multimodal systems, regulatory considerations, and the still‑nascent progress in reinforcement learning.
This article reviews three consecutive works from Alibaba DAMO Academy on compressing and distilling large pretrained language models—AdaBERT, L2A, and Meta‑KD—detailing their motivations, neural‑architecture‑search‑based designs, loss formulations, experimental results, and insights from a Q&A session.
GitHub Copilot once auto‑completed a seemingly real Chinese ID number for a public figure, sparking concerns that large language models can unintentionally expose personal data learned from public web sources, highlighting privacy and security challenges in AI‑driven code assistants.
GitHub Copilot unexpectedly generated a fabricated ID for Bilibili CEO Chen Rui, sparking concerns about how large language models trained on public data can inadvertently expose synthetic personal information and highlighting broader privacy and ethical issues in AI‑driven code assistants.
GitHub Copilot, powered by the Codex model, recently generated a seemingly real Chinese ID number for Bilibili CEO Chen Rui, sparking concerns about privacy leaks, model training data, and the broader risks of AI code assistants inadvertently exposing personal information.
This article surveys the evolution of pre‑trained models, covering the origins of transfer and self‑supervised learning, the rise of transformer‑based PTMs such as BERT and GPT, efficient architecture designs, multimodal and multilingual extensions, theoretical analyses, and future research directions for scalable and robust AI systems.
This article shares the Xiaomi AI Lab NLP team's experience in the NLPCC lightweight language model competition, discusses efficiency challenges of large pretrained models like BERT, and details practical inference optimizations—including model distillation, batching, FP16 quantization, and FasterTransformer integration—that dramatically reduce latency and hardware costs in production.
