Essential AI Reading List: LLMs, AutoGPT, Distributed Training & More

Dolly 2.0: The First Fully Open‑Source Instruction‑Following LLM

Two weeks ago Databricks released Dolly, a ChatGPT‑like LLM trained for under $30. Dolly 2.0 is the industry’s first open‑source instruction‑tuned LLM, fine‑tuned on 15,000 high‑quality prompt/response pairs. It is based on EleutherAI's Pythia series and has 12 B parameters. The model, training code, dataset, and weights are fully open and commercially usable.

Links: https://huggingface.co/databricks, https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm

Comprehensive LLM Survey: From T5 to GPT‑4

Researchers from Renmin University of China compiled a thorough review of recent LLM advances, focusing on pre‑training, instruction fine‑tuning, usage, and evaluation, and provided a curated list of resources and future directions.

Link: https://mp.weixin.qq.com/s/7HRr55Md2Wl6EHQMGioumw

OpenAI Founder on GPT‑4 Origins and Secrets

John Brockman discusses the years of research behind GPT‑4 and emphasizes that reinforcement learning from human feedback (RLHF) is the “secret sauce” that enabled ChatGPT’s breakthrough.

Link: https://mp.weixin.qq.com/s/hO1ZdqgOjpA328luobQ9eg

Technical Deep‑Dive: BLOOM’s 176 B‑Parameter Training

An analysis of the hardware and software engineering required to train the 176 billion‑parameter BLOOM model, aiming to spark discussion on large‑scale model training techniques.

Link: https://zhuanlan.zhihu.com/p/615839149

Distributed Training: Top 10 Common Errors and Solutions

A practical guide enumerating the most frequent pitfalls in distributed model training and offering concrete remedies.

Link: https://neptune.ai/blog/distributed-training-errors

AutoGPT: The Rise of Autonomous AI

AutoGPT, highlighted by former Tesla AI director Andrej Karpathy, represents a new trend of autonomous AI agents capable of completing tasks without human intervention.

Link: https://mp.weixin.qq.com/s/bV1tPc7hNn2z06YOpzyanw

LLM Reading List for Beginners

A curated list of essential papers and resources to help newcomers understand the impact of Transformers and LLMs.

Link: https://sebastianraschka.com/blog/2023/llm-reading-list.html?

Large‑Scale Model Summary (Models >1 B Parameters)

An overview of mainstream large language models exceeding one billion parameters.

Link: https://zhuanlan.zhihu.com/p/611403556

ML Systems Intro: TVM, MLIR, LLVM

A starter guide for developers interested in ML systems and compiler technologies.

Link: https://zhuanlan.zhihu.com/p/618229430

OpenAI Triton Explained

An overview of Triton, an open‑source project built on MLIR that delivers high performance on major AI accelerators.

Link: https://zhuanlan.zhihu.com/p/613244988

mperf: Performance Tuning for Mobile/Embedded Operators

A toolbox for micro‑architectural performance tuning of operators on mobile and embedded CPUs/GPUs, aiming to close the feedback loop for optimization.

Link: https://zhuanlan.zhihu.com/p/610346564

Mini Python Compiler Project

A beginner‑friendly project that builds a small Python compiler using CuPy’s rawKernel to compile CUDA code into Python functions, with the entire codebase under 100 lines per module.

Link: https://zhuanlan.zhihu.com/p/603352525

CUDA Programming Tips and Tricks

A collection of practical CUDA coding techniques and examples to improve efficiency and discover better solutions.

Link: https://zhuanlan.zhihu.com/p/584501634

NCCL Source Code Walkthrough: Initialization and Unique ID Generation

An analysis of NVIDIA’s open‑source GPU communication library NCCL, covering its initialization process and unique ID creation.

Link: https://mp.weixin.qq.com/s/_SOmkGoo9DblXb8ddyEeaQ

OneFlow Enhancements: FX Integration for Simplified Quantization‑Aware Training

OneFlow adds an FX module (one‑fx) that enables straightforward quantization‑aware training via import onefx as fx.

Link: https://mp.weixin.qq.com/s/O8yGUuTL-o_gHQV4xez_nQ

One‑YOLOv5 v1.2.0 Release: Supports Classification, Detection, Instance Segmentation

The new version aligns with Ultralytics YOLOv5 v7.0, adds Flask REST API support, integrates wandb for experiment tracking, and improves training performance through various optimizations.

Link: https://mp.weixin.qq.com/s/… (original article)