Stanford CS329Z: Build AI Agents from Scratch — No Frameworks Allowed
Stanford's new CS329Z course teaches AI agent engineering from fundamentals to production, requiring students to build agents without frameworks and rigorously evaluate them using quadruple frameworks and LLM-as-Judge methods.
Stanford University launched CS329Z: Engineering AI Agents in fall 2024, taught by Diyi Yang, Michael Ryan, and John Yang. The course runs from September 23 to December 2 and covers the full lifecycle of building reliable AI agents.
Course Structure: Four Phases
The curriculum divides into four progressive phases:
Phase 1 — Foundations: Agent basics, RAG (retrieval-augmented generation), tool calling, and how agents fetch external information, invoke tools, and decide next actions based on execution results. Lectures include slides and videos for Agent System Basics (Sep 23), LLMs for Developers (Sep 28), RAG (Sep 30), and Tool Use & Function Calling (Oct 5).
Phase 2 — System Construction: Building agent systems using Agent Harness, DSPy, LangGraph, and other frameworks. Topics cover task-flow orchestration, context and memory management, and multi-agent collaboration. Lectures: Frameworks & Orchestration (Oct 7), Agent Design Patterns (Oct 12), Agent Memory Architecture (Oct 14), Multi-Agent Systems (Oct 19).
Phase 3 — Optimization & Data: Improving agent efficiency and stability via optimization algorithms, fine-tuning, reinforcement learning, and execution-data feedback. Lectures: Agent Optimization (Oct 21), Terminal-Bench Guest Lecture (Oct 26), Agent System Data (Oct 28), Data Selection & Quality (Nov 2).
Phase 4 — Evaluation, Safety & Application: Assessing whether code agents truly fix bugs, measuring real-world success rates, and preventing permission abuse and prompt injection. Lectures: Evaluation Fundamentals & Benchmark Design (Nov 4), LLM-as-Judge & Evaluation Infrastructure (Nov 11), Agent Safety & Defense (Nov 18). Guest lecturers include creators of Terminal-Bench and SWE-bench.
Lecture Materials
All lecture slides are hosted at https://web.stanford.edu/class/cs329z/slides/ with filenames lecture01.pdf through lecture17.pdf. Videos are on YouTube; some later lectures note "official site not yet available."
Two Core Assignments: Build, Then Evaluate
Assignment 1 — Build an Agentic Harness ( https://github.com/cs329z/assignment1-harness): Students create an enterprise internal AI assistant from scratch using only basic model APIs and their own code — no existing agent frameworks allowed. The environment provides corporate email archives and internal documents. Tasks include judging email importance, summarizing emails by date, and answering employee questions from internal docs. Automated tests, model-call caching, token-cost tracking, and budget controls force students to confront real-world cost, fault-tolerance, and debugging challenges.
Assignment 2 — Evaluate an Agent ( https://github.com/cs329z/assignment2-evals): Students switch roles to tester. Given a pre-built agent, they design a complete evaluation suite using a quadruple framework (execution environment, task goal, stop condition, scoring rules). Scoring combines automatic code-based scorers with at least one LLM-as-Judge method. Students must analyze error sources: task understanding, tool use, environment limits, or scoring-method flaws.
Course Project & Philosophy
The semester-long project requires building a complete agent (e.g., a coding agent for a specific codebase) or deeply evaluating an existing agent's failure modes. Crucially, students must pre-define evaluation tasks, data sources, success criteria, compare against baselines, and analyze success/failure conditions — not just deliver a working demo.
Two defining principles emerge:
From-scratch implementation over frameworks — students learn why agents work by building the harness themselves.
Evaluation as a first-class concern — because agent tasks are multi-step and stochastic, developers must know success rates, costs, reproducibility, and failure modes.
The course permits generative AI assistance for coding and debugging but requires students to explain and verify every submitted line. Prerequisites include computer-science and NLP foundations; it is not suited for absolute beginners.
Reference Links
Course site: https://cs329z.stanford.edu/ Logistics: https://cs329z.stanford.edu/logistics.html Project page: https://cs329z.stanford.edu/project.html Assignment 1: https://github.com/cs329z/assignment1-harness Assignment 2:
https://github.com/cs329z/assignment2-evalsSigned-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
