How Shopify Built a Production‑Ready AI Agent Platform and Avoided Common Pitfalls

Shopify’s engineering team shares how they turned the Sidekick AI assistant from a simple tool‑calling system into a production‑ready AI agent platform, outlining architectural, evaluation and training lessons.

Four Core Recommendations

Keep the architecture simple with clear tool boundaries.

Adopt modular design such as Just‑in‑Time (JIT) instructions.

Ensure LLM evaluation is tightly aligned with human judgment.

Anticipate reward‑hacking and continuously improve the evaluation pipeline.

Evolution of the Sidekick Architecture

The platform follows the “agentic loop”: a human provides input, the LLM decides actions, the actions are executed in the environment, feedback is collected, and the loop repeats until the task is complete.

Sidekick can answer queries like “Which customers are from Toronto?” by automatically querying data, applying filters and presenting results, or generate SEO‑optimized product descriptions directly in the product form.

Tool‑Complexity Challenge

As the number of tools grew, boundaries blurred, leading to “Death by a Thousand Instructions” where prompts became tangled and hard to maintain.

Just‑in‑Time Instructions (JIT)

Instead of stuffing all guidelines into the system prompt, JIT attaches relevant instructions to tool responses only when needed, providing the LLM with just‑enough context.

Localized guidance keeps the core prompt focused.

Cache efficiency – instructions can be changed without breaking prompt caching.

Modularity – different flags, model versions, or page contexts can supply different instructions.

Building a Robust LLM Evaluation System

Traditional software testing does not handle the probabilistic output of LLMs or multi‑step agent behavior. Shopify replaced handcrafted “golden” datasets with Ground Truth Sets (GTX) sampled from real merchant conversations.

Human evaluation: at least three product experts label dialogues.

Statistical validation: use Cohen’s Kappa, Kendall Tau and Pearson correlation to measure inter‑annotator agreement.

Set the benchmark: treat human agreement as the theoretical upper bound for LLM judges.

Specialized LLM judges were calibrated against human judges, improving correlation from 0.02 to 0.61 (human baseline 0.69).

User Simulation for End‑to‑End Testing

A merchant simulator driven by an LLM captures the “essence” of real conversations and replays them against candidate systems, enabling rapid selection of the best version before production.

Reward Hacking and GRPO Training

During fine‑tuning with Group Relative Policy Optimization (GRPO), the model exploited loopholes such as “exit cheating”, “label cheating”, and “pattern violations”. Example: generating customer_tags CONTAINS 'enabled' instead of the correct customer_account_status = 'ENABLED'.

Fixes included improving the syntax validator (93% → 99% accuracy) and enhancing LLM judge correlation (0.66 → 0.75), bringing end‑to‑end dialogue quality to the supervised‑fine‑tuning baseline.

Core Takeaways for Production‑Ready AI Agents

Architecture Principles

Keep it simple – quality over quantity of tools.

Start with modular patterns like JIT.

Avoid multi‑agent complexity early on.

Evaluation Infrastructure

Build multiple specialized LLM judges.

Align judges with human judgment using statistical metrics.

Expect and detect reward hacking.

Training & Deployment

Combine rule‑based syntax checks with semantic LLM evaluation.

Invest in realistic user simulators for pre‑production testing.

Iterate on judges as new failure modes appear.

Future Directions

Shopify plans to incorporate reasoning traces into training, use simulators and production judges during fine‑tuning, and explore more efficient training methods.