The Six Critical Choices Every AI Engineer Must Make

1. Build vs. Buy in the LLM Era

When calling an API no longer makes sense, engineers have three options: use an API, fine‑tune an open‑source model, or build and host their own stack. Each path has distinct cost curves and failure modes. Omdia’s 2025 survey of 376 stakeholders found that 95% value the customisation of building, while 91% appreciate the faster delivery of pre‑built platforms. For request volumes under 100 k per day, APIs such as gpt‑40 Mini are cost‑effective; above 1 M requests, token costs erode profit. A 2024 analysis shows hardware and electricity account for only 20‑30% of total cost, with engineering labour consuming the remaining 70‑80%.

Practical framework:

Start with an API.

Log cost, latency, and feature attribution from day one.

Switch only when operational advantages decline.

2. Model Complexity vs. Maintainability

The CACE principle (changing anything changes everything) highlights hidden technical debt in ML systems (Sculley et al., 2015). Data dependencies are often more expensive than code dependencies (MLIP, 2024). A more complex model may improve accuracy by 2% but can require 18 months of debugging, retraining, and undocumented knowledge transfer. Teams must ask: who will debug this program in six months?

3. Data Quantity vs. Data Quality

While larger corpora improve foundation models, adding low‑quality data beyond a noise threshold flattens or degrades performance (Qi et al., 2018). Excessive “data swamp” without governance leads to weeks of cleaning, higher storage costs, and slower experiments (Sigari, 2023). In domains like medical AI, small high‑quality labelled datasets outperform large noisy ones.

4. Throughput vs. Latency (Batch vs. Real‑Time)

Batch inference aggregates predictions on a schedule, lowering cost and simplifying infrastructure but introduces staleness. Real‑time inference offers sub‑second responses at higher cost and operational complexity (Zhou, 2025). Most business problems (e.g., nightly churn scoring, weekly recommendation updates) do not need sub‑second latency; using batch inference can reduce costs dramatically. A practical signal: if users cannot tell whether a prediction is five minutes old versus five milliseconds old, choose batch.

5. Prompt Engineering vs. Fine‑Tuning

Prompt engineering is cheap, fast, and flexible, suitable for most tasks with capable models. Its downside is fragility: small input changes can cause inconsistent outputs, and long, complex prompts may break. Fine‑tuning incurs high compute and data‑prep costs (e.g., $10 k and six weeks for a customer‑support chatbot) but yields reliable, scalable performance. Studies show prompt‑optimisation tools (DSPy) can outperform fine‑tuning by 6‑19 points while reducing usage by 35× (LLM Stats, 2026). The recommended path: start with prompting; upgrade to fine‑tuning only when prompts hit a performance ceiling (≈100 k queries) or the task is stable and well‑defined.

6. Automation vs. Human‑in‑the‑Loop (HITL)

HITL exists on a spectrum: full human review of every output, full automation with anomaly monitoring, or selective oversight where low‑confidence or high‑risk predictions trigger human review. Real‑time human intervention slows systems and introduces reviewer inconsistency. In regulated domains (medical imaging, finance, law), HITL is often mandatory. Designing the boundary requires clear authority for humans to override model decisions.

Key Takeaway

The six trade‑offs share a common principle: the cost of a decision rarely pays off at the point of decision. Complex models increase maintenance after six months; real‑time systems demand continuous infrastructure. Poor data quality incurs retraining costs. Fragile prompts can cause failures, and full automation can lead to irreversible errors. Asking the right questions early helps optimise cost, performance, and risk.