Agent Developer Interviews: Why Building Demos Isn't Enough for Production

Interviewing Agent Developers: The Gap Between Demos and Production

The author interviewed three candidates for Agent developer roles. All resumes listed keywords like LangGraph, RAG, MCP, Workflow, and Memory, and described projects involving LLM integration, workflow orchestration, knowledge bases, and external tools. However, deeper questioning revealed a common pattern: candidates could explain the happy path but struggled with edge cases and production concerns.

Tool Calling: Exception Handling Is Not Enough

When asked how to handle incorrect parameters in tool calls, candidates typically suggested try-catch and retry. Further probing exposed gaps:

How many retries? Which errors warrant retry?

For third-party timeouts, how to know if the operation succeeded?

If the tool is an order or payment API, retries could create duplicate orders or double charges.

The author emphasizes that catching an exception does not mean the business risk is handled. A robust solution requires:

Pre-call validation of parameters (types, required fields, business constraints).

Error classification: transient faults get limited retries; parameter errors need correction first.

Retry limits and backoff to avoid amplifying problems.

Idempotency for operations like ordering, payment, messaging to prevent duplicate execution.

Post-timeout status verification via query or reconciliation.

Clear exit after repeated failures: degrade, pause task, or escalate to human.

The system must enforce boundaries; an Agent's desire to retry cannot lead to infinite loops.

RAG: Retrieval Pipeline Setup Is Only the Start

Candidates could describe the standard RAG flow: chunk documents, generate embeddings, store in vector DB, retrieve, generate answer. The author focuses on deeper details:

Handling duplicate content, tables, separated headers and bodies, multi-paragraph answers.

Justification for chunk size and overlap — not just default settings.

Evaluation: moving beyond "ask a few questions, feels okay" to a test set with expected answers and source references, measuring recall, answer correctness, citation support, and hallucination when no answer exists.

Distinguishing retrieval failures from generation errors — they require different fixes.

Tracking bad cases to avoid regressions when changing chunking, retrieval parameters, or models.

Knowledge updates: how new documents enter the index, how old versions expire, ensuring deleted content is not recalled.

Memory: Storing Chat Logs Is Not Long-Term Memory

One candidate claimed long-term memory by storing chat history in a database and retrieving it. Questions revealed issues:

After 20 turns, why is earlier important information missing? Could be retrieval failure, summary loss, or context window truncation.

Different memory types (task progress, user preferences, long-term facts) should not be mixed.

Example: "write shorter this time" doesn't mean the user always wants short answers; updated user info should override old records.

Memory design must answer: what to write, when to write, when to update, what expires, and conflict resolution. Storing everything often just stores noise.

Agent Loop Control: Preventing Runaway Execution

The author asks: "What if the Agent repeatedly calls the same tool?" Without constraints, the Agent may loop until budget exhaustion or business incident. Required constraints:

Maximum turns, maximum execution time, token and cost budgets.

Detection of anomalous repeated calls — but not mechanically; legitimate polling may repeat. Key is whether task state advances and results change.

Confirmation steps and permission limits for high-risk operations.

An Agent proposing a step does not mean the system must execute it.

Observability: Full Trace for Debugging

Post-deployment debugging cannot rely on "run again and see." Need a complete trace per execution: inputs, model outputs, tool calls with parameters, failure points, latency, and cost. Without visibility into the execution process, explaining failures and validating fixes is difficult.

Advice for Interview Preparation

The author's takeaway: many projects demonstrate a happy path but haven't seriously addressed off-nominal scenarios. Real development starts after the normal path works. Candidates should re-examine their projects on:

Data processing details.

Tool failure fallback.

Post-timeout result confirmation.

Memory design rationale.

Loop prevention mechanisms.

Logging, permissions, and cost management.

Concrete failure cases, reasoned trade-offs, and before/after evaluation results are more convincing than a list of framework names. A running demo shows you connected the pieces; explaining why it fails and how you handle failure shows you master the system.