From OnCall to Work Automation: How We Evolved AI from Answering Questions to Continuously Solving Problems

Problem Context

Product and engineering teams rotate on OnCall, handling live incidents. Answers are scattered across documentation, code, databases, and tacit expertise, causing high information‑retrieval cost and repeated manual triage.

Phase 1 – Answering Questions

Goal: let the AI agent act as a junior OnCall colleague by performing five core functions:

Identify the business object and infer the problem domain.

Search product documentation, knowledge maps, historical cases, and code repositories for clues.

Return a complete troubleshooting path instead of a single vague conclusion.

Explicitly indicate missing information when evidence is insufficient.

Persist high‑frequency issues and their resolution paths for future reuse.

Result: reduced cost of information retrieval. Example – when a user asked why a piece of content disappeared, the agent queried the content’s status, processing records, and applicable rules, producing a concrete evidence chain rather than a generic list of possible causes.

Phase 2 – Solving Problems

Answering alone leaves users to execute the suggested steps. The focus shifted to delivering actionable outcomes:

Fetch real configuration values.

Run data queries and verify business‑logic metrics.

Invoke backend tools to perform corrective actions.

Return verified artifacts such as reports or files.

A comparison table highlighted the difference between merely telling a user why something failed and actually performing the fix.

Phase 3 – Discovering Automation Opportunities

Automation candidates were screened using five criteria:

High frequency – the issue recurs regularly and consumes stable manpower.

Evidence availability – required information can be obtained from docs, databases, APIs, code, or UI.

Decomposable workflow – the process can be split into identifiable steps (detect, decide, execute, verify).

Measurable outcome – success can be objectively validated.

Controllable risk – permissions, human approval, gray‑release, circuit‑break, or rollback mechanisms can limit impact.

The transformation from a manual OnCall case to a repeatable workflow involved extracting stable evidence, abstracting it into SOPs, and encoding the steps as Skills and Workflows .

Case 1 – Ad‑Performance Diagnosis

The agent accessed the ad system, read the targeting configuration and audience size, and pinpointed an overly narrow audience as the root cause, delivering a concrete diagnosis instead of a checklist.

Case 2 – Content‑Operation Workflow

Daily content curation was broken into Skills and Workflows: humans define goals and risk boundaries; the AI gathers candidates, applies quality rules, and performs actions such as targeted invitations or support. The workflow runs on a schedule, records business impact, and reserves human approval for high‑risk steps.

Quality Challenges When AI Starts Doing Work

Initial success rates were >87% for simple tasks but only 18.8%–38.2% for complex tasks. After three weeks of focused engineering—adding permission checks, deterministic task‑state tracking, retry logic, and observability—the simple‑task success rose to 94.5% and complex‑task success to 74.5%.

Harness – Engineering Skeleton Around the Model

A four‑layer Harness wraps the probabilistic model with deterministic controls:

Capability Harness – aggregates knowledge maps, code, databases, OpenAPI, MCP tools, and Skills so the agent can fetch factual evidence.

Execution Harness – defines task states (queued, running, waiting, completed, failed), supports retries, file delivery, and permissioned writes.

Quality Harness – records good and bad cases, converts failures into reusable Eval test suites, and prevents regressions.

Release Harness – runs improvements in isolated Git worktrees, logs changes, executes tests and Eval, requires manual review before production, and supports rollback.

Continuous Improvement Loop

User feedback is classified as Good or Bad cases. Good cases become reusable Skills; Bad cases generate Eval suites and Knowledge updates. Successful fixes are codified; failures become traceable quality tickets. The system tracks metrics such as effective problem‑solving count, first‑answer usability, average follow‑up rounds, failure rate, Eval pass rate, tool‑call success, and automation coverage to guide further optimization.

Cognitive Shifts

From knowledge Q&A to evidence‑based investigation.

From answering to actually solving the problem.

From completing a single task to continuously improving the capability.

The journey transformed a simple OnCall helper into an AI‑driven work‑automation system that captures human expertise, automates repeatable actions, learns from failures, and delivers reliable outcomes.