How CTOs Can Build Systems That Make Their Own Decisions

Why Self‑Decision Is Relevant Now

Three years ago, proposing that a system automatically decides whether to scale would be met with skepticism because a wrong judgment could cause incidents and accountability issues. Two developments have changed the landscape.

First, large language models have reached a maturity point. In 2026, Claude Opus 4 series and GPT‑5 can perform structured reasoning and tool use with accuracy surpassing average human judgment in limited domains, and they can explain their reasoning in natural language, mitigating the "black‑box" fear.

Second, observability infrastructure has become standardized. OpenTelemetry is now the de‑facto standard, providing unified collection of metrics, traces, and logs, which supplies the raw material needed for automated decision making.

In short, the prerequisite capabilities are finally in place, making self‑decision a competitive necessity rather than an impossible ambition.

Core Architecture of a Self‑Decision System

The overall design follows the classic OODA loop (Observe‑Orient‑Decide‑Act) with engineering adaptations at each stage. The four‑layer architecture consists of perception, decision, execution, and feedback layers, forming a closed loop rather than a linear pipeline.

Perception Layer: From Data Collection to Situation Understanding

The perception layer does more than gather raw monitoring data; it must achieve "situational understanding"—interpreting what a metric like CPU = 80 % means in the current business context.

Key technical choices:

Unified data foundation. Use OpenTelemetry Collector for unified ingestion, storing metrics, traces, and logs in a columnar analytics engine such as ClickHouse or Apache Doris. Consolidating the three data types reduces the cost of correlation queries compared with maintaining separate systems.

Real‑time feature computation. Raw data must be transformed into high‑level features via a real‑time feature‑engineering pipeline (e.g., Apache Flink or RisingWave) that computes sliding‑window aggregates like "P99 latency change rate over the past 5 minutes" or "order conversion rate deviation from the same period".

Semantic events. Translate low‑level metric anomalies into business‑semantic events. For example, combine "payment service latency spikes" with "downstream bank channel error rate rise" into a single event: "Bank channel anomaly causing payment‑path degradation". A rule engine performs coarse filtering, followed by an LLM for fine‑grained aggregation.

Decision Layer: Agent‑Driven Reasoning Engine

The decision layer implements a "three‑brain collaboration" design:

Fast brain – rule policy store. Handles high‑certainty‑low‑latency scenarios with deterministic rules, e.g., "If a service fails health checks three times consecutively, immediately evict the instance". Implemented with a Complex Event Processing (CEP) engine, latency in the millisecond range.

Slow brain – LLM Agent reasoning. Tackles ambiguous, context‑rich situations such as "Traffic is 30 % above forecast but conversion rate is falling – should we scale out?". The LLM Agent is equipped with tool‑calling capabilities: it can query monitoring APIs, consult a historical incident knowledge base, and invoke a cost calculator. The reasoning output must be a structured chain: observation → root‑cause hypothesis → option comparison → final recommendation.

Security brain – risk‑control gateway. Every decision passes a gateway that enforces immutable "iron rules", e.g., "Production database DDL cannot be executed automatically", "Single scaling action must not exceed 50 % of current instance count", "Core transaction path cannot be auto‑downgraded".

Model selection recommendation: use Claude Opus 4 for complex reasoning steps and Claude Haiku 4.5 for high‑frequency, low‑complexity pre‑screening, balancing cost and quality.

The following diagram (textual) illustrates the internal flow of a decision request:

Semantic event input → Event classifier (Haiku fast‑path) → Rule policy store (Fast brain)
                                          ↓
                                   LLM Agent (Slow brain)
                                          ↓
                               Structured decision output
                                          ↓
                               Security gateway (Security brain)
                                          ↓
               ├─ Pass → Execute
               └─ Block → Human intervention

Execution Layer: Safe and Controllable Automation

Once a decision is approved, execution follows the principle of progressive empowerment:

Stage 1 – Suggestion mode. The system proposes actions, which on‑call engineers manually confirm. This stage collects decision logs for quality validation.

Stage 2 – Semi‑automatic mode. Low‑risk actions (e.g., non‑core service horizontal scaling, cache warm‑up) are executed automatically; high‑risk actions still require human approval. Risk is assessed by the gateway's "explosion radius" evaluation.

Stage 3 – Autonomous mode. Within the constraints defined by the security gateway, the system executes actions autonomously, each equipped with an automatic rollback mechanism. If post‑execution metrics do not improve within a predefined window, the system rolls back.

Implementation requires a unified "action abstraction layer" that wraps Kubernetes API calls, configuration changes, traffic‑routing adjustments, and database operations into a standard Action interface with two methods:

execute()

rollback()

Only actions that implement both methods are eligible for automatic execution.

Feedback Loop: Continuous Evolution of Decision Quality

The feedback layer records the full decision lifecycle—input data, reasoning chain, chosen action, and execution outcome—into persistent storage for audit and future training.

Effectiveness is evaluated at three intervals after execution: T+5 min, T+15 min, and T+60 min, measuring whether target metrics improved, whether side effects occurred, and whether decision latency was acceptable.

Periodically, senior engineers sample decision logs for post‑mortem review. Their assessments are written back to the knowledge base, refining the LLM Agent's prompts and tool‑calling strategies. High‑quality review cases can also be used for RLHF fine‑tuning of the agent.

After the loop is operational, decision quality improves over time; the first three months may lag behind experienced engineers, but by six months the system typically surpasses human response speed and stability.

Practical Hard Problems in Production

Latency of LLMs. Achieving sub‑100 ms decision latency is unrealistic for LLM inference. The three‑brain design ensures millisecond‑critical decisions go to the rule engine, while LLMs handle scenarios tolerant of seconds‑to‑tens‑of‑seconds latency. In practice, >95 % of decisions that need "thinking" are not latency‑sensitive.

Hallucination mitigation. Two safeguards: (1) Every reasoning step must be grounded in data returned by tool calls; no pure memory‑based inference is allowed. (2) The security gateway performs a final check—if the proposed action violates any iron rule, it is blocked regardless of the LLM's output.

Cost control. A single complex decision using the Opus 4 model costs a few tenths of a Chinese yuan. Processing hundreds of decisions daily results in a monthly cost in the low‑thousands of yuan, which is comparable to the salary of a senior SRE. Using Haiku for the majority of simple cases keeps overall spend low.

Team adoption. Start with non‑core scenarios such as test‑environment resource scheduling or non‑critical microservice scaling. Demonstrating value builds trust before expanding to core pipelines; heavy‑handed rollout tends to provoke resistance.

Conclusion

Self‑decision capability is not about replacing the SRE team with AI, but about freeing engineers from repetitive judgment tasks so they can focus on architecture and strategy. The technical barrier is low in 2026; the real challenge lies in organizational trust, establishing a robust feedback loop, and maintaining composure when issues arise rather than reverting to fully manual operation.

Run the system, let the data speak, and let trust become the flesh that animates the architectural skeleton.