FDE #07: Why Launch ≠ Value — Production Gates & Metrics That Prevent False Success

The article continues the FDE (Field Data Engineering) series, using a synthetic manufacturing scenario where an AI agent identifies at-risk orders, queries inventory and contracts, and proposes evidence-based dispositions for human confirmation. The focus shifts from integration (covered in prior articles) to the evidence required for production entry and sustained operation.

Core principle: Production gates answer "can it enter production?" while value metrics answer "is it worth continuing?" FDE must establish both evidence types but never let one substitute for the other.

Two Evidence Types That Must Not Be Conflated

Production admission and value validation occur at different times and serve different decisions:

Hard gates (admission criteria): Quality, security, operations, governance. Evaluated before first production entry and before major changes. Failure triggers block, degrade, or scope reduction.

Value evidence (operational verdict): Business outcomes, user adoption, cross-scenario reuse. Assessed during an agreed observation window after production entry. Results trigger continue, expand, contract, or exit.

Hard gates are non-negotiable: irreversible high-risk actions, unapproved data egress, unauditable failure, or missing audit trails cannot be overridden by business enthusiasm. Value evidence requires real runtime; long-term value cannot be proven on day one, but teams must show the value hypothesis is reasonable, the controlled loop has minimum validation evidence, and a concrete observation plan exists.

Two common metrics cause misalignment: offline accuracy (does not prove online data distribution, permissions, dependencies, alerts, rollback, or human takeover reliability) and active user count (may stem from mandates, training, or duplicate clicks, not business results). Even if both are true, they are insufficient to conclude production readiness and value creation.

Six-Dimensional Metrics Are Not a Pretty Dashboard

The author proposes a six-dimension metric card to check whether an FDE project is forming complete results. This is the author's framework, not an industry standard, and not all projects must use identical metrics. The dimensions are:

Delivery speed: Time from problem to controlled loop and production admission. Metrics: on-site fact confirmation cycle, change lead time, admission cycle. Illusion: rushing to shorten cycle while pushing risk to production.

Capability quality: Whether AI stably completes tasks under target conditions. Metrics: online accuracy, false positives/negatives, regression pass rate, human correction rate. Illusion: reporting only average offline accuracy while hiding long-tail failures.

Business outcomes: Whether real processes achieve measurable improvement. Metrics: disposition timeliness, delay loss, manual workload, fulfillment changes. Illusion: attributing all concurrent improvements to the project.

User adoption: Whether users continuously use the system at the correct work nodes. Metrics: target user coverage, real-path usage rate, abandonment rate, bypass rate. Illusion: counting logins, training attendance, or mandated trials as adoption.

Production operations: Whether the system sustainably and recoverably provides service. Metrics: SLI/SLO, error budget, task completion rate, recovery time, takeover rate. Illusion: watching only API availability, not business task completion.

Cross-scenario reuse: Whether capabilities migrate at acceptable cost. Metrics: common capability ratio, adaptation effort, migration cycle, regression cost. Illusion: copying code or agent instances and claiming reusability.

These six dimensions are not six parallel scores to be averaged into a single "project health" number. A blocking safety or governance issue cannot be offset by high scores elsewhere. Business outcomes still under observation should not be zeroed out to negate a system that just passed admission. The six dimensions and hard gates are two intersecting perspectives: capability quality and production operations directly contribute admission evidence; business outcomes, user adoption, and cross-scenario reuse primarily support operational decisions; delivery speed checks loop efficiency and engineering discipline. They relate but are not one-to-one mappings.

The author borrows DORA's measurement philosophy — combining throughput metrics (lead time, deployment frequency) with stability metrics (change failure, recovery) — emphasizing that speed and stability must be observed together in context. DORA metrics are not equated to the FDE six dimensions.

Every Metric Must Carry Seven Fields

A metric name alone has no decision value. Without definition and boundaries, any team can cherry-pick a flattering number. Each key metric must specify at least seven fields:

Definition: Numerator, denominator, object scope, exclusions, calculation method.

Data source: Source business system, agent task logs, monitoring platform, or manual sampling.

Observation window: Daily, weekly, monthly, or business cycle; start and end points.

Owner: Who maintains data, interprets anomalies, has authority to confirm results.

Target: What value or range constitutes success, and why that target.

Guardrails: Which side effects must not worsen while the primary metric improves.

Exit conditions: Below what boundary to pause expansion, degrade, contract, or stop.

Example: "Risk task validity rate" for the abnormal order scenario. Definition: confirmed valuable tasks / completed review tasks within agreed order scope. Data source: risk task records, confirmation results, order object references. Window: weekly observation, reviewed per agreed business cycle. Owner: business owner interprets validity, operations team maintains collection, FDE verifies chain integrity. Target: jointly determined by baseline, risk tolerance, pilot goals — no fabricated universal number. Guardrails: miss rate, high-risk false positives, manual review duration, task duplication rate must not cross boundaries. Exit condition: no improvement across agreed windows and no new evidence path → contract scope or stop investment.

Metrics must stay as close to final business facts as possible. Model confidence scores cannot replace business confirmation; agent logs cannot replace source system state; survey satisfaction cannot replace real usage paths.

Four Hard Gates Decide Production Entry

First production admission must verify four blocking categories. Specific thresholds depend on scenario risk and customer responsibility model; the article provides no context-free universal numbers.

Quality gate: capability effectiveness under near-production conditions

Beyond average accuracy, teams must cover normal, long-tail, conflict, missing data, dependency failure, and permission denial samples. High-risk false positives and false negatives must stay within agreed bounds. Regression baselines must hold after model, prompt, rule, skill, or tool changes.

Security gate: whether the responsibility chain from Article 06 is truly implemented

Evidence required for data egress inventory, identity delegation with least privilege, sensitive action confirmation, trace and audit linkage, source system final verification, exception takeover, and residual risk. Platform security capabilities do not equal customer-approved usage boundaries.

Operations gate: observability, recoverability, and controllability

Key business tasks need user- and process-oriented SLIs/SLOs, not just CPU, API success rate, or model latency. Alerts must point to concrete actions. Release, rollback, degrade, reconciliation, and human takeover require completed drills. When dependencies are unavailable or state unknown, the system must stop rather than amplify impact. The article references Google SRE's SLI/SLO/error budget framework as a control logic reference, but SLO targets must be jointly decided by business impact, user needs, and acceptable risk — not unilaterally set by FDE.

Governance gate: whether authorized parties have made responsibility decisions

Responsibility matrix, version baseline, audit evidence, applicable scope, blocking items, and residual risks must be explicit. FDE aggregates and verifies evidence; business owner accepts business risk; customer security/compliance roles make respective judgments; source system owner owns transaction and operational conditions. NIST AI RMF supports "admission is not a one-time stamp" with continuous monitoring and periodic review, but it is not the certification basis for this admission checklist nor does it automatically prove specific system compliance.

If any gate has an unaccepted high-risk blocking item, the conclusion must be "no admission" or "re-evaluate after scope reduction." Good-looking business results cannot defer security, operations, and governance to post-launch.

Value Validation Requires Longer Observation Windows

Gate passage only means the system meets entry conditions within current boundaries. Business outcomes, user adoption, and cross-scenario reuse still need validation in real operation.

Business outcomes must compare against business baselines

The abnormal order project can observe risk detection lead time, disposition cycle, repeated communication volume, or fulfillment results, but must have pre-implementation baselines. Showing only a post-launch downward curve without original levels, seasonality, or definition changes cannot prove improvement came from the system.

User adoption must observe real work paths

Logins, training attendance, or task clicks do not equal integration into work. Meaningful evidence: target roles viewing evidence, confirming, handling exceptions at prescribed nodes; how many tasks are bypassed or revert to offline; why users correct, reject, or abandon results.

Cross-scenario reuse must calculate migration cost

Deploying the same code to another business line does not prove reusability. Real observation: which business semantics, rules, data mappings, permissions, action contracts, and evaluation assets inherit directly vs. require redesign; whether migrated version still passes original gates.

These three evidence types must be written into the observation plan before admission: baseline definition, window length, data collector, review timing, results that trigger expand/contract/exit. Windows need not be uniform 30 or 90 days; they must cover the project's real business cycle. Low-frequency orders, seasonal production, and high-frequency service tasks need entirely different observation approaches.

Business Results Must State Attribution Boundaries

Business metric improvements do not inherently belong to the FDE project. Concurrent promotions, supply chain recovery, staffing changes, process governance, system upgrades, or market shifts may occur. Unrecorded, projects easily attribute all good results to AI and all bad results to external factors.

A minimum attribution statement includes:

Pre-implementation baseline: Same-definition metric level before project intervention.

Observation window: Start to end, whether it covers a full business cycle.

Project changes: Which users, objects, processes, actions actually changed.

External factors: Concurrent organizational, market, system, data changes.

Comparison method: Whether phased rollout, similar-object comparison, or interrupted time series can form a comparison.

Attributable scope: Which improvements have direct evidence, which only show correlation.

When feasible, use phased rollout, similar-object controls, or interrupted time series to strengthen judgment; when not, at minimum do same-definition before/after comparison and explicitly note external factors. The goal is not strict causal experiments in every project, but avoiding packaging correlation as definitive causation.

FDE can push experimental boundaries, data pipelines, and evidence recording, but cannot claim sole credit for business outcomes. Business improvement typically results from business people, customer tech teams, product platforms, process changes, and field engineering combined. Reports must clarify contribution and attribution boundaries, not manufacture an unauditable "AI efficiency percentage."

Metrics Must Ultimately Trigger Decisions

If metrics only feed weekly reports and dashboards without changing scope, releases, or investment, they remain decoration. After each review window, teams must choose one of four decisions:

Continue: Gates continuously satisfied, value signals directionally correct but evidence still accumulating. Action: maintain scope, continue observation and improvement.

Expand: Gates have headroom, value evidence stable, new scope risks assessed. Action: phased expansion of users, objects, or business lines.

Contract: Partial value holds but specific objects, actions, or users carry excessive risk. Action: reduce scope, degrade to assistive mode, strengthen human confirmation.

Exit: Key assumptions persistently invalid, or investment/risk/benefit imbalance. Action: stop capability, retain evidence, enter exit disposition.

Distinguish "pause release" from "exit project." Exhausted error budget may only require freezing changes and fixing reliability, not business value failure. Conversely, stable operation does not justify continuation if real users chronically bypass, business outcomes don't improve, and no new validation path exists — then contract or exit.

This is why production gates and value evidence must stay separate: they trigger different actions.

FDE Owns Evidence Integrity, Not Success Declaration

In the value proof phase, FDE is not a report compiler. It connects field problems, business models, architectural decisions, versions, operational tasks, and business results; personally participates in metric collection and verification; spots definition drift behind pretty numbers; and pushes for contraction or stop when evidence is lacking.

Specifically, FDE must drive at least four things:

Freeze core metric definitions, data sources, windows, and owners at initiation and controlled loop stages.

Summarize four hard gate evidences before production admission; do not bypass blockers with value visions.

Record baselines, external factors, and attribution boundaries during operational windows; do not attribute all concurrent improvements to the project.

Ensure every review produces a clear continue/expand/contract/exit decision with accountable actions.

This responsibility has limits. FDE cannot unilaterally set business metric targets, cannot approve production on behalf of security/compliance/source system owners, cannot redefine metrics to prove success, and cannot record mandated trials as voluntary adoption.

Success declarations should be made by accountable parties based on shared evidence, noting applicable scope and observation window. FDE owns the integrity of the evidence chain and engineering artifacts; it is not the sole arbiter of value, nor the sole creator of business results.

Gate Passage Is Only the Start of Operations

Feature launch ≠ value proof. True production admission requires valid value hypothesis, minimum validation evidence, and zero unclosed blockers across quality, security, operations, and governance. True value proof requires demonstrating in a longer window whether business outcomes, user adoption, and cross-scenario reuse support continued investment.

The six dimensions are not for scoring a project total, but so that speed, quality, outcomes, adoption, operations, and reuse can each be inspected. Metric cards are not for adding fields, but so every number has source, window, owner, guardrails, and exit conditions. Ultimately, evidence must trigger continue, expand, contract, or exit — not just make reports look better.

Article 07 answered whether the system can enter production and whether it's worth sustaining. The next article will address: when capabilities are handed to customers or agreed operations teams for independent maintenance, how to achieve verifiable handover of operational responsibility, release rollback, permissions, knowledge, and unclosed risks, and under what conditions FDE should exit.

References

AI pilots cannot stop at PoC: why explicit "graduation conditions" are needed

Trustworthy AI is not adding a review: it's building a continuous operational governance system

From LLMOps to AgentOps: why enterprise agents need a new operational system

Feature complete ≠ production ready: why AI programming needs more engineering

DORA: Software delivery performance metrics

Google SRE: Service Level Objectives

NIST AI Risk Management Framework Core

Agent tool calling is not enough: how FDE plugs AI into data, permissions, and real processes