Go Error Handling & Observability: From Error Trees to Microservice Fault Validation

The article opens with a scenario: an order creation endpoint shows rising error rates, logs only say query failed, the caller receives gRPC Unknown, and monitoring shows a single red line. The on-call engineer cannot determine whether the user doesn't exist, the database briefly disconnected, the write result is uncertain, or the caller already cancelled. The core problem is that the same failure loses explainable semantics across code, protocol, logs, metrics, and traces.

High-quality error handling must stably answer five questions: what failed, where it happened, what the caller can do, whether governance or alerting is needed, and how to find evidence. The article provides a progressively migratable practice: preserve the underlying cause, establish domain semantics at boundaries, safely propagate protocol results, and build a verifiable closed loop with logs, metrics, and traces.

1. Separate Five Categories of Information

A table defines five distinct information types, their examples, primary consumers, and whether they can be exposed externally:

Domain Cause (e.g., USER_NOT_FOUND, ORDER_CONFLICT) – consumed by business code and clients; must be stable and public.

Protocol Result (e.g., HTTP 404, gRPC NotFound) – consumed by HTTP/RPC callers; public.

Internal Cause (e.g., sql.ErrNoRows, driver codes, connection resets) – consumed by adaptation layers and diagnostics; not public.

Operation Context (e.g., order.create, user.repo.lookup) – consumed by logs and traces; controlled output.

Execution Decision (e.g., whether to retry, whether to count toward circuit breaker) – consumed by retryers and dependency governance; cannot be decided by error alone.

These categories are related but not interchangeable. Unavailable does not mean a payment retry is safe; HTTP 404 does not mean the service is unavailable; an interface failure does not necessarily warrant an alert. Stable business codes are public contracts; external messages are a safe presentation layer. Never use err.Error() as an API message; localization should be based on code or message key, not error text matching.

2. Understand Error Trees, Not Error Strings

2.1 %w Preserves Inspectable Causes

Using fmt.Errorf("order.create: %w", ErrUserNotFound) wraps the error so errors.Is(wrapped, ErrUserNotFound) returns true. Using %v flattens the error and errors.Is returns false. errors.Is inspects the error tree and supports custom Is(error) bool methods; it is not string comparison. Two independently created errors.New("user not found") will not match just because the text is identical. errors.As extracts types or interfaces. The target must be a pointer (e.g., &pathErr). Ordinary type assertions only check the current value and do not automatically traverse wrapper layers.

2.2 Join Is an Error Tree, Not a "Primary Error"

errors.Join(io.ErrUnexpectedEOF, context.DeadlineExceeded)

creates an aggregate that implements Unwrap() []error. errors.Unwrap only handles Unwrap() error; Is and As can traverse both forms. errors.Join ignores nil; if all arguments are nil it returns nil. However, errors.As takes the first match in depth-first order. Join argument order is neither severity nor protocol priority. Therefore, do not join multiple domain errors and then use a single errors.As to decide HTTP or gRPC responses . Join is suitable for preserving parallel diagnostics; final protocol semantics must be adjudicated by a single explicit outer error.

2.3 Typed Nil and Stack Boundaries

Returning a typed nil (e.g., var e *AppError; return e) results in a non-nil interface because the dynamic type is *AppError. Success paths must explicitly return nil. Adding nil guards to Error() does not fix err != nil semantics; fix the construction and return paths. The standard library errors.New and fmt.Errorf do not capture call stacks. Operation names explain "which business boundaries were crossed"; stacks explain "which frames were executed"; they should be designed independently. Expected business rejections usually don't need stacks; panics or few critical internal failures can collect limited stacks at designated boundaries.

3. Layered Conversion and Deterministic Adjudication

A flow diagram illustrates: Database/HTTP/RPC → Adapter Layer (identify dependency errors) → Business Layer (decide domain meaning) → Entry Boundary (map protocol and observability) → Safe Response → Logs, Metrics, Traces. The repository identifies driver errors; the service decides what they mean in the current business; the handler/RPC interceptor handles final protocol mapping and terminal state recording.

The same sql.ErrNoRows when querying a user may mean "user not found", but when reading a required record after order creation it may mean an internal invariant violation. Therefore, a generic database helper should not automatically convert all ErrNoRows to 404.

3.1 Small, Closed Error Model

The article defines a minimal AppError struct with Kind (enum: Invalid, Unauthorized, Forbidden, NotFound, Conflict, Unavailable, Deadline, Canceled, Internal), Code (registered, low-cardinality, stable public reason code), Op (operation name like order.create), and Err (internal diagnostic only). It implements Error() and Unwrap(). The model deliberately omits Retryable bool because retry depends on idempotency, commit status, remaining deadline, and global budget – a single error object cannot carry the full judgment.

Production should validate Kind, Code, and Op via constructors or a registry to avoid empty values, unknown kinds, or user input spliced into codes. A registry map example maps "USER_NOT_FOUND" to NotFound, "ORDER_CONFLICT" to Conflict, etc.

3.2 Boundaries Reject Unadjudicated Joins

Final responses follow four rules:

Business layer returns a single explicit AppError or a plain error without domain semantics.

Multiple parallel causes may be saved with errors.Join but wrapped by a deterministic outer AppError.

A Join containing multiple candidate AppError s must not enter the protocol mapper directly.

On ambiguous trees, safely downgrade to INTERNAL and log error_tree_ambiguous=true.

A ClassifyBoundary function demonstrates this: it returns a Classification with Kind and Code. It first checks for multi-unwrap (indicating a Join with multiple branches) and returns Internal/INTERNAL. Then it attempts errors.As to extract a validated AppError (checking registry consistency). Fallbacks handle context.Canceled and context.DeadlineExceeded; default is Internal/INTERNAL. A helper hasMultiUnwrap detects Unwrap() []error in the chain.

This is a conservative but deterministic boundary strategy: if aggregated diagnostics are needed, first create an explicitly classified outer AppError whose internal Err holds the Join. Do not let tree traversal order decide what the user receives.

Repository conversion example: on sql.ErrNoRows return

&AppError{Kind: NotFound, Code: "USER_NOT_FOUND", Op: "user.repo.lookup", Err: err}

; other errors wrap with fmt.Errorf("user.repo.lookup: %w", err). Do not lump all database errors into Unavailable. Unique constraint violations, syntax errors, cancellations, permission errors, and connection faults have different meanings; prioritize structured driver error codes, fallback to Internal when undetermined.

4. HTTP: Safe Responses, Handle Already-Committed Responses

http.Handler

does not return error. Recommended: business handlers return error, a single boundary handles classification, logging, and response. But once a handler has written headers or body, it cannot write JSON 500; therefore "whether response is committed" must be explicit in the model.

A mapping table aligns each Kind to HTTP status, gRPC code, and caller action:

Invalid → 400 / InvalidArgument → correct input

Unauthorized → 401 / Unauthenticated → refresh credentials

Forbidden → 403 / PermissionDenied → no auto-retry

NotFound → 404 / NotFound → business decides

Conflict → 409 / AlreadyExists or Aborted → handle conflict

Unavailable → 503 / Unavailable → only enters retry eligibility

Deadline → 504 or gateway convention / DeadlineExceeded → distinguish budget source

Internal → 500 / Internal → safe message, internal diagnostics

504 typically for upstream timeout; 408 for server waiting for complete client request; 499 is a proxy-internal convention, not standard HTTP.

Code shows a trackedWriter wrapping http.ResponseWriter to track committed, status, and bytes. The Adapt function runs the handler, classifies the error, and handles three cases:

Client canceled and context canceled: log info with response_committed=false, stop.

Response already committed: log error with response_committed=true, http_status, bytes_written, and safe error code; do not write again.

Normal: determine status and safe message via publicHTTP, log at info (or error for 5xx), write JSON {code, message}.

SSE, downloads, and streaming APIs are not suited for the generic JSON error adapter; use protocol-internal termination events or connection abort. 401 must set WWW-Authenticate per auth scheme. Production services must also configure read/write timeouts, graceful shutdown, and access logs.

5. Cross-Service: Propagate Contracts, Rebuild Local Errors

%w

only works within the current process. Across HTTP, gRPC, MQ, the caller receives a protocol result and cannot/should not use errors.Is to recover the remote sql.ErrNoRows. The server emits stable protocol codes, business reasons, and safe messages; the client adapter rebuilds local domain errors. Establish a versioned registry for public codes: owning Kind, HTTP/gRPC mapping, caller actions, deprecation policy, owner.

gRPC example uses

status.New(codes.NotFound, "resource not found").WithDetails(&errdetails.ErrorInfo{Reason: "USER_NOT_FOUND", Domain: "user-service"})

. Forbidden: status.New(code, appErr.Error()) may leak SQL, paths, addresses, parameters. Client validates status code, then checks ErrorInfo.Domain and Reason against registry; unknown reason preserves protocol semantics and degrades, never guesses business branches by string.

Interceptors distinguish: domain AppError mapped; existing valid status preserves code, rebuilds details via whitelist; unknown errors log controlled diagnostics and return Internal. Cancellation and deadline must not be swallowed by unknown error fallback.

Retry and Circuit Breaking Are Not the Same

A table contrasts failure types, retry judgment, and circuit breaker judgment:

Invalid input, no permission → usually no retry; exclude from breaker.

Read request transient disconnect → consider within budget; usually count as dependency failure.

Write request timeout → first confirm commit status, idempotency protection; per dependency policy.

Local queue timeout → fix local congestion; do not attribute to remote.

Caller canceled → stop work; usually exclude.

Auto-retry at minimum requires: error allowed, operation idempotent or reliable idempotency key, result uncertainty converged, deadline has budget, retry budget not exhausted. Backoff and jitter only mitigate synchronous retries, cannot fix duplicate writes; orders and payments should prioritize unique constraints, idempotency records, and result queries.

6. Observability: Diagnostic Safety, Stable Aggregation

"Only log one error line" means each service entry's terminal failure has a single point of responsibility, not that the entire distributed call has only one log. Low-level wraps return; entry records terminal state; retriers record key attempts; consumers record message terminal state.

A table defines required log fields and rules: service / version – service and release version. operation / route – template route or operation name, not raw URL. error.kind / error.code – low-cardinality enums from registry. trace_id / span_id – only when valid SpanContext. request_id – generated by this service or strictly validated. dependency – logical service name, no connection strings. diagnostic.* – whitelisted, sanitized, length-limited internal evidence. request_id ≠ trace_id; do not trust arbitrary client headers. Diagnostics prioritize structured driver codes, controlled constraint mappings, and dependency logical names, not full err.Error(). Logs, trace events, span status, DLQ reasons, and protocol details must share the same data classification, sanitization, and retention governance.

Code example: after ClassifyBoundary, for Internal, Unavailable, Deadline set span status to Error, set attributes app.error.kind, app.error.code, error.type, and record error event. RecordError adds event but does not auto-set span error. 404 may be success in existence probe but failure in order creation; span status depends on the operation the span represents. Auto HTTP/RPC instrumentation owns protocol spans; manual business spans own business results; avoid duplicate recording.

Metrics examples (Prometheus-style):

app_requests_total{service,route,outcome}
app_request_duration_seconds{service,route,error_type}
app_failures_total{service,operation,error_kind,error_code}
app_dependency_attempts_total{service,dependency,result}
app_retries_total{service,dependency,reason}
app_dlq_publish_total{service,topic,result}

Labels must be controlled; trace_id, user/order IDs, raw URLs, raw SQL, raw error text cannot be labels. Alert on user impact and SLO: success rate, error budget, latency, dependency failure rate, backlog, DLQ failures. Traces are sampled; logs become the correlatable fallback diagnostic path.

7. Kafka: Errors End in Offset State Machine

Kafka has no unified "handler returns error auto-retry" contract. Client, framework, auto-commit vs manual commit all change actual behavior; must crash-test against the specific Go client used.

A diagram shows the flow: Read Message → Process & Classify → Save Business Result & Deduplication Record in Same Transaction → Commit Continuous Completed Offset Prefix → Retry with Budget / Pause Partition → Confirm DLQ Reliable Write.

Constraints:

Persist business result before acknowledging completion; writing dedup record first then failing would incorrectly skip re-read.

Same partition can only commit continuous completed offset prefix; cannot skip earlier failed item.

DLQ write failure means source message still incomplete; cannot skip for throughput.

DLQ success but source offset not committed before crash allows duplicate delivery; DLQ consumer must also be idempotent.

Pause only pauses fetch, not poll/heartbeat. Slow processing must respect client liveness limits; rebuild pause and in-flight state after rebalance.

Common relational DB pattern: write business result and unique event_id dedup record in same transaction, commit transaction then commit offset. Window remains; rely on unique constraint and idempotency to converge duplicates. Kafka transactions coordinate Kafka-internal outputs with offset, cannot automatically include MySQL; cross-database event publishing should evaluate Outbox, not claim exactly-once.

DLQ must record at least: event_id, topic/partition/offset, schema version, attempt, first failure timestamp, safe error code, tracing context; message payload access permissions, retention, and replay approval also need design.

8. Resources, Goroutines, and Panic

File Close failure may change persistence result; HTTP body Close failure after read may not overturn success. errors.Join can preserve both diagnostics but cannot replace outer domain semantics.

Goroutine errors don't automatically return to caller. Use bounded result channels or errgroup.WithContext to manage completion and cancellation; tasks must still check context and use cancellable I/O. Don't use context.Background() unboundedly after request ends to continue placing orders; background work must have independent lifecycle, deadline, persistent scheduling, and terminal state observability. recover only works in deferred function of the panicking goroutine. Suitable for isolated boundaries like plugin or task executors, not for ordinary business rejection control flow; if shared state invariants are broken, blindly recovering and continuing may be more dangerous.

9. Verification and Fault Acceptance

Unit tests must cover: error wrapping, typed nil, Join ambiguity, unknown code, cancellation/timeout, external response no leakage, committed response no double-write. Real module execution examples:

func TestBoundaryRejectsAmbiguousJoin(t *testing.T) {
    err := errors.Join(
        &AppError{Kind: NotFound, Code: "USER_NOT_FOUND"},
        &AppError{Kind: Conflict, Code: "ORDER_CONFLICT"},
    )
    got := ClassifyBoundary(err)
    if got.Kind != Internal || got.Code != "INTERNAL" {
        t.Fatalf("got %#v, want INTERNAL", got)
    }
}

func TestHTTPDoesNotLeakDiagnostic(t *testing.T) {
    log := slog.New(slog.NewTextHandler(io.Discard, nil))
    h := Adapt(log, "/users", func(http.ResponseWriter, *http.Request) error {
        return &AppError{Kind: NotFound, Code: "USER_NOT_FOUND", Op: "user.repo.lookup",
            Err: errors.New("private SQL detail")}
    })
    w := httptest.NewRecorder()
    h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, "/users", nil))
    if w.Code != http.StatusNotFound || strings.Contains(w.Body.String(), "private SQL detail") {
        t.Fatalf("unsafe response: %d %s", w.Code, w.Body.String())
    }
}

Run go test ./..., go test -race ./..., go vet ./....

More importantly, fault acceptance (chaos engineering) requires injecting faults and proving outcomes:

User not found → preserve business reason, not counted as dependency availability failure.

Database disconnect → safe classification, controlled diagnostics, logs correlatable to request.

Write request timeout → no blind duplicate write; converged by idempotency key or query.

Client cancel → stop work without side effects, no circuit breaker pollution.

Partial HTTP response then failure → no second JSON write; record committed status and bytes.

gRPC unknown reason → preserve protocol semantics, no string guessing.

Token/email input → response, logs, trace, DLQ no leakage.

DLQ unavailable → do not advance source offset, explicit alert.

DB commit then consumer crash → re-read does not duplicate business side effects.

Same partition out-of-order completion → do not commit offset across earlier failure.

10. Performance and Migration

Do not cite environment-detached conclusions like " errors.Is is Nx slower" or "panic is Nx slower". Error tree depth, compiler, CPU, error rate, log backend, sampling all affect results. Measure in real modules: Is on constructed errors, error creation cost, and end-to-end throughput/latency percentiles including logs/traces.

Migration steps: inventory existing business codes and protocol compatibility; pick one entry to establish registry, boundary classification, response whitelist, unique terminal log; then refactor repositories and remote clients, remove string-based judgments; finally integrate trace propagation, bounded metrics, retry, circuit breaker, and MQ state machine. Goal is not "more logs" but fewer invalid retries, false alerts, and reduced time from alert to evidence.

When an alert can locate affected operation, jump from metrics to correlated logs, follow trace_id to dependency stage and controlled cause, and every conclusion maps to code branch, protocol field, and runtime evidence – only then does error handling form a true observability closed loop.

References

Go errors: Is, As, Join, Unwrap

Working with Errors in Go 1.13

gRPC Error handling

gRPC Status codes

OpenTelemetry: Recording errors

Apache Kafka Consumer