Observability Essentials: Three Pillars, Structured Logging, Distributed Tracing & Health Checks

Observability: From Blind Flying to Instrument Panel

Observability is the ability to understand a system's internal state by examining its external outputs. In cloud-native and microservice architectures, a single user request may traverse dozens of services; without observability, diagnosing problems is akin to "blind flying."

Why observability is critical for cloud-native applications:

Understand system behavior : Provides insights into how applications execute and where bottlenecks or errors occur.

Debugging and troubleshooting : When issues arise, observability tools supply detailed information about what happened in the system.

Continuous improvement : By monitoring system performance over time, teams identify opportunities to optimize code and improve performance.

Proactive issue detection : Detect problems before users encounter them.

Implementing observability in cloud-native applications is not just a best practice—it is a necessity for maintaining, optimizing, and continuously improving applications.

The Three Pillars of Observability

Observability rests on three primary pillars, each providing a different perspective on system behavior:

The three pillars together form comprehensive observability: Metrics tell you something changed, traces tell you where the problem occurred, logs tell you why it happened .

Architect perspective : The three pillars are not mutually exclusive; they are complementary . A well-designed observability system lets them correlate—detect anomalies from metrics, locate the service via traces, and determine root cause from logs.

Structured Logging — Making Logs "Consumable"

3.1 The Problem with Text Logs

Many projects don't lack logs; they lack useful logs . Traditional text logs expose several hard problems in engineering:

High retrieval cost : Text logs suit human reading, not machine analysis. Fields buried in sentences require fuzzy matching or regex, which is costly and unstable.

Unstable fields, high collaboration cost : The same business meaning may be written as customerId today, userId tomorrow, and client_id the next day—log systems cannot form stable queries.

Cannot naturally integrate with observability stack : Without stable TraceId, SpanId, RequestPath fields, logs and distributed traces cannot be linked—only manual timestamp guesswork remains.

3.2 What Is Structured Logging

Many first encounter structured logging and focus on JSON output. In reality, JSON is merely the representation; the core is whether fields have explicit semantics and can be stably recognized by downstream systems .

Compare two approaches:

Plain text log (string concatenation, fields "buried" in sentence):

logger.LogInformation(
  $"Order {order.Id} created for customer {order.CustomerId}, cost {elapsedMs}ms"
);

Structured log (fields independent, system-recognizable):

logger.LogInformation(
  "Order {OrderId} created for customer {CustomerId}, cost {ElapsedMs}ms",
  order.Id,
  order.CustomerId,
  elapsedMs
);

With structured logging, the framework records not a single string but a LogEvent containing:

Template :

Order {OrderId} created for customer {CustomerId}, cost {ElapsedMs}ms

Properties : OrderId, CustomerId, ElapsedMs (independently queryable and aggregatable)

Metadata : timestamp, level, exception, source, thread, TraceId, and other context

Rendered to console as human-readable sentences; sent to Elasticsearch, Seq, or OpenTelemetry backends as field-structured data for querying and aggregation.

3.3 Serilog Design Philosophy and Core Mechanism

In the .NET ecosystem, Serilog is one of the most common and mature structured logging solutions. Its core advantages:

Designed around structured logging from the ground up , not bolted on later.

Mature Message Template semantics , clear field model.

Complete Sink ecosystem , supporting Console, File, Seq, Elasticsearch, OpenTelemetry, etc.

Enricher, Destructuring, filtering, level control suited for long-term production use.

Serilog core pipeline :

LogEvent is central : It is not a string but a log event object containing Timestamp, Level, MessageTemplate, Properties, Exception, and other fields. This is the foundation of Serilog's structured capability.

3.4 Integrating Serilog in ASP.NET Core

Example based on .NET 8 showing standard Serilog integration:

Install packages :

Serilog.AspNetCore
Serilog.Sinks.Console
Serilog.Sinks.File
Serilog.Enrichers.Environment

Program.cs configuration :

using Serilog;

var builder = WebApplication.CreateBuilder(args);
builder.Host.UseSerilog((context, services, loggerConfiguration) => {
  loggerConfiguration
    .ReadFrom.Configuration(context.Configuration) // from appsettings.json
    .Enrich.FromLogContext() // auto-enrich context
    .Enrich.WithMachineName() // add machine name
    .Enrich.WithEnvironmentName() // add environment name
    .Enrich.WithProperty("Application", "OrderService") // fixed app identifier
    .WriteTo.Console(
      outputTemplate: "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} {Properties:j}{NewLine}{Exception}"
    )
    .WriteTo.File(path: "logs/order-service-.log", rollingInterval: RollingInterval.Day, retainedFileCountLimit: 14);
});

// Request logging middleware (unified HTTP request logging, no per-endpoint code)
app.UseSerilogRequestLogging(options => {
  options.MessageTemplate = 
    "HTTP {RequestMethod} {RequestPath} responded {StatusCode} in {Elapsed:0.0000} ms";
});

app.Run();

Key design points : UseSerilog() plugs Serilog into the ASP.NET Core default logging pipeline.

Enrichers automatically add context (environment name, machine name, request context).

Request logging middleware records all HTTP requests uniformly, avoiding repetitive per-endpoint logging.

3.5 Structured Logging Best Practices

Field contract convergence : Unified field naming is the foundation for team collaboration and query persistence. Recommended conventions: TraceId for trace identifier, OrderId for order identifier, UserId for current user, ElapsedMs for latency in milliseconds.

Never log sensitive information : Passwords, tokens, ID numbers, credit card numbers must never appear in logs.

Use appropriate log levels : Debug (development debugging), Information (normal business flow), Warning (recoverable anomalies), Error (errors needing attention), Critical (system-level failures).

Structured error logging : Must include the Exception object so the log platform can automatically extract stack traces.

OpenTelemetry — The Unified Standard for Observability

OpenTelemetry (OTel) is a vendor-neutral open-source observability framework maintained by the Cloud Native Computing Foundation (CNCF), providing unified APIs, SDKs, and tools for generating, collecting, and exporting telemetry data.

4.1 OpenTelemetry Core Components

API : Defines interfaces for creating and managing telemetry data, embeddable directly in application code and libraries.

SDK : Implements the API, providing configuration options for processing and exporting telemetry (batching, sampling, resource detection).

Instrumentation libraries : Automatically capture telemetry from common frameworks and libraries without writing instrumentation code.

Exporters : Serialize collected telemetry and transmit to backend analysis tools (Jaeger, Prometheus, Azure Monitor, etc.).

Architect perspective : OpenTelemetry's vendor-neutral design is its core principle. Instrument once with the OTel API, export to any compatible backend without changing instrumentation code—avoiding lock-in to a specific monitoring vendor.

4.2 Distributed Tracing Core Concepts

Distributed tracing is one of OpenTelemetry's most essential capabilities. Key concepts:

Trace : Represents the complete journey of a request through a distributed system. A trace comprises multiple spans forming a tree structure.

Span : A named, timed operation within a trace. Each span contains:

TraceId : Globally unique identifier shared by all spans in the same trace.

SpanId : Unique identifier for this specific span.

ParentSpanId : Parent operation's SpanId (root span has no parent).

Name : Operation name (e.g., "HTTP GET /orders").

Start and end timestamps : Precise timing.

Attributes : Key-value pairs providing operation context (HTTP method, URL, status code).

Status : Success or failure.

Context Propagation : Mechanism for passing TraceId and SpanId across service boundaries. OpenTelemetry uses the W3C TraceContext standard via HTTP headers:

traceparent:00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

This header contains: version (00), TraceId (32 hex chars), ParentSpanId (16 hex chars), TraceFlags (sampling flag). The receiving service reads this header, creates a new span with the same TraceId, sets the caller's SpanId as its parent—linking all spans into a single, correlated trace.

4.3 Integrating OpenTelemetry in ASP.NET Core

Example: Console trace output (.NET 8):

Install packages :

dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Exporter.Console
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore

Program.cs configuration :

using OpenTelemetry;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry()
  .WithTracing(tracing => {
    tracing
      .AddAspNetCoreInstrumentation() // auto-capture ASP.NET Core Activity
      .AddConsoleExporter(); // output to console
  });

var app = builder.Build();
app.Run();

Example: Application Insights integration (YARP reverse proxy scenario):

using Azure.Monitor.OpenTelemetry.AspNetCore;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddReverseProxy().LoadFromConfig(
  builder.Configuration.GetSection("ReverseProxy")
);
builder.Services.AddOpenTelemetry()
  .UseAzureMonitor(o => {
    o.ConnectionString = builder.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];
  })
  .WithTracing(t => {
    // Listen to YARP tracing activity
    t.AddSource("Yarp.ReverseProxy");
  });

var app = builder.Build();
app.MapReverseProxy();
app.Run();

4.4 YARP Distributed Tracing Support

YARP (Yet Another Reverse Proxy), as ASP.NET Core's reverse proxy component, natively supports OpenTelemetry distributed tracing:

Automatic trace context propagation : On inbound requests, ASP.NET Core automatically propagates or creates TraceId.

YARP-specific Activities : Creates Activities for "forward request" and "cluster proactive health checks."

Listen to Yarp.ReverseProxy ActivitySource : Simply call t.AddSource("Yarp.ReverseProxy") in configuration.

Architect perspective : YARP's tracing capability at the gateway layer is critical—it is the first touchpoint for all external requests. Generating TraceId here and ensuring its propagation to all downstream services is the key enabler for end-to-end distributed tracing.

Health Checks — Foundation of Production Monitoring

ASP.NET Core provides built-in health check middleware and libraries for reporting the health of application infrastructure components.

5.1 Health Check Use Cases

Container orchestrators (Kubernetes) : Use health check probes to inspect application state; on failure, pause rolling deployments or restart containers.

Load balancers : Redirect traffic from unhealthy instances to healthy ones.

Resource monitoring : Monitor memory, disk, and physical server resource health.

Dependency availability : Test database and external service endpoint availability.

5.2 Basic Health Check Configuration

Minimal configuration for a "liveness" probe reporting whether the app can handle requests:

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHealthChecks(); // register health check service

var app = builder.Build();
app.MapHealthChecks("/healthz"); // create health check endpoint
app.Run();

Default response:

App responds to /healthz endpoint → Healthy (200).

All registered checks pass → plain text Healthy.

5.3 Custom Health Checks

Implement IHealthCheck to create custom checks:

public class DatabaseHealthCheck : IHealthCheck {
  private readonly AppDbContext _dbContext;

  public DatabaseHealthCheck(AppDbContext dbContext) {
    _dbContext = dbContext;
  }

  public async Task<HealthCheckResult> CheckHealthAsync(
    HealthCheckContext context, CancellationToken cancellationToken = default) {
    try {
      // Execute simple query to verify database connectivity
      await _dbContext.Database.ExecuteSqlRawAsync("SELECT 1", cancellationToken);
      return HealthCheckResult.Healthy("Database is reachable");
    } catch (Exception ex) {
      return HealthCheckResult.Unhealthy("Database is unreachable", ex);
    }
  }
}

Register custom checks :

builder.Services.AddHealthChecks()
  .AddCheck<DatabaseHealthCheck>("database")
  .AddCheck<RedisHealthCheck>("redis")
  .AddCheck<ExternalApiHealthCheck>("external-api");

Third-party health check libraries (e.g., PostgreSQL via Npgsql):

// PostgreSQL health check (Npgsql)
builder.Services.AddHealthChecks()
  .AddNpgSql(connectionString);

5.4 Three Kubernetes Probes

In Kubernetes, health checks typically map to three probe types:

ASP.NET Core mapping example :

app.MapHealthChecks("/health/live", new HealthCheckOptions {
    Predicate = _ => false // basic check only
});

app.MapHealthChecks("/health/ready", new HealthCheckOptions {
    Predicate = _ => true // all registered checks
});

app.MapHealthChecks("/health/startup", new HealthCheckOptions {
    Predicate = _ => false
});

5.5 Health Check Options and Customization

Filter checks (execute only checks with specific tags):

app.MapHealthChecks("/healthz", new HealthCheckOptions {
    Predicate = healthCheck => healthCheck.Tags.Contains("sample")
});

Custom HTTP status codes (default: Healthy→200, Degraded→200, Unhealthy→503):

app.MapHealthChecks("/healthz", new HealthCheckOptions {
  ResultStatusCodes = {
    [HealthStatus.Healthy] = StatusCodes.Status200OK,
    [HealthStatus.Degraded] = StatusCodes.Status200OK,
    [HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable
  }
});

Security control : Endpoints relying on host header validation are easily spoofed; combine with RequireAuthorization() to prevent unauthorized access.

Production-Grade Monitoring Design

6.1 Complete Monitoring Architecture

A production-grade observability stack comprises three layers:

1. Data Collection Layer

Structured logs (Serilog + Enrichers for automatic context enrichment)

System metrics (dotnet-counters, Prometheus .NET)

Distributed tracing (OpenTelemetry SDK)

2. Storage and Processing Layer

Log storage: Elasticsearch, Seq, Loki

Metrics storage: Prometheus, Azure Monitor

Trace storage: Jaeger, Azure Application Insights

3. Visualization and Alerting Layer

Dashboards: Grafana, Azure Dashboards

Alerting: Prometheus Alertmanager, Azure Alerts

6.2 Key Design Principles

Correlation first : Logs must contain TraceId and SpanId to correlate logs with distributed traces.

Standardization : Use OpenTelemetry as the unified collection standard to avoid single-vendor lock-in.

Sampling strategy : Full collection is prohibitively expensive at high concurrency. Sample low-value logs (e.g., 10%), collect error logs fully.

Alert tiering : P0 (core functionality failure, immediate response), P1 (performance degradation, 1-hour response), P2 (non-core issues, handle on business days).

Interview Deep-Dive Questions

"What problems do the three pillars of observability each solve?" Logs record "what events occurred" — detailed event records; Metrics answer "how severe is the problem" — numerical measures of system performance and health; Distributed tracing answers "which component is at fault" — tracks the complete request path across microservices.

"What is the fundamental difference between structured logs and plain text logs?" The fundamental difference lies in field semantics . Plain text logs output a single string with fields "buried" in the sentence, preventing stable system recognition and retrieval; structured logs output a log event object with independent property fields, directly queryable, filterable, and aggregatable by log platforms.

"How does OpenTelemetry achieve cross-service context propagation?" OpenTelemetry uses the W3C TraceContext standard , propagating context via the HTTP traceparent header. The caller includes TraceId and current SpanId in the request header; the receiver reads the header, creates a new span with the same TraceId, sets the caller's SpanId as its parent—linking all spans into a single trace.

"What is the difference between Liveness Probe and Readiness Probe in health checks?" Liveness determines whether to restart the container (container crashed or deadlocked); on failure, K8s executes RestartPolicy . Readiness determines whether to receive traffic (container starting up or dependencies not ready); on failure, K8s removes the Pod from the Service's load balancer. They serve different purposes; both are indispensable.

Closing Summary

Logs and observability are the "eyes" that let architects "see" system state in production :

Three pillars : Logs record event details → Metrics provide trends and severity → Distributed tracing locates the faulty component. All three are complementary; none can be omitted.

Structured logging (Serilog) : Essence is event + fields , not merely changing log format to JSON. Message Template + LogEvent + Sink form the core pipeline. Field contract convergence is the basis for team collaboration.

OpenTelemetry : CNCF-maintained vendor-neutral standard. Core concepts: Trace (complete request flow), Span (single operation), Context Propagation (W3C TraceContext cross-service propagation).

Health checks : Liveness (restart decision), Readiness (traffic decision), Startup (startup protection). In production monitoring, health checks underpin container orchestration and load balancer decisions.

Production monitoring stack : Data collection → Storage/processing → Visualization/alerting three-layer architecture. Key principles: Correlation first (logs contain TraceId), Standardization (OpenTelemetry), Sampling strategy, Alert tiering.