From Hard‑Coded Secrets to YAML‑Powered Configs: A Backend Configuration Journey

When building or developing a backend project, we often need to manage basic, nested, or reusable configuration properties that determine how the application runs. Typical properties include API keys, database configuration, logging settings, and application port numbers.

API Key : used to authenticate access to external services.

Database Configuration : host, port, and credentials for connecting.

Logging Settings : control runtime error logging level and format.

Application Port : the port on which the app listens for incoming requests.

Managing these configuration attributes is crucial in any backend project. Below is a staged process that reflects a typical beginner’s starting point and progressive improvements over time.

Stage 0 – Hard‑coding configuration in server files

Initially, I defined configuration attributes as constants directly in the server file, usually at the top of the initialization logic. It felt straightforward to have everything in one place.

Stage 1 – Moving configuration to a constants file

As the project grew, the server file became cluttered with configuration values and logic code. Different parts of the project—database connectors, service clients, utility modules—needed access to the same configuration values.

Instead of importing these configurations directly from the server file, I centralized them in a config.py file located at the project root.

<code># config.py

def get_config():
    return {
        "APP_PORT": 8000,
        "APP_AUTO_RELOAD": True,
        "API_KEY": "api-key-12345",
        "LOG_LEVEL": "DEBUG",
        "DB_HOST": "localhost",
        "DB_PORT": 5432,
    }

# app.py
from fastapi import FastAPI
from config import get_config

config = get_config()
app = FastAPI()

@app.get("/health-check")
async def get_health_status():
    return {
        "api_key": config["API_KEY"],
        "log_level": config["LOG_LEVEL"],
        "db_connection": f"{config['DB_HOST']}:{config['DB_PORT']}",
    }

if __name__ == "__main__":
    import uvicorn
    print(f"Server started on port {config['APP_PORT']} with log level = {config['LOG_LEVEL']}")
    uvicorn.run(app="app:app", host="0.0.0.0", port=config["APP_PORT"], reload=config["APP_AUTO_RELOAD"])
</code>

Stage 2 – Moving configuration properties to environment‑specific .env files

After centralizing configuration in config.py , two limitations appeared:

Environment switching difficulty : Development and production environments require different settings, and manual edits become impractical as the project scales.

Sensitive data exposure risk : API keys, database credentials, and other secrets remain in code, increasing the chance of accidental leaks.

To address this, I moved configuration attributes into environment‑specific .env files stored under config/envs . The application now loads the appropriate file based on an ENV variable.

Project Structure

<code>project-root/
  ├── app.py
  ├── configs/
  │   ├── config.py
  │   └── envs/
  │       ├── .dev.env
  │       └── .prod.env
</code>

.dev.env (for local development)

<code>APP_PORT=8000
APP_AUTO_RELOAD=True
API_KEY=dev-api-key-12345
LOG_LEVEL=DEBUG
DB_HOST=localhost
DB_PORT=5432
</code>

.prod.env (for production)

<code># This file is intentionally left empty!
# In production, these settings (including secrets) will be downloaded dynamically from a cloud‑based credential management system at deployment time.
</code>

To load the correct .env file, I use the python‑dotenv package and an ENV variable passed at server start.

<code># Install python‑dotenv first:
# pip install python-dotenv
import os
from dotenv import load_dotenv

# Determine environment and load the corresponding .env file
ENV = os.getenv("ENV", "DEV").upper()  # defaults to DEV if not set
if ENV == "DEV":
    load_dotenv(dotenv_path="config/envs/.dev.env")
elif ENV == "PROD":
    load_dotenv(dotenv_path="config/envs/.prod.env")
else:
    raise Exception(f"Unknown environment: {ENV}")

def get_config():
    return {
        "APP_PORT": int(os.getenv("APP_PORT", 8000)),
        "APP_AUTO_RELOAD": os.getenv("APP_AUTO_RELOAD", "True") == "True",
        "API_KEY": os.getenv("API_KEY", "default-api-key"),
        "LOG_LEVEL": os.getenv("LOG_LEVEL", "INFO"),
        "DB_HOST": os.getenv("DB_HOST", "localhost"),
        "DB_PORT": int(os.getenv("DB_PORT", 5432)),
    }
</code>

In app.py the import changes to:

<code>from configs.config import get_config
</code>

Now we can switch environments simply by setting the ENV variable when launching the app:

<code># Load config from .dev.env
ENV=DEV python app.py

# Load config from .prod.env
ENV=PROD python app.py
</code>

Environment flexibility : Switch configurations via the ENV variable without editing files.

Enhanced security : Production secrets are managed by a cloud credential system, keeping the local .prod.env as a placeholder.

Stage 3 – Using Pydantic models to improve configuration handling

While .env files solve environment switching and secret management, they still load all values as strings and lack validation. To address these issues, I introduced Pydantic, which validates and structures environment variables.

Update config.py to use a Pydantic settings model

<code># Install Pydantic and Pydantic Settings:
# pip install pydantic
# pip install pydantic-settings
from pydantic import Field
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    APP_PORT: int = Field(8000, description="Port to run the app on")
    APP_AUTO_RELOAD: bool = Field(True, description="Enable hot reload")
    API_KEY: str = Field(..., description="API key for external services")
    LOG_LEVEL: str = Field("INFO", description="Logging level")
    DB_HOST: str = Field("localhost", description="Database host")
    DB_PORT: int = Field(5432, description="Database port")

    class Config:
        env_file = "configs/envs/.dev.env"
        env_file_encoding = "utf-8"

def get_config(env: str = "DEV") -> Settings:
    if env.upper() == "PROD":
        Settings.Config.env_file = "configs/envs/.prod.env"
    return Settings()
</code>

Update app.py to use the validated configuration:

<code>from fastapi import FastAPI
from configs.config import get_config
import os

env = os.getenv("ENV", "DEV")
config = get_config(env)
app = FastAPI()

@app.get("/health-check")
async def get_health_status():
    return {
        "api_key": config.API_KEY,
        "log_level": config.LOG_LEVEL,
        "db_connection": f"{config.DB_HOST}:{config.DB_PORT}",
        "db_port": config.DB_PORT,
    }

if __name__ == "__main__":
    import uvicorn
    print(f"Server running on port {config.APP_PORT} with log level {config.LOG_LEVEL}")
    uvicorn.run(app="app:app", host="0.0.0.0", port=config.APP_PORT, reload=config.APP_AUTO_RELOAD)
</code>

Pydantic ensures correct types for environment variables and raises errors at startup if required values are missing or invalid, reducing runtime failures.

Stage 4 – Embracing YAML for advanced configuration

Flat key‑value pairs in .env files work well until we need nested structures. Using dot notation (e.g., database.host ) quickly becomes messy. YAML natively supports nesting, making configuration more intuitive and maintainable.

Why switch from .env to YAML?

Native nested structure support : Organize settings hierarchically without manual key concatenation.

Cleaner reuse and overrides : YAML offers anchors, aliases, and merge keys for reusable blocks and flexible overrides.

<code>app:
  port: 8000
  log_level: DEBUG
  database:
    host: localhost
    port: 5432
</code>

This clear hierarchy lets us group related settings as objects.

Example 1 – Reusing common database settings

<code># Define common DB settings with an anchor
default_db: &amp;default_db
  host: localhost
  port: 5432

env:
  app:
    port: 8000
    log_level: DEBUG
  database:
    &lt;&lt;: *default_db
    user: primary_user
  replica_database:
    &lt;&lt;: *default_db
    user: replica_user
</code>

Example 2 – Sharing logging configuration across components

<code># Define common logging settings with an anchor
default_logging: &amp;default_logging
  level: INFO
  format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

env:
  app:
    port: 8000
  module_a:
    logging:
      &lt;&lt;: *default_logging
    file: module_a.log
  module_b:
    logging:
      &lt;&lt;: *default_logging
    file: module_b.log
</code>

Using Pydantic with YAML‑based configuration

Project structure now includes YAML files for each environment:

<code>project-root/
  ├── app.py
  ├── configs/
  │   ├── config.py
  │   └── envs/
  │       ├── dev.yaml
  │       └── prod.yaml
</code>

dev.yaml

<code>app:
  port: 8000
  auto_reload: true
  log_level: DEBUG
auth:
  api_key: dev-api-key-12345
database:
  host: localhost
  port: 5432
</code>

Combine Pydantic models with YAML loading to validate nested structures:

<code>import os
import yaml
from pydantic import BaseModel, Field

class AppSettings(BaseModel):
    port: int = Field(..., description="Port the app runs on")
    auto_reload: bool = Field(..., description="Enable hot reload")
    log_level: str = Field(..., description="Logging level")

class AuthSettings(BaseModel):
    api_key: str = Field(..., description="External service API key")

class DBSettings(BaseModel):
    host: str
    port: int

class Settings(BaseModel):
    app: AppSettings
    auth: AuthSettings
    database: DBSettings

def get_config() -> Settings:
    env = os.getenv("ENV", "dev").lower()
    file_path = f"configs/envs/{env}.yaml"
    with open(file_path, "r") as f:
        raw_config = yaml.safe_load(f)
    return Settings(**raw_config)
</code>

Final app.py

<code>from fastapi import FastAPI
from configs.config import get_config

config = get_config()
app = FastAPI()

@app.get("/health-check")
async def get_health_status():
    return {
        "api_key": config.auth.api_key,
        "log_level": config.app.log_level,
        "db_connection": f"{config.database.host}:{config.database.port}",
    }

if __name__ == "__main__":
    import uvicorn
    print(f"Server running on port {config.app.port} with log level {config.app.log_level}")
    uvicorn.run(app="app:app", host="0.0.0.0", port=config.app.port, reload=config.app.auto_reload)
</code>

Hard‑coded values → constants file → environment files → Pydantic‑enhanced → YAML‑based configuration. This evolution provides a clear, scalable, and maintainable way to manage backend settings.

Total improvement space always remains…