Octavia’s Scalable Load Balancing in OpenStack: Architecture & Failover

Overview

Octavia is OpenStack’s native Load Balancer as a Service (LBaaS) solution. It replaces the legacy neutron-lbaas plugin with a dedicated service that manages load balancer objects, virtual IPs (VIPs), listeners, pools, members, L7 policies, and health monitors.

Octavia LBaaS Basics

LBaaS provides a RESTful API (v2) for creating load balancers. The service consists of three main components:

Octavia API : external REST endpoint for users.

Octavia Controller Worker : orchestrates the workflow, communicates with the Amphorae, and updates the database.

Amphora Agent : runs inside each Amphora (a lightweight VM) and hosts HAProxy, Keepalived, and a Flask‑based API for configuration updates.

Core Concepts

LoadBalancer : top‑level object that owns a VIP.

VIP : virtual IP address advertised to clients; managed by Keepalived/VRRP.

Listener : traffic entry point (protocol, port) attached to a VIP.

Pool : collection of backend members with a load‑balancing algorithm.

Member : actual backend server (IP:port) inside a pool.

Health Monitor : periodic health‑check (PING, HTTP, TCP, etc.) for members.

L7 Policy & Rule : HTTP‑level routing decisions (e.g., redirect based on host name).

Network Topology

Octavia creates three logical networks for each load balancer:

lb‑mgmt‑net : connects Amphorae to the OpenStack management network.

tenant‑net : where backend members reside.

vip‑net : provides the floating IP address reachable by clients.

Creating a Load Balancer

The creation flow consists of the following steps:

Validate request and quota.

Create database records for load_balancer and vip.

Allocate a VIP port via Neutron.

Select or create Amphorae (single or ACTIVE_STANDBY topology).

Plug the VIP port into each Amphora’s network namespace.

Generate Keepalived configuration and start the service.

Key tasks in the controller worker include AllocateVIP, PlugVIP, AmphoraePostVIPPlug, AmphoraVRRPUpdate, and AmphoraVRRPStart.

Listener Creation

When a listener is created (e.g., HTTP on port 8080), Octavia updates the HAProxy configuration on each Amphora and reloads the HAProxy service.

# openstack loadbalancer listener create --protocol HTTP --protocol-port 8080 lb1

The resulting haproxy.cfg contains a frontend section bound to the VIP and a default_backend reference.

Pool and Member Management

Creating a pool adds a backend section to the HAProxy config. Members are added as server lines with health‑check parameters.

# openstack loadbalancer pool create --protocol HTTP --lb-algorithm ROUND_ROBIN \
    --listener <listener-id>
# openstack loadbalancer member create --subnet-id <subnet-id> \
    --address 192.168.1.14 --protocol-port 80 <pool-id>

HAProxy then balances traffic among the defined members.

L7 Policies, Rules, and Health Monitors

L7 policies allow HTTP‑level routing. Example: redirect requests whose host starts with "server" to a specific pool.

# openstack loadbalancer l7policy create --action REDIRECT_TO_POOL \
    --redirect-pool <pool-id> <listener-id>
# openstack loadbalancer l7rule create --type HOST_NAME \
    --compare-type STARTS_WITH --value "server" <l7policy-id>

Health monitors are defined per pool and generate external-check entries in HAProxy.

# openstack loadbalancer healthmonitor create --type PING \
    --delay 5 --timeout 10 --max-retries 3 <pool-id>

Amphora Architecture and Communication

Each Amphora runs:

HAProxy (load‑balancing daemon).

Keepalived (VRRP for VIP failover).

Amphora Agent (Flask + Gunicorn) exposing a REST API over HTTPS.

The agent loads a server certificate ( server.pem) and a CA bundle ( client_ca.pem) to enforce mutual TLS.

# /opt/rocky/octavia/octavia/amphora-agent.conf
[amphora_agent]
agent_server_ca = /etc/octavia/certs/client_ca.pem
agent_server_cert = /etc/octavia/certs/server.pem

Octavia’s AmphoraAPIClient uses the client certificate to authenticate to the agent and verifies the server certificate against the CA.

SSL Certificate Generation

Octavia ships a helper script ( create_certificates.sh) that builds a private CA and signs server certificates using the cryptography library. The generated files ( ca_01.pem, server.pem, client.pem) are injected into the Amphora via Nova user‑data.

Health Monitoring and Failover

The octavia-health-manager service receives UDP heartbeats from each Amphora. It stores the latest timestamp in the amphora_health table. If a heartbeat is missing beyond heartbeat_timeout, the Amphora is marked stale.

When a stale Amphora is detected, the HealthManager.health_check() routine triggers the failover flow:

Delete the faulty Amphora (Nova delete, port detach).

Allocate a new Amphora from the spare pool.

Copy the old Amphora’s database references (VIP, listener, pool).

Re‑plug networks, regenerate Keepalived and HAProxy configs, and start the services.

The process ensures that traffic is seamlessly redirected to the new Amphora without manual intervention.

Testing Failover

Shutting down the master Amphora’s host triggers a heartbeat timeout. The health manager logs a stale Amphora, initiates the failover, and a new Amphora appears with identical HAProxy and Keepalived configurations, confirming successful recovery.

Key Takeaways

Octavia separates control plane (API/worker) from data plane (Amphorae) for scalability.

Network namespaces isolate VIP, management, and tenant traffic.

Mutual TLS between controller and Amphora guarantees secure configuration updates.

Automated health checks and VRRP provide high availability without external load balancers.