Credential Pool Multi-Key Rotation: How Hermes Makes Rate‑Limiting Self‑Healing

Hello, I’m James.

In the previous post we broke down Hermes' unified message format that abstracts over 200+ model APIs. This article dives into the next dimension of model‑agnostic design: credential‑agnosticism.

Background – Why a single API key isn’t enough

Using only one key leads to several failure modes:

429 Rate‑limit : 1,000 requests per hour are quickly exhausted, halting the agent for an hour.

402 Payment required : Insufficient balance causes every request to fail.

401 Token expired : OAuth tokens expire after an hour, breaking long‑running tasks.

Concurrency bottleneck : A single key supports only ten concurrent calls, causing queueing and timeouts.

Rate‑limiting is a normal API design, not a bug. A robust agent must treat it as an expected condition.

Hermes' answer: a credential pool with automatic self‑healing.

Industry Survey – How other agents handle rate limits

LangChain – Fallback chain

# LangChain: manual fallback chain
chat = ChatOpenAI(openai_api_key="key1").with_fallbacks([
    ChatOpenAI(openai_api_key="key2"),
    ChatOpenAI(openai_api_key="key3"),
])

Pros : Simple, transparent to the caller.

Cons : No cooldown logic; a key that hits 429 may be switched back to while still in cooldown, causing endless ping‑pong. The fallback list is hard‑coded, so changing keys requires code changes.

OpenAI SDK – Automatic retry

The SDK retries on 429/5xx using the Retry-After header.

Pros : Fully transparent.

Cons : Retries only the same request; it never switches to a different key, so a throttled key forces the caller to wait an hour.

VS Code Copilot – Silent token pool

Copilot maintains an internal token pool and silently switches on failure.

Pros : No user interaction.

Cons : Black‑box; users cannot see or manage the tokens.

Hermes – Differentiators

Hermes solves four independent problems:

Selection strategy – which key to use when several are available.

Status management – how to mark a key as exhausted and when to recover.

Automatic refresh – how to renew OAuth tokens without interrupting tasks.

Credential discovery – how to locate keys from environment variables, Claude Code login, gh CLI, etc.

These map to four architectural layers.

Design – Hermes' Four‑Layer Credential Architecture

Layer 1: Selection Strategy

Four built‑in strategies configurable per provider in config.yaml: fill_first (default) – always pick the highest‑priority key. round_robin – rotate keys and persist the order across processes. random – pick a random available key. least_used – pick the key with the smallest request_count.

Layer 2: State Machine

Each credential follows a lifecycle ok → exhausted → cooling → ok driven by three fields: last_status, last_error_code, and last_error_reset_at. Cool‑down defaults are:

# Python: cooldown calculation
EXHAUSTED_TTL_401_SECONDS = 5 * 60   # 401 → 5 min
EXHAUSTED_TTL_429_SECONDS = 60 * 60  # 429 → 1 h
EXHAUSTED_TTL_DEFAULT_SECONDS = 60 * 60  # other errors → 1 h

401 errors get a short 5‑minute cooldown because they usually indicate a transient auth issue; 429 errors get a full hour because the limit applies to the whole organization.

Layer 3: OAuth Refresh

Supports Anthropic, Nous, Codex, and xAI providers. Refresh is lazy – it runs only when a credential is about to be used:

# Python: trigger refresh during selection
if refresh and self._entry_needs_refresh(entry):
    refreshed = self._refresh_entry(entry, force=False)
    if refreshed is None:
        continue  # refresh failed, skip this entry
    entry = refreshed

Because many refresh tokens are single‑use, Hermes uses a three‑step race‑condition mitigation:

Pre‑refresh sync : Check auth.json (or ~/.claude/.credentials.json) for a newer token that another process may have already written.

Post‑refresh sync : Write the new token back to auth.json so other processes can see it.

Exception retry : If refresh fails, reload the latest token from auth.json and retry once.

This lock‑free approach relies on the auth store as the source of truth.

Layer 4: Seed Injection

Hermes discovers credentials from seven sources:

Environment variables (including ~/.hermes/.env).

Claude Code credentials ( ~/.claude/.credentials.json).

Hermes PKCE OAuth ( ~/.hermes/.anthropic_oauth.json).

Device‑code tokens for Nous, Codex, xAI (stored in auth.json).

GitHub CLI token ( gh auth token).

Qwen CLI token ( ~/.qwen/oauth_creds.json).

Custom providers defined in config.yaml or model.api_key.

Each source has an independent removal step, ensuring that hermes auth remove cleans up without leaving “ghost” entries.

Concurrency Lease – Soft Limits Instead of Hard Blocks

# Python: soft lease mechanism
DEFAULT_MAX_CONCURRENT_PER_CREDENTIAL = 1

def acquire_lease(self, credential_id=None):
    below_cap = [e for e in available if self._active_leases.get(e.id, 0) < self._max_concurrent]
    candidates = below_cap if below_cap else available
    chosen = min(candidates, key=lambda e: (self._active_leases.get(e.id, 0), e.priority))
    self._active_leases[chosen.id] = self._active_leases.get(chosen.id, 0) + 1
    return chosen.id

The algorithm always returns a credential, even if all are over the soft limit, because the agent must keep working; the lease merely guides traffic distribution.

Industry Comparison – Who Actually Manages Credentials?

Compared to Claude Code (single OAuth token), VS Code Copilot (black‑box pool), and Cursor (simple fallback), Hermes provides:

Unlimited multi‑key pool with per‑error‑code cooldown.

OAuth auto‑refresh for four providers with race handling.

Seven‑source seed injection.

Soft concurrency limits.

Cross‑process synchronization via auth.json.

Full user visibility through hermes auth list.

Conclusion

Hermes' credential‑pool system is a living, self‑healing component that prevents agents from stalling due to rate limits, token expiry, or concurrency caps. Its four independent layers—selection strategy, state management, lazy OAuth refresh, and multi‑source seed injection—allow each concern to evolve without affecting the others. The next article will explore model routing and dynamic switching across providers.