Ray History Server: Post-Mortem Observability for Ephemeral Ray Clusters

Background: The Observability Gap in Ephemeral Ray Clusters

Ray has become the foundational runtime for AI training, batch inference, data engineering, and distributed workloads. In production, Ray clusters are increasingly ephemeral — created for a specific job (training, offline inference, or a RayJob) and destroyed once the task completes. This elastic model improves resource utilization and suits expensive GPU sharing, but it introduces a critical problem: the Ray Dashboard disappears with the cluster. After a task fails, if the cluster has already been reclaimed, task status, actor lifecycles, node events, resource usage, and runtime logs become difficult to inspect fully.

Solution: Ray History Server

Ray History Server, introduced by the community and now in beta as of KubeRay v1.7, addresses this by continuously archiving key historical data produced during Ray runtime to object storage and providing a long-lived access endpoint. Even after cluster termination, users can query historical jobs, locate failure causes, view historical dashboards, and reconstruct the context of the cluster run.

In essence:

Ray Dashboard serves "currently running clusters".

Ray History Server serves "terminated clusters" and provides a unified entry point for both historical and online clusters.

Core Architecture

The architecture comprises the following core modules:

KubeRay Operator : Manages cluster lifecycle and injects the collector sidecar on demand.

Collector Sidecar : Runs alongside RayCluster head/worker pods, collecting logs, events, and dashboard snapshots.

HistoryServer : Reads archived data from object storage and rebuilds a Ray Dashboard-compatible API and UI.

Proxy Head : Serves the version-matched Ray Dashboard frontend.

OSS (Object Storage) : Backend for historical data; supports Google Cloud Storage, AWS S3/MinIO, Alibaba Cloud OSS, Azure Blob Storage, etc.

ALB / Nginx Ingress (optional) : Exposes a unified entry point to platform users.

Key Technical Design Points

2.1 Decoupled Record and Replay Phases

The system operates in two independent phases:

Record phase : The collector runs during the RayCluster pod lifetime, gathering logs, events, and dashboard snapshots, then writes them to object storage.

Replay phase : The HistoryServer reads the archived data on demand and reconstructs the dashboard API and pages.

The phases are fully decoupled: collection does not require the HistoryServer to be online, and replay does not require the original cluster to be alive. As long as data remains in OSS, historical clusters remain queryable after destruction.

2.2 Automatic Configuration Injection

In the ACK implementation, users only declare a collection toggle in the RayCluster spec. The operator automatically injects the collector container, storage configuration, and access credentials. Users do not need to repeatedly specify OSS endpoints, RAM roles, or other component-level configurations for each cluster.

2.3 Deterministic Storage Layout

Data is organized by resource ownership, with paths containing resource type, namespace, cluster name, and session. This layout naturally isolates data by namespace and cluster, facilitating listing of historical jobs, multi-tenant filtering, auditing, and lifecycle management.

2.4 Lazy Loading with Bounded LRU Cache

HistoryServer employs on-demand loading: opening the cluster list reads only lightweight indexes; detailed data is loaded only when a specific session is selected, and cached in an LRU cache with configurable entry count and memory limits. Consequently, a single HistoryServer can serve multiple clusters and large volumes of historical tasks.

2.5 Unified Access for Live and Dead Clusters

HistoryServer simultaneously supports running (live) and terminated (dead) clusters: live clusters are proxied to the active Ray Dashboard; dead clusters are reconstructed from archived data. In both cases, users see the same dashboard interface.

2.6 Authentication and Security

Kubernetes token-based authentication enabled by default; browser access redirects to a login page.

HTTPS supported; authentication cookies use the Secure attribute in HTTPS mode.

OSS credentials obtained via RRSA (RAM Role for Service Accounts), eliminating static access keys.

No Ingress created by default, avoiding unnecessary load balancer resources.

2.7 Typical Use Cases

RayJob post-mortem debugging : After a job finishes and the cluster is released, users can still open the historical session to inspect job status, events, and logs.

Unified entry for training platforms : Multiple teams share one HistoryServer endpoint; historical tasks are organized by namespace.

Long-term audit and data analysis : Historical data stored in OSS can be retained long-term using lifecycle and archival policies.

Reduced resource costs : No need to keep idle clusters running just to preserve dashboard access.

Using Ray with History Server on ACK

3.1 Access Modes

HistoryServer exposes a unified entry point: all traffic reaches the entry first, then routes to either HistoryServer or the dashboard proxy. Users access a single address without needing to understand internal service relationships. Three access modes are provided:

Local verification : Port-forward the unified entry to localhost; browser accesses directly. Suitable for functional validation and development debugging.

Production access : Expose the unified entry via ALB Ingress or Nginx Ingress (NLB) to platform users, with custom domain and HTTPS support.

VPC internal access : The entry resolves to a private IP, accessible within the VPC (ECS, Direct Connect, VPN).

All three modes share the same entry, preserving consistent login, cluster selection, and dashboard navigation experience.

3.2 Enablement and Alibaba Cloud Integration

Enabling on ACK is straightforward: activate the full HistoryServer stack via the kuberay-operator component with one click, then declare the collection toggle in the RayCluster. The operator automatically injects the collector container, storage config, and credentials. Native Alibaba Cloud ecosystem integration includes:

Deep integration with Alibaba Cloud OSS as the default historical data storage backend.

RRSA for temporary storage credentials, avoiding static access keys.

RAM roles for pod-level permission isolation.

Compatibility with ALB and Nginx Ingress (NLB), commonly used in China.

Built-in authentication and HTTPS capabilities meeting production requirements.

3.3 Additional ACK Ray Platform Capabilities

Beyond History Server, ACK provides a production-grade Ray platform around KubeRay: kuberay-operator component: Managed installation and upgrades, supporting full lifecycle management of RayCluster, RayJob, and RayService.

Native Alibaba Cloud ecosystem integration: OSS storage, RRSA permissions, RAM roles, ALB/Nginx Ingress entry points.

AI workload-oriented cluster capabilities: GPU resource scheduling, node pool management, and elastic scaling.

Teams already running Ray workloads on ACK can enable HistoryServer on demand, verify via port-forward, then combine with ALB or Nginx Ingress to provide a stable entry point for platform users, ensuring every Ray job has a traceable history.