Design and Implementation of a Multi‑Cluster Arthas‑Based Online Diagnosis Platform

Problem Statement

When Java applications are scattered across many Kubernetes clusters, traditional troubleshooting via SSH or kubectl exec becomes impractical due to network isolation, lack of auditability, and low efficiency.

Solution Overview

We built a unified Arthas online diagnosis platform that provides a web‑based interactive terminal without SSH, eliminates the need for kubectl, and records all operations for full audit.

Architecture

The system follows a classic control‑plane (Master) + data‑plane (Agent) model:

Master (management backend) handles cluster registration, HMAC‑signed HTTP communication, WebSocket tunneling, three‑level RBAC, heartbeat monitoring, and audit logging.

Agent (deployed in each business cluster) exposes HTTP APIs for resource queries, performs Arthas download and JVM attach, establishes a bidirectional WebSocket stream, and runs with a read‑only ServiceAccount token.

Technology Stack and Code Structure

Backend is built with Go + Gin (gin‑vue‑admin scaffold); frontend uses Vue 3 + Element Plus + xterm.js . Core directory layout:

server/
├── model/diagPlatform/         # Data models
│   ├── diag_cluster.go        # Cluster config model
│   └── diag_permission_rule.go # 3‑level RBAC model
├── service/diagPlatform/      # Business logic
│   └── diag_cluster.go       # CRUD, credential generation
├── service/agent/             # Agent communication
│   └── http_client.go        # HMAC request signing
├── core/websocket/            # WebSocket engine
│   ├── handler.go             # Handshake & message routing
│   ├── pool.go                # Connection pool (sync.Map)
│   ├── client.go              # Master→Agent client
│   ├── redis_routing.go       # Redis routing table
│   ├── auth/handshake.go      # Challenge‑response protocol
│   └── streaming/streaming.go # Terminal session manager
├── model/websocket/message.go # JSON frame definition
├── api/v1/diagPlatform/       # API controllers
│   ├── diag_cluster.go        # Cluster management API
│   ├── websocket.go          # Diagnosis operations API
│   └── diag_permission_rule.go # Permission API
├── router/diagPlatform/        # Route registration
└── task/cluster_heartbeat_monitor.go # Heartbeat task

Three‑Layer Security

1. Pre‑registered Credential Mechanism

Clusters are created via the UI; the system generates a unique ClusterId (Snowflake) and a 32‑byte random secret key. The secret is stored as a bcrypt hash ( SecretHash) for Agent→Master verification and as AES‑encrypted plaintext ( SecretKeyEncrypted) for HMAC signing. The plaintext secret is returned only once at creation.

func (dcService *DiagClusterService) CreateDiagCluster(dc *diagPlatform.DiagCluster) (plaintextSecretKey string, err error) {
    // 1. Generate unique ClusterId
    dc.ClusterId = utils.GenerateClusterID()
    // 2. Generate 32‑byte random secret
    plaintextSecretKey = utils.GenerateSecretKey()
    // 3. Store bcrypt hash
    dc.SecretHash = utils.BcryptHash(plaintextSecretKey)
    // 4. AES‑encrypt plaintext for HMAC
    encryptedSecretKey, err := utils.Encrypt([]byte(plaintextSecretKey))
    dc.SecretKeyEncrypted = encryptedSecretKey
    // 5. Initial offline status
    onlineStatus := 0
    dc.OnlineStatus = &onlineStatus
    err = global.GVA_DB.Create(dc).Error
    return plaintextSecretKey, err // plaintext returned only once
}

2. Challenge‑Response Handshake

When an Agent connects, the Master sends a 16‑byte random challenge (valid for 30 s). The Agent replies with an HMAC‑SHA256 of the challenge using the secret key. The Master verifies the response with bcrypt, preventing replay and MITM attacks.

// Master sends challenge
Master ──► Agent : Challenge (16 bytes)
// Agent replies
Agent ──► Master : HMAC‑SHA256(SecretKey, Challenge)
// Master validates with bcrypt and establishes session

3. HMAC Request Signing

All HTTP requests from Master to Agent include a timestamp, method, path, and SHA‑256 body hash, signed with HMAC‑SHA256 using the secret key. The signature is sent via X‑Timestamp and X‑Signature headers. The /health endpoint is exempted to keep heartbeat checks lightweight.

func SignRequest(method, path string, body []byte, secretKey string) (timestamp, signature string) {
    timestamp = fmt.Sprintf("%d", time.Now().Unix())
    bodyHash := sha256.Sum256(body)
    signingString := fmt.Sprintf("%s
%s
%s
%s", timestamp, method, path, hex.EncodeToString(bodyHash[:]))
    mac := hmac.New(sha256.New, []byte(secretKey))
    mac.Write([]byte(signingString))
    signature = hex.EncodeToString(mac.Sum(nil))
    return timestamp, signature
}

WebSocket Tunnel Engine

Communication uses a unified JSON frame with four message types: REQ, RSP, STREAM, and HEARTBEAT. The protocol covers all interactions, from authentication to terminal I/O.

const (
    MsgTypeREQ       = "REQ"       // request, await response
    MsgTypeRSP       = "RSP"       // response, matches msg_id
    MsgTypeSTREAM    = "STREAM"    // streaming data (terminal I/O)
    MsgTypeHEARTBEAT = "HEARTBEAT" // keep‑alive
)

type Message struct {
    Header MessageHeader `json:"header"`
    Body   MessageBody   `json:"body"`
}

The connection pool uses sync.Map for lock‑free concurrent reads/writes, maintaining a bidirectional index of ClusterID ↔ SessionID. Lifecycle management includes registration, heartbeat updates, stale‑connection cleanup (every 30 s), and graceful removal on disconnect.

type ConnectionPool struct {
    connections sync.Map // clusterID → *AgentConnection
    sessionMap  sync.Map // sessionID → *AgentConnection
}

func (cp *ConnectionPool) AddConnection(clusterID string, conn *AgentConnection) {
    cp.connections.Store(clusterID, conn)
    cp.sessionMap.Store(conn.SessionID, conn)
}

Three‑Tier Data Path

The terminal data flow is:

Browser xterm.js  ◄──────►  Master StreamingManager  ◄──────►  Agent  ◄──────►  Arthas process
 (WebSocket)            (session bridge)                (k8s exec)

Before establishing a terminal, the front‑end fetches connection parameters via a REST API, which returns the WebSocket URL (with HMAC signature), Arthas download URL, and a derived password.

func (wsApi *WebSocketApi) GetAgentConnectInfo(c *gin.Context) {
    clusterID := c.Param("clusterId")
    agentAddr, _ := agentSvc.GlobalAgentClient.GetAgentAddress(clusterID)
    arthasUrl, _ := agentSvc.GlobalAgentClient.GetArthasUrl(clusterID)
    secretKey, _ := agentSvc.GlobalAgentClient.GetSecretKey(clusterID)
    wsURL := agentSvc.DeriveWebSocketURL(agentAddr, secretKey)
    response.OkWithData(gin.H{"websocketUrl": wsURL, "arthasTarDownloadUrl": arthasUrl, "arthasPassword": arthasPassword}, c)
}

The StreamingManager creates a TerminalSession and launches two goroutines to forward data between the browser and the Agent, handling both directions and automatically closing the session when either side disconnects.

type TerminalSession struct {
    SessionID    string
    ClusterID    string
    FrontendConn *gorillaWs.Conn // browser WebSocket
    AgentConn    *wsCore.AgentClientConnection // Agent WebSocket
    Active       bool
}

func (sm *StreamingManager) ActivateSession(sessionID string, frontendConn *gorillaWs.Conn) error {
    session.FrontendConn = frontendConn
    session.Active = true
    go sm.handleFrontendStream(session) // browser → Agent
    go sm.handleAgentStream(session)    // Agent → browser
    return nil
}

RBAC Permission Model

Permissions are stored as a JSON tree: cluster → namespace → []deployments. The API layer filters resources at every level (Namespace, Deployment, Pod, Cluster) based on the user’s authorized tree, ensuring that non‑superusers only see resources they can access.

type ClusterAuthorizations map[string]NamespaceDeployments
type NamespaceDeployments map[string][]string

type DiagPermissionRule struct {
    UserId               uint   `gorm:"uniqueIndex"`
    ClusterAuthorizations ClusterAuthorizations `gorm:"type:json"`
}

Cluster Lifecycle Management

Deployment Script Generation

When an admin creates a cluster, the system generates a full K8s deployment YAML. The secret key is decrypted and injected as AUTH_SECRET_KEY. The YAML includes a Deployment with anti‑affinity, health probes, a Service + Ingress for Agent exposure, and minimal RBAC (ServiceAccount, ClusterRole, ClusterRoleBinding).

Heartbeat Monitoring

A scheduled task iterates over all clusters, calls the Agent’s /health endpoint, and updates the online_status and last_heartbeat fields. Stale clusters (no heartbeat for 60 s) are marked offline.

func CheckClusterHeartbeat() {
    var clusters []diagPlatform.DiagCluster
    global.GVA_DB.Find(&clusters)
    for _, cluster := range clusters {
        if cluster.AgentAddress == "" {
            diagClusterService.UpdateClusterOnlineStatus(cluster.ClusterId, 0)
            continue
        }
        err := agentSvc.GlobalAgentClient.HealthCheck(cluster.ClusterId)
        if err != nil {
            diagClusterService.UpdateClusterOnlineStatus(cluster.ClusterId, 0)
        } else {
            diagClusterService.UpdateClusterOnlineStatus(cluster.ClusterId, 1)
            global.GVA_DB.Model(&diagPlatform.DiagCluster{}).Where("cluster_id = ?", cluster.ClusterId).Update("last_heartbeat", time.Now())
        }
        if cluster.LastHeartbeat != nil && time.Since(*cluster.LastHeartbeat) > 60*time.Second {
            diagClusterService.UpdateClusterOnlineStatus(cluster.ClusterId, 0)
        }
    }
}

Front‑End Terminal Integration

The browser uses Vue 3 and xterm.js. Users select Cluster → Namespace → Deployment → Pod → Container; non‑Running pods are disabled. After selection, the front‑end opens a WebSocket connection, sends a connect payload containing the target pod info and Arthas download URL, and streams user keystrokes (base64‑encoded) to the Agent. The Agent forwards the data to Arthas, and the resulting output is rendered back in the xterm terminal.

// Initialize xterm
const terminal = new Terminal({
    theme: { background: '#1a1a2e', foreground: '#e0e0e0', cursor: '#ffd700' }
})
// Send user input to WebSocket
terminal.onData(data => {
    if (ws.readyState === WebSocket.OPEN) {
        ws.send(JSON.stringify({ type: 'stdin', data: { content: btoa(unescape(encodeURIComponent(data))) } }))
    }
})
// Open WebSocket and send connection request
ws = new WebSocket(connectInfo.websocketUrl)
ws.onopen = () => {
    ws.send(JSON.stringify({
        type: 'connect',
        data: {
            namespace: selectorForm.value.namespace,
            pod: selectorForm.value.podName,
            container: selectorForm.value.containerName,
            arthasTarDownloadUrl: connectInfo.value.arthasTarDownloadUrl,
            cols: terminal.cols,
            rows: terminal.rows
        }
    }))
}

Conclusion

The platform demonstrates that by combining a WebSocket tunnel, HMAC‑based request signing, and fine‑grained RBAC, a cloud‑native Java diagnosis tool can safely operate across isolated Kubernetes clusters, eliminate manual SSH steps, provide full auditability, and deliver a seamless web‑based Arthas experience.