Common Cluster Issues and Practical Solutions for Apps, DBs, Caches, MQ, Files, and Search

Application Cluster Issues

Deploying the same business system on multiple servers creates a cluster that must handle larger traffic and avoid downtime when a node fails.

Session loss on login : Tomcat stores sessions locally; a subsequent request may be routed to a different node without the session. Solution : Use JWT (stateless) or store sessions centrally in Redis; avoid Nginx ip_hash for long‑term use.

Repeated scheduled tasks : Each node runs its own scheduler, causing duplicate executions. Solution : Adopt a distributed scheduler like XXL‑Job, use Redis distributed locks, or temporarily run the task on a single node (with single‑point risk).

File upload 404 errors : Files saved on a node’s local disk are unavailable when the next request hits another node. Solution : Store files in shared object storage (MinIO, OSS) or use NFS in intranet environments.

Uneven load balancing : Default round‑robin ignores server capacity, leading to some CPUs at 90% while others idle. Solution : Enable weighted round‑robin in Nginx, configure health checks, and add backend rate limiting.

502 errors after a node crash : Load balancer only checks port connectivity, so a failed node continues receiving traffic. Solution : Enable active health checks that remove unhealthy nodes after repeated failures.

Service “hang” without crashes : Threads deadlock or connection pools exhaust, causing timeouts while the process stays alive. Solution : Expose a /health endpoint for business health checks and monitor thread‑pool and DB‑connection metrics.

Clock drift across nodes : Unsynchronized server clocks cause log inconsistencies, token expiration, and lock failures. Solution : Deploy NTP on all nodes and keep time deviation under one second.

Rolling upgrades causing interface errors : New and old versions run simultaneously, leading to parameter mismatches. Solution : Ensure backward‑compatible APIs (add parameters only), use gray releases, and provide transition periods for breaking changes.

Duplicate requests due to retries : Network glitches trigger automatic retries, causing duplicate orders or charges. Solution : Implement idempotent APIs with unique request IDs stored in Redis or the database.

Scattered logs : Each node writes logs locally, making fault diagnosis painful. Solution : Deploy a centralized log platform (ELK, Loki) for unified collection and search.

Configuration drift : Manual edits on each server lead to inconsistent settings. Solution : Use a configuration center (Nacos, Apollo) for global config management.

Traffic spikes causing cascade failures : No rate limiting or circuit breaking lets a single overloaded node bring down the whole cluster. Solution : Apply gateway rate limiting, integrate Sentinel for circuit breaking, and set sensible thread‑pool limits.

Environment inconsistency across nodes : Different JDK versions, OS parameters, or dependencies cause errors on some machines. Solution : Package applications in Docker containers or standardized VM images.

Service discovery difficulty : Hard‑coded IPs require code changes on scaling. Solution : Introduce a service registry (Nacos, Eureka) for automatic discovery and load balancing.

Local cache causing data inconsistency : In‑memory caches (Caffeine, HashMap) are not shared, leading to stale data across nodes. Solution : Prefer Redis for shared caching; if local cache is necessary, implement proactive refresh mechanisms.

Connection‑pool exhaustion : Small pool sizes cause random timeouts under load. Solution : Increase max connections, monitor pool usage, and set appropriate timeouts.

Duplicate business IDs : Using timestamps or random numbers on multiple nodes can generate colliding IDs. Solution : Use Snowflake algorithm or Redis auto‑increment for globally unique IDs.

New nodes not taking traffic after scaling : Hot caches or DB bottlenecks limit throughput. Solution : Check downstream resources, pre‑warm new nodes, and avoid sticky routing.

Message‑queue consumer duplication : Multiple consumers in a group process the same message, causing repeated business actions. Solution : Use a single consumer per partition or ensure idempotent processing.

Insufficient node‑level monitoring : Only aggregate metrics are observed, delaying detection of memory leaks or GC spikes. Solution : Deploy Prometheus + Grafana to collect per‑node CPU, memory, GC, thread‑pool, and latency metrics with alerts.

Database Cluster Issues

Master‑slave lag : Writes to the master are not immediately visible on the slave, causing stale reads. Solution : Route real‑time queries to the master, limit large transactions, and monitor lag thresholds.

Master failure without automatic failover : Manual promotion of a slave is slow. Solution : Deploy HA components (MGR, Orchestrator, Keepalived) and use virtual IPs for transparent failover.

Stale connection strings after failover : Applications still point to the old master IP. Solution : Connect via a virtual VIP or use middleware (MyCat, Sharding‑JDBC) that abstracts node addresses.

Replication breakage and data inconsistency : Errors like Slave_IO_Running=No cause long‑term divergence. Solution : Re‑build the slave from a fresh backup, enforce schema consistency, and set up immediate alerting on replication status.

Read‑write imbalance : Queries all hit the master despite read‑only workloads. Solution : Enforce routing rules that send non‑real‑time reads to slaves and use SQL routing tools.

Large batch updates overwhelming slaves : Massive binlog traffic stalls replication. Solution : Split large batches into smaller chunks, enable parallel replication, and schedule bulk updates during off‑peak periods.

Cross‑shard joins impossible : Data resides in separate shards. Solution : Redesign tables to avoid cross‑shard joins, perform in‑memory joins in the application, or use middleware like ShardingSphere for limited cross‑shard queries.

Hot‑spot shards : Skewed shard key (e.g., user ID) concentrates load. Solution : Choose a more uniform shard key, isolate hot data, or add more shards and migrate data.

Distributed transaction failures : Partial commits across shards leave data inconsistent. Solution : Avoid cross‑shard transactions; if needed, use flexible transaction frameworks (ShardingSphere, Seata) or reliable messaging for eventual consistency.

Charset or collation mismatch : Inconsistent settings cause query errors and index loss. Solution : Standardize my.cnf across all nodes and enforce uniform parameters.

Reporting workload hurting replication : Heavy reporting on a slave delays binlog replay. Solution : Deploy dedicated reporting slaves and separate real‑time queries.

Disk exhaustion on master : Full disks stop binlog writes and crash the cluster. Solution : Monitor disk usage, set cleanup policies, and keep free space buffers.

Backup on master causing lock contention : Full logical dumps lock tables. Solution : Perform backups on slaves or use physical backup tools like XtraBackup.

MGR network sensitivity : Packet loss triggers frequent partitioning. Solution : Keep MGR within a stable LAN, increase timeout settings, and monitor inter‑node latency.

Shard expansion difficulty : Adding shards requires massive data migration. Solution : Pre‑allocate shards, use middleware supporting online migration, and plan capacity early.

Read‑write routing errors after enabling read‑write separation : Mis‑annotated queries still hit the master. Solution : Enforce code reviews, enable SQL audit, and regularly audit routing rules.

Master crash with duplicate binlog replay : Re‑playing binlogs causes primary‑key conflicts. Solution : Prevent unexpected restarts, ensure idempotent SQL, and have DBA procedures for conflict resolution.

Node‑level performance variance : Different hardware or MySQL configs cause uneven query times. Solution : Standardize hardware specs and core MySQL parameters across nodes.

Unable to locate slow queries on specific nodes : Lack of per‑node slow‑log aggregation. Solution : Deploy a unified SQL audit and slow‑log collection platform with node identifiers.

Cache Cluster Issues

Cache avalanche : Simultaneous key expiration floods the database. Solution : Add random jitter to TTLs, keep hot keys permanent with background refresh, and apply rate limiting at the gateway.

Cache breakdown : A single hot key expires, causing a thundering herd. Solution : Use a distributed lock to allow only one request to rebuild the cache, keep hot data permanent, and pre‑warm caches.

Cache penetration : Requests for non‑existent keys bypass the cache and hit the DB. Solution : Cache null values with short TTLs, employ Bloom filters, and validate parameters at the front end.

Cache‑DB inconsistency after updates : Stale cache shows old data. Solution : Update DB first, then delete the cache; add retry mechanisms for cache eviction failures; use distributed locks for high‑concurrency writes.

Redis‑master/slave lag : Reads from a slave may return stale data. Solution : Force strong‑consistency reads from the master for critical paths, monitor replication delay, and fall back to the master when lag exceeds thresholds.

Hash slot imbalance : Keys concentrate on a few slots, overloading some nodes. Solution : Add random suffixes to keys, split large keys, and manually rebalance slots.

Big key blocking : Extremely large keys block the single Redis thread. Solution : Split big keys into smaller ones and paginate large data reads.

Network instability causing node isolation : Packet loss marks nodes offline. Solution : Deploy clusters within the same data center, adjust failure‑detection timeouts, and optionally use sentinel or local secondary caches as fallback.

Sentinel failover delay : Switching master takes seconds. Solution : Add graceful degradation, use local caches during failover, and tune sentinel detection intervals.

Scaling adds nodes but no traffic redistribution : New nodes stay idle. Solution : Perform manual slot migration during low‑traffic windows and limit migration speed.

Connection‑pool exhaustion : Applications create too many Redis connections. Solution : Configure a connection pool, reuse connections, and monitor connection counts.

Memory pressure and eviction : Full memory triggers LRU eviction, losing business data. Solution : Set appropriate maxmemory, choose an eviction policy (e.g., allkeys‑lru), and enforce expiration times.

Persistence‑induced latency spikes : RDB snapshots or AOF flushing pause the server. Solution : Schedule snapshots during off‑peak hours, use asynchronous AOF, or run persistence on replica nodes.

Distributed lock timeout : Locks expire before business finishes, causing duplicate work. Solution : Use Redisson’s watchdog for automatic renewal, estimate max execution time, and split long tasks.

Message‑Queue Cluster Issues

Duplicate consumption : Messages processed multiple times lead to duplicate orders or notifications. Solution : Ensure idempotent processing with unique msgId, set appropriate consumer timeouts, and keep business logic short.

Message loss : Failures at producer, broker, or consumer stages cause missing data. Solution : Enable producer ACKs, persist messages to disk, acknowledge only after business success, and configure multiple replicas.

Out‑of‑order delivery : Parallel consumption scrambles sequence. Solution : Route ordered messages to the same partition (e.g., hash by order ID) and run a single consumer per partition.

Message backlog : Production outpaces consumption, filling disk and slowing the cluster. Solution : Scale consumer instances, optimise consumer logic, set alert thresholds, and define message TTLs.

Dead‑letter explosion : Unlimited retries flood DLQ. Solution : Limit retry attempts, move persistent failures to a DLQ, and manually investigate.

Network jitter causing leader election delays : Short outages trigger temporary unavailability. Solution : Add producer retries with back‑off, colocate brokers in the same LAN, and implement local fallback storage.

Hot partitions : Uneven topic partition distribution overloads a single broker. Solution : Increase partition count, split hot topics, and monitor per‑broker throughput.

Consumer imbalance : Fewer partitions than consumers leaves many idle. Solution : Ensure consumer count ≤ partition count or increase partitions.

Large delayed messages : Massive delayed‑message scans consume CPU. Solution : Reduce the number of short‑delay messages and offload large delayed workloads to a dedicated scheduler like XXL‑Job.

Transactional message hanging : Half‑messages remain unresolved after producer crash. Solution : Provide a transaction status query interface, set a max retry count, and prefer eventual consistency where possible.

Oversized messages : Multi‑MB payloads strain network and disk. Solution : Send only business identifiers in MQ, store large payloads in object storage, and enforce message size limits.

Slow replica sync : Multiple replicas increase write latency. Solution : Choose sync strategy based on reliability needs, limit replica count, and tune sync parameters.

Version mismatch during rolling upgrades : New and old consumers cannot parse changed message schemas. Solution : Keep schemas backward compatible (add fields only), use gray releases, and add robust parsing with fallback handling.

Disk growth from retained messages : No retention policy leads to storage exhaustion. Solution : Configure message TTL (e.g., 72 hours) and clean up automatically.

Producer connection churn : Frequent connect/disconnect hits max‑connection limits. Solution : Use connection pools, monitor connection counts, and raise max‑connection settings as needed.

Cross‑region latency : Inter‑city replication slows down. Solution : Deploy core MQ clusters within the same city, use dual‑cluster designs for cross‑region traffic, and apply compensation logic for delayed sync.

Message discard without trace : Acknowledging on exception loses the message. Solution : Do not ack on failure; route to retry or DLQ and log full message details for debugging.

Expansion without automatic rebalancing : New brokers stay idle. Solution : Manually trigger partition migration during low‑traffic windows and plan partition count ahead of scaling.

File‑Server Cluster Issues

Random 404 after upload : Files saved locally disappear when another node serves the request. Solution : Disable local storage, use distributed object storage (MinIO, Ceph) or NFS for shared access.

Concurrent write conflicts : Simultaneous edits corrupt files. Solution : Apply "delete‑then‑upload" strategy, use distributed locks, and version files.

Single‑disk failure loss : No replication leads to total data loss. Solution : Enable multi‑replica storage (2‑3 copies) across disks.

Uneven disk utilization : Hot files concentrate on a few disks. Solution : Adjust sharding/hash routing, run periodic re‑balancing.

Massive small‑file overhead : Metadata pressure slows the system. Solution : Aggregate small files, optimise object‑store metadata indexing, and tier cold files.

Bandwidth contention during peaks : Upload/download slows down. Solution : Use link aggregation, apply per‑service bandwidth limits, and schedule large transfers off‑peak.

NFS hard‑mount hangs on network glitches : Applications block indefinitely. Solution : Use soft mounts with timeout, migrate to object storage, and monitor mount health.

Expansion without data migration : New nodes stay empty. Solution : Manually trigger data re‑balancing during low‑traffic periods and plan capacity in advance.

Special‑character filenames : Upload succeeds but download fails. Solution : Rename files to UUIDs on storage, keep original names in DB, and ensure UTF‑8 encoding with proper URL encoding.

Duplicate file storage : Same file uploaded repeatedly consumes space. Solution : Implement hash‑based deduplication; store only references when a file already exists.

Cross‑region latency : Users far from storage experience timeouts. Solution : Deploy multi‑region clusters with CDN, route users to the nearest node.

Insecure direct links : Anyone can guess URLs to download confidential files. Solution : Disable public URLs, use signed temporary URLs, and enforce authentication on download endpoints.

Garbage file accumulation : Temporary files never cleaned up. Solution : Define lifecycle policies (e.g., delete after 30 days) and separate temporary from permanent buckets.

Metadata server single‑point failure : Cluster becomes unavailable when metadata node crashes. Solution : Deploy a metadata cluster or use storage solutions (MinIO) without a separate metadata node.

Interrupted large‑file uploads : No resume capability forces full re‑upload. Solution : Implement chunked upload with resume support.

Backup tasks degrading performance : Full backups saturate I/O. Solution : Schedule backups off‑peak, throttle read speed, and rely on redundancy instead of frequent full backups.

Clock drift affecting signed URLs and lifecycle : Expired signatures and wrong deletions. Solution : Synchronise all nodes via NTP.

Node reboot causing file corruption : In‑flight writes lost. Solution : Use synchronous flush for critical data, confirm upload completion before responding, and verify file integrity (MD5).

Malicious file uploads : Executables or viruses spread through the cluster. Solution : Restrict allowed file types, integrate virus scanning, and isolate executable downloads.

Missing monitoring leads to late detection : Disk full or node offline discovered too late. Solution : Build a monitoring system for disk usage, node health, I/O, and network, with alerting via messaging platforms.

Search Cluster Issues

Red cluster status (missing primary shards) : Data unavailable. Solution : Recover failed nodes, ensure each index has at least one replica, and avoid taking nodes offline without migration.

Yellow status (unassigned replicas) : No redundancy, risk of data loss. Solution : Check disk usage, ensure enough nodes for replica allocation, and review shard allocation filters.

Write performance degradation under heavy ingest : Bulk indexing slows down. Solution : Increase refresh_interval during bulk loads, use Bulk API, and size shards appropriately.

Deep pagination slowness : Large from values cause timeouts. Solution : Disable deep pagination; use scroll or search_after and limit maximum page numbers.

Hot shards causing CPU spikes : One node bears most load. Solution : Plan shard count to distribute data evenly and rebalance hot shards manually.

Disk space exhaustion : Nodes become read‑only. Solution : Set disk watermarks, implement index lifecycle management (ILM) to delete old indices, and use hot‑cold node architecture.

Expensive aggregations : CPU overload on group‑by or histogram queries. Solution : Limit query time ranges, pre‑aggregate data into Redis, and map aggregation fields as keyword.

Excessive shard count : Management overhead and GC pressure. Solution : Keep shard size 10‑50 GB, merge small indices, and enforce template limits.

Frequent Full GC causing timeouts : JVM memory mis‑configuration. Solution : Limit heap to ≤31 GB, disable OS swap, and optimise queries to avoid loading massive result sets.

Incorrect analyzer configuration : Poor search relevance. Solution : Choose appropriate analyzers (e.g., IK for Chinese), separate text and keyword fields, and rebuild indices when changing analyzers.

Cluster expansion without auto‑balancing : New nodes stay idle. Solution : Manually trigger shard rebalancing during low‑traffic windows and monitor distribution.

Master node overload : Management operations become slow. Solution : Separate master‑eligible nodes from data nodes, deploy at least three dedicated masters, and avoid heavy query load on masters.