Lakehouse Capacity Planning: Stress Testing, Bottleneck Detection & Scaling Decisions

1. Problem Background and Technical Evolution

Many teams approach capacity planning backwards: they buy machines first, then rely on "feel" and alerts to add capacity after launch. The previous article (Article 13) demonstrated that this approach carries high risk during major promotions because you don't know each layer's true processing ceiling or where the bottleneck actually lies. Stress testing and capacity planning aim to replace intuition with data: before launch and before major promotions, gradually push the full chain to target load and inflection points, find when degradation begins, and calculate how much headroom production needs.

Lakehouse stress testing is harder than single-component testing because it's a chain: Kafka, Flink, Paimon, StarRocks — any segment can become the bottleneck. High Kafka throughput in isolation doesn't mean end-to-end can run that fast. The core of this article is not "how to push a single component's numbers higher" but how to test the real processing capacity of the entire chain, locate the first link that chokes, and convert stress test results into machine counts.

2. What You Will Gain

Distinguish write stress testing, query stress testing, and mixed stress testing — what each tests and why end-to-end chain testing is mandatory over single-component testing.

Understand the role and limitations of benchmarks like TPC-DS, and how to adapt them to your actual business workload.

Master identification methods for four bottleneck categories — CPU, memory, disk I/O, network — so you can judge from monitoring which resource layer is saturated.

Establish a capacity planning derivation framework (peak demand ÷ measured single-node capacity × safety factor) instead of guessing.

Understand the decision basis for vertical scaling vs. horizontal scaling, and when to add resources versus when to optimize code.

Be able to answer open-ended interview questions like "how to do capacity assessment before a new business launches."

3. Core Concepts and Architecture Positioning

The first principle of stress testing: any system's effective processing capacity is limited by the resource, component, or coordination mechanism that becomes the bottleneck first. Adding disk when CPU is saturated usually doesn't help; adding memory when network is the bottleneck may not help either. Therefore stress testing isn't about watching a single throughput number — it's about monitoring CPU, memory, disk I/O, network, and application-level metrics simultaneously while increasing load, to find the inflection point where "throughput growth visibly slows or stops, and a certain resource or component hits saturation first" — that is typically the primary bottleneck under current load.

Note: 100% resource utilization is not the sole criterion for identifying a bottleneck. For example, at 70% CPU the system may already be limited by single-core hotspots, GC, lock contention, or memory bandwidth. Low disk utilization doesn't mean I/O is fine — await, queue depth, throughput, and IOPS must be observed together. True bottleneck judgment follows this chain: business load increases → throughput growth slows/stops → a resource or component metric degrades synchronously.

Capacity planning is a separate matter: it translates "how much business must handle" into "how many machines needed." Business provides peak write volume, query concurrency, data retention, SLA; stress testing provides "under target SLA, target data distribution, and target concurrency, how much can a single node or cluster stably handle." Combining both, plus resource headroom, failure redundancy, and growth space, yields production scale. This derivation chain is indispensable — having only business volume without measured capacity, or only single-node scores without business volume, cannot produce reliable capacity numbers.

4. Deep Dive into Underlying Principles

How to Use Benchmarks in Lakehouse

TPC-DS is TPC's decision support benchmark, containing a retail business model, data generation rules, and 99 query templates for testing analytical systems under complex decision-support loads. Its value: a repeatable, standardized test framework. Its limitation: its schema, data distribution, and query templates almost never match your real business exactly.

Correct usage: first use TPC-DS for engine capability verification and horizontal comparison (A vs B under same data scale, hardware, config, test method), then replace or supplement with your real SQL templates, real data volumes, and real data distributions to get numbers meaningful for production. Directly using TPC-DS scores to set production capacity is a common pitfall.

What Write Stress Testing Tests

The write chain's goal: find the maximum sustained write rate where "end-to-end has no unacceptable data loss/duplication, backlog can converge, latency meets SLA." Ramp up gradually from Kafka/Fluss write: watch if Flink consumption keeps up, if Paimon commit and file generation are stable, if compaction can continuously converge. The key is not peak burst height but whether at target rate sustained for long enough (e.g., 1–2 hours) lag converges, checkpoints succeed stably, compaction backlog does not keep growing — a burst that leaves long-term unconverged backlog does not count as passing.

"1–2 hours" is just an example, not a universal standard. If business has hourly/daily compaction cycles or obvious cache warm-up, stress test duration must cover those key states.

What Query Stress Testing Tests

Query chain tests latency and stability under concurrency: not how fast a single query runs, but "N users clicking simultaneously, is P95/P99 latency still within SLA, are resources continuously stable." Must simulate real concurrency distribution (peak high, valley low), and include slow queries and large queries per the resource group isolation strategy from Article 12.

For lake table queries, must also simulate real partition counts, file counts, object storage access, cache hit rates — because the same SQL can produce completely different performance under different file scales and cache states.

How to Identify Four Bottleneck Types

CPU continuously near saturation while throughput no longer grows linearly with load — typically compute bottleneck; further investigate whether operator compute, single-thread hotspot, GC, lock contention, or parallelism is the cause.

Memory pressure continuously rising, OOM or frequent GC — possibly due to state, intermediate results, cache, or concurrency.

Disk I/O shows high await, queue buildup, or throughput/IOPS approaching device limits — common in Paimon compaction, RocksDB, or lake table scans.

Network throughput approaching NIC/link capacity with queuing, retransmission, or shuffle latency increase — common in shuffle, data sync, external table scans.

Identify the bottleneck type correctly, and scaling direction will be right; identify it wrong, and adding more machines is waste.

5. Core Process and Key Workflows

A complete capacity assessment process is a closed loop.

Step 1: Clarify business inputs — peak write volume (records/sec, bytes/sec), daily growth, data retention, query concurrency and latency SLA, promotion multiplier.

Step 2: Layered single-component stress testing — in standard test env, separately test Kafka single-partition/single-node throughput, Flink processing capacity under target operators and state scale, Paimon write and compaction capacity, StarRocks query capacity under target SQL and concurrency, to get each layer's "measured capability baseline."

Step 3: End-to-end chain stress testing — connect them and ramp up, find the link that degrades first in the full chain — usually lower than theoretical capability from isolated single-component tests due to network transfer, shuffle, state management, file system, compaction, query concurrency overhead.

Step 4: Calculate safety margin based on inflection-point resource utilization and SLA, and reserve failure redundancy and future growth.

Illustrative derivation (numbers for method demo only, not commitments): business peak needs X records/sec; stress test finds single Flink node can stably handle Y records/sec under target checkpoint stability, lag convergence, SLA compliance; theoretical node count ≈ X ÷ Y. But this is only the theoretical floor.

Actual production node count must also consider: resource safety margin, traffic peak fluctuation, failure redundancy, scale-up/scale-down and recovery windows. Therefore cannot simply be "theoretical node count × a fixed factor." For example, if any single node failure must still meet peak SLA, recalculate per N+1 or appropriate failure model, not treat "failure redundancy" as a simple percentage.

6. Real Business Scenarios and Engineering Practice

Using e-commerce promotion scaling as example, here's how it lands in engineering.

First, data preparation: build stress test dataset from production desensitized data, preserving real primary key distribution, hot keys, payload size distribution, time distribution — because Article 13 showed data skew makes single partitions bottlenecks, uniform stress data masks real problems.

Second, ramp-up scripts: use step-wise ramp, hold each level for a period, while collecting four resource metrics and end-to-end latency.

Key observation points are not "how fast it runs" but "which metric starts visibly degrading, where does the throughput inflection point appear."

Example: ramp up to a level where a few Flink TaskManagers approach CPU saturation, processing latency grows, but Kafka still has clear headroom — compute layer may be bottleneck, scaling should prioritize adding Flink effective compute resources or parallelism, not more Kafka capacity. But if only a few TaskManagers show high CPU, must further check data skew, key hotspots, task assignment — cannot directly conclude "entire Flink cluster lacks CPU."

Pre-promotion capacity conclusion must land in a table: per layer, how many nodes needed under peak + promotion multiplier, how many currently have, gap, how much reserved. This table is also the source of Article 13's "three water levels" — healthy water level isn't simply "half of inflection-point resource utilization" but should be set based on the "safe operating range" derived from stress testing; warning water level must leave operation window from alert to scale/downgrade; emergency water level corresponds to region near SLA or resource bottleneck.

Stress testing should also incidentally verify degradation plans: turn off non-core streams, reduce query concurrency, or switch to low-cost compute paths, see if core chain can withstand promotion peak.

7. Performance, Cost, and Technology Selection

Capacity planning is fundamentally a trade-off among SLA, cost, and headroom. Too much headroom = idle machines wasting money; too little = higher promotion risk. Engineering typically provisions for peak not average, combined with failure models, growth expectations, and recovery windows to leave safety space.

Cost-wise, distinguish two scaling types:

Vertical scaling (add CPU/memory/faster disks per machine) — lower cost, faster effect, but has physical limits; suitable when single-node resources are insufficient and business has vertical scaling room.

Horizontal scaling (add nodes) — suitable when system can effectively utilize more parallel resources.

Decision basis still returns to bottleneck type — CPU/compute bottlenecks usually prioritize adding compute resources or parallelism; disk I/O bottlenecks may need faster disks, more disk bandwidth, or resource isolation; network bottlenecks require checking NIC, switch fabric, data exchange patterns.

But also note: not all bottlenecks can be solved by scaling. Examples:

Single-thread operator becomes bottleneck — adding nodes may be ineffective.

Hot key causes single-partition bottleneck — adding nodes may be ineffective.

Unreasonable SQL execution plan — adding nodes may only delay the problem.

Suboptimal small-file/compaction strategy — simply adding machines may further increase management overhead.

Must also recognize the cost trap of stress testing: occupying large clusters long-term for testing is expensive, so common approach is "small cluster establishes capability baseline → medium scale validates scaling trend → near-production scale does final validation." But note: not all capabilities grow linearly with node count — shuffle, metadata management, network topology, compaction coordination, object storage access may introduce extra overhead; larger scale means actual effective capacity increasingly needs confirmation via real measurement. This is why at least one near-real-scale chain stress test is mandatory, not pure linear extrapolation from small clusters.

8. Production Issues and Troubleshooting Methods

Issue 1: Stress test results don't match production feel

Phenomenon: stress test fast, production slow. Root cause usually unrealistic test data or load model — e.g., data too uniform, payloads too small, no hotspots, query concurrency model too simple, or production has cache cold starts, object storage latency, business retries not covered in test. Diagnosis: compare test vs production data distribution, query templates, concurrency model, cache state, data scale. Fix: re-test with production-sampled data and real concurrency templates. Prevention: regularly update stress test sets with production samples.

Issue 2: Throughput drops when load increases

Phenomenon: continue increasing pressure, system throughput actually falls. Principle: some resource or internal queue hits saturation and enters inefficient state — e.g., memory pressure causes GC/OOM, network queuing increases, compaction contends I/O with writes, thread scheduling overhead rises. Diagnosis: watch which of the four resource categories plus app-level metrics shows inflection first. Fix: find true bottleneck, then scale, rate-limit, optimize, or isolate per corresponding direction.

Issue 3: Scaling shows no obvious effect

Phenomenon: added machines, still slow. Principle: may have misidentified bottleneck, or system isn't effectively using new resources — e.g., CPU starved but added memory, or real bottleneck in shuffle, network, hot partition, metadata, or single-thread operator. Diagnosis: compare pre/post scaling resource utilization, throughput, latency, and whether bottleneck location shifted. Fix: reconfigure per true bottleneck type. Prevention: re-test after every scaling, verify with data not assumptions.

Issue 4: Small-cluster extrapolation severely distorted

Phenomenon: small cluster runs great, large scale performance drops significantly. Principle: coordination overhead and resource contention don't necessarily grow linearly — e.g., metadata, shuffle, compaction scheduling, object storage access, network communication. Fix: do one near-real-scale stress test for critical chains, record scaling curves, not just apply a fixed "discount factor." Ultimately should obtain scaling curves of: node scale → throughput → P95/P99 → CPU/memory/I/O/network. This has far more capacity planning value than a single "single-node throughput" number.

9. Advanced Interview and Architecture Thinking

Q1: New real-time business launching — how to do capacity assessment?

Key points: clarify business peak and SLA first, then layered stress testing for capability baseline, end-to-end stress test for inflection point, calculate scale via "peak demand ÷ measured effective capacity + resource headroom + failure redundancy," and simulate degradation. Here "measured effective capacity" must satisfy: lag converges, checkpoints stable, compaction doesn't continuously backlog, P95/P99 meets SLA — not just raw throughput peak.

Q2: CPU saturated vs disk I/O saturated — how do scaling directions differ?

Key points: CPU bottleneck — first check compute operators, parallelism, data skew; I/O bottleneck — check disk throughput, IOPS, await, queue depth, read/write patterns; if needed, faster disks, more I/O bandwidth, or resource isolation; blindly adding CPU usually ineffective.

Q3: TPC-DS scores high, why is production still slow?

Key points: benchmark data and query templates don't represent real load; must swap in real data distribution, real concurrency, real SQL templates, and test end-to-end chain, not just single-engine scores.

Q4: Why can't we fully rely on small-cluster linear extrapolation?

Key points: shuffle, metadata, compaction, network, object storage access overheads may change non-linearly with scale; should obtain scaling curves via multi-scale stress testing, then derive capacity, not apply a fixed discount factor.

Q5 (Connecting Question): How does capacity planning prevent Article 13's promotion avalanche?

Key points: establish capacity baseline per peak + promotion multiplier, stress test to verify safe operating range, set three water level alerts, pre-verify degradation plans — expose capacity issues in stress test env, not first time at promotion scene.

10. Summary, Knowledge Extraction, and Next Article

This article gave Article 13's "reserve per peak, three water levels" a quantification method: first separate what write/query/mixed stress testing each tests, then use TPC-DS for engine capability verification and real load for capacity assessment; during stress testing watch CPU/memory/disk/network plus app metrics for inflection points; derive production scale via "peak demand ÷ measured effective capacity + resource headroom + failure redundancy"; finally decide vertical vs horizontal scaling per bottleneck type.

Recap: we built a complete capacity assessment closed loop, mastered four bottleneck identification and scaling directions, understood benchmark limitations, linear extrapolation risks, and why final capacity must return to real business load and SLA.

One-sentence conclusion: The essence of capacity planning is "using real inflection points from stress testing to replace guesswork estimates" — first identify the bottleneck resource correctly, then calculate effective capacity and safety margin; if direction is wrong, adding more is waste.

Next article (Article 15) enters lakehouse + AI intersection: Lakehouse + AI Engineering Implementation — RAG system production ops, embedding pipelines, Milvus performance tuning, recall rate continuous optimization, LLM-lakehouse integration architecture, and AI data governance. Previous articles built the data pipelines; next article starts feeding them to vector retrieval and LLMs.