Beyond Linpack: How HPCG, Graph500, and IO‑500 Redefine Supercomputer Rankings

In a previous post the author analyzed the 2017 TOP500 and Green500 lists, noting that China’s Taihu Light and Tianhe‑2 continued to dominate the Linpack‑based rankings. This article shifts focus to benchmarks that capture aspects of supercomputer performance that Linpack cannot measure.

Why Linpack Is No Longer Sufficient

The TOP500 ranking relies on the Linpack (HPL) benchmark, which has been unchanged since 2008. As applications increasingly use more complex differential equations, Linpack’s focus on solving dense linear systems no longer reflects real‑world workloads. Jack Dongarra, co‑creator of TOP500, has advocated for the High Performance Conjugate Gradient (HPCG) benchmark, which stresses memory, network latency, and overall system balance.

HPCG scores are typically much lower than Linpack scores; the HPCG/HPL efficiency ratio for most systems is under 5 %, with many around 1–3 %.

HPCG Rankings (2017)

In the first half of 2017, China’s Tianhe‑2 achieved 580 TFLOPS on HPCG, placing second behind Japan’s K computer (602 TFLOPS). Taihu Light ranked third with 480 TFLOPS, but its HPCG/HPL efficiency was only 0.4 %, the lowest among the top ten.

Graph500 and Green Graph 500

Beyond TOP500 and Green500, the Graph500 list evaluates data‑intensive workloads using the GTEPS (Giga‑Traversed Edges per Second) metric. Green Graph 500 applies the same metric but ranks systems by performance per watt, combining the Green500 power model with Graph500 results.

In the 2017 Graph500 half‑year ranking, Japan’s K computer took first place, Taihu Light was second, and Tianhe‑2 fell to eighth.

IO‑500 Benchmark

Released in 2017, IO‑500 measures overall I/O subsystem performance through two parts: a bandwidth test using the IOR program and a metadata test using mdtest and the find command.

The 2017 IO‑500 champion was the IME system from DDN (deployed at JCA‑HPC), scoring 43 % higher than the runner‑up. The second place went to DataWarp from Cray (deployed at KAUST). Lustre, BeeGFS, and Spectrum Scale (GPFS) were the most common parallel file systems among the top entries.

Application and Micro‑Benchmarks

Common application benchmarks include:

GTC‑P (Gyrokinetic Toroidal Code) : Particle‑in‑cell simulation of ion motion in a tokamak, solving Vlasov‑Poisson equations.

Meraculous : Large‑scale parallel genome assembly using de Bruijn graphs.

MILC : Quantum Chromodynamics lattice calculations for sub‑atomic physics.

MiniDFT : Plane‑wave density functional theory for material modeling.

MiniPIC : Solves the Boltzmann equation for electrostatic fields in arbitrary domains.

PENNANT : 2‑D unstructured finite‑element mesh for advanced architecture research, supporting MPI, OpenMP, and CUDA.

SNAP : Proxy application modeling neutral particle transport, derived from Los Alamos’ PARTISN.

UMT : Scalable solver for time‑dependent, energy‑dependent radiation problems on distributed memory systems.

Micro‑benchmarks (more generic tools) include:

Crossroads/N9 DGEMM : Multi‑threaded dense matrix‑multiply to measure per‑node floating‑point throughput.

IOR : Parallel POSIX and MPI‑IO bandwidth measurement.

mdtest : MPI‑coordinated metadata benchmark for file and directory operations.

STREAM : Measures sustainable memory bandwidth; Crossroads/N9 is an enhanced version.

Conclusion

While each benchmark highlights a different aspect of supercomputer capability, together they provide a multi‑dimensional view of the world’s most powerful machines. Linpack remains relevant, but HPCG, Graph500, IO‑500 and the suite of application/micro‑benchmarks serve as essential complements for a more comprehensive performance assessment.