How to Seamlessly Transition HPC Workloads to a Hybrid Cloud Strategy

Background

Cloud computing has become a fundamental IT infrastructure strategy for most enterprises, with Gartner predicting a 36.8% increase in IaaS demand and 95% of companies having a cloud strategy, half of which adopt a hybrid approach.

Why HPC Lagged Behind the Cloud

Despite more than a decade of HPC running in the cloud—starting with Univa’s AWS cluster in 2008—many organizations hesitated to move HPC workloads due to concerns over data security and a lack of suitable technology, outweighing the benefits of reduced management costs and avoided hardware ownership.

Recent Shift Toward Hybrid Cloud HPC

A 2017 customer survey showed a ten‑fold increase in interest and usage of cloud‑based HPC. Companies now see economic and business opportunities in hybrid cloud, such as accessing dedicated hardware like Nvidia GPUs without affecting CAPEX, and scaling compute‑intensive workloads such as TensorFlow machine‑learning models.

Key Strategies for a Successful Hybrid Cloud Adoption

Automation of Policies : Implement policy‑management tools that automatically decide which workloads run locally versus in the cloud, adjust based on workload metrics and history, and can, for example, launch a cloud instance for a task waiting too long or terminate idle instances.

Cloud Bursting : Use a deployment model where applications run in a private cloud or data center and burst to the public cloud when demand peaks, linking the bursting logic to the workload management layer rather than the application or infrastructure layer.

Preserve Existing Workflows : Keep familiar submission methods (e.g., qsub) and scheduling pipelines to avoid errors and delays caused by sudden changes in pipelines.

Custom Machine Images : Build and use your own cloud images to ensure applications run consistently across on‑premises and cloud nodes, rather than relying on vendor‑specific VM images.

Embracing the Hybrid Future

Hybrid cloud offers a win‑win for public‑cloud providers and HPC users. As more HPC workloads migrate, public‑cloud services will continue to improve security, GPU, and machine‑learning offerings, further driving adoption.