GPU memory optimization

DataFunTalk

Dec 6, 2023 · Artificial Intelligence

Distributed Training Techniques and Quantitative Analysis for Large Language Models (GPT‑175B)

This article presents a comprehensive overview of state‑of‑the‑art distributed training methods for large language models, using GPT‑175B as a case study to analyze memory, communication, and compute overheads, and to recommend practical optimization strategies such as tensor, pipeline, and sequence parallelism, ZeRO‑1 optimizer, and selective activation checkpointing.

GPU memory optimizationLLMMegatron

0 likes · 22 min read

Distributed Training Techniques and Quantitative Analysis for Large Language Models (GPT‑175B)

Kuaishou Large Model

Jul 29, 2022 · Fundamentals

How Automatic Quantization Slashes Memory Use in High‑Resolution Physical Simulations

This article explains how researchers applied quantization techniques to high‑resolution physical simulations, enabling over 50% memory reduction without noticeable visual loss, by modeling error propagation, using constrained optimization, and introducing dithering, with results demonstrated on GPU‑based smoke, fluid, and elastic body simulations.

GPU memory optimizationSIGGRAPHcomputer graphics

0 likes · 6 min read

How Automatic Quantization Slashes Memory Use in High‑Resolution Physical Simulations