Demystifying NVIDIA GPU Core Architecture: From Basics to AI Performance

Why GPUs Matter Today

In the era of massive compute demand, GPUs have moved beyond gaming graphics to become the backbone of large‑model training, intelligent clusters, and autonomous‑driving simulations. Developers often use PyTorch or CUDA without fully grasping the underlying architecture, leading to slow models, out‑of‑memory errors, and communication stalls.

1. GPU vs. CPU: Opposite Design Philosophies

CPU chips allocate roughly half of their die to cache and a quarter to complex control logic, leaving only 25% for arithmetic. GPUs invert this ratio: about 90% of the die is filled with simple compute cores, while cache and scheduling occupy a tiny fraction. Think of a CPU as a master chef handling complex, dependent tasks, whereas a GPU is a factory floor with tens of thousands of workers each performing simple add‑multiply operations in lockstep.

1.1 Deciding the Right Processor

Ask two questions: Is the computation logic complex? Is the data volume huge? Complex, branch‑heavy, low‑data‑volume tasks belong on the CPU; simple, massively parallel, data‑heavy workloads belong on the GPU.

1.2 CUDA Liberates GPUs from Graphics

Early GPUs could only render images; numerical work required disguising data as textures. NVIDIA’s CUDA opened a direct path for developers to write kernel functions that run on thousands of GPU threads, turning GPUs into programmable parallel processors and paving the way for today’s AI boom.

1.3 Three Eras of Compute Power

Ten years ago CPUs dominated server design. The big‑data wave exposed CPU serial bottlenecks, allowing GPUs to rise. In the current AI era, massive matrix multiplications in Transformers push GPUs to the core of compute, relegating CPUs to task orchestration.

2. Graphics Rendering: The Birthplace of Parallel Logic

The original GPU pipeline—vertex processing, tessellation, geometry, rasterization, fragment shading, pixel output—processes millions of primitives in parallel. This pipeline gave rise to the Streaming Multiprocessor (SM) and the CUDA Core, originally a pixel‑shader unit that evolved into a general‑purpose compute element.

SIMT: The Secret Sauce

CPU SIMD executes one instruction on multiple data streams but stalls all lanes on a branch. GPU SIMT issues a single instruction to a whole warp (32 threads on NVIDIA) while each thread maintains its own program counter, allowing divergent branches with limited performance loss.

3. Full Execution Model: Kernel → Grid → Block → Warp

3.1 Kernel

A kernel is a device‑side parallel function launched by the CPU. For example, a PyTorch matrix‑multiply call is compiled down to a kernel that runs on the GPU.

3.2 Thread Hierarchy

CUDA organizes threads in three layers: Thread (single execution unit), Block (a group sharing shared memory and synchronisation), and Grid (all blocks launched by one kernel). Hardware limits the maximum block size, and the scheduler maps blocks onto SMs.

3.3 Warp Scheduling

Each SM schedules warps, not individual threads. A warp’s 32 threads execute the same instruction; divergent branches cause warp divergence and modest performance loss.

3.4 Latency Hiding

Unlike CPUs that rely on large caches, GPUs hide memory latency by maintaining a massive pool of ready warps. When one warp stalls on a memory request, the SM instantly switches to another ready warp, keeping the hardware busy.

4. GPU Hardware Stack

A GPU chip is a hierarchy: Chip → GPC (graphics cluster) → TPC (texture cluster) → SM (streaming multiprocessor) → CUDA Core / Tensor Core. The NVIDIA H100, for instance, contains 132 SMs, each with 128 FP32 cores, totaling ~1.7 × 10⁴ arithmetic units.

4.1 SM Internal Units

CUDA Core – basic floating‑point/integer arithmetic.

Tensor Core – matrix‑multiply‑accumulate unit for AI, supporting BF16, FP8, FP4.

Warp Scheduler – dispatches warps to execution pipelines.

Register File – private high‑speed storage per thread.

Shared Memory – on‑chip cache shared by threads in a block.

L1 Cache & Load/Store Units.

Increasing SM count directly raises peak throughput; H100 adds 24 SMs over A100, delivering roughly three‑fold performance.

4.2 Evolution Toward AI

GPU generations have added AI‑specific hardware: programmable shaders → unified rendering → CUDA → Tensor Cores → low‑precision engines and chiplet‑based multi‑GPU packages (GH200, MI300) with NVLink or Infinity Fabric interconnects.

5. Memory System: The Real Bottleneck

Performance stalls are rarely due to compute limits; they stem from memory bandwidth. The hierarchy is: Register → Shared Memory → L1/L2 Cache → HBM (high‑bandwidth memory) → PCIe host memory. HBM offers several times the bandwidth of traditional GDDR, making it the default for data‑center GPUs.

5.1 Memory Coalescing

When 32 threads of a warp read consecutive addresses, the hardware merges the accesses into a single transaction. Scattered accesses explode the number of transactions and cripple bandwidth.

5.2 Shared‑Memory Tiling

Loading a matrix tile into shared memory lets all threads in a block reuse the data, cutting global‑memory reads dramatically. Classic GEMM kernels achieve 5–10× speedups this way.

6. Software Stack from Python to Silicon

Typical PyTorch execution flow: Application (PyTorch) → Operator library (cuBLAS/cuDNN/NCCL) → CUDA/HIP programming model → Driver runtime → GPU ISA. A call like torch.matmul is lowered to a cuBLAS kernel that runs on Tensor Cores, pulling data from HBM.

Compilation proceeds via NVCC: host code + device code → PTX (intermediate) → driver JIT compiles PTX to device‑specific machine code, which is finally dispatched to SMs.

7. Specialized Units for Modern Workloads

Tensor Core – accelerates the D=A×B+C pattern ubiquitous in deep‑learning layers, supporting low‑precision formats.

RT Core – speeds ray‑tracing collision and traversal, enabling real‑time ray‑traced graphics and high‑fidelity simulation.

Multi‑GPU Interconnect – NVLink (intra‑node) and InfiniBand/RoCE (inter‑node) provide the bandwidth required for trillion‑parameter models.

8. Training vs. Inference GPUs

Training workloads demand massive HBM capacity, high‑speed interconnect, and strong TF32/FP8 compute (e.g., H‑series, MI300X). Inference prefers cards with large per‑card memory, quantisation support, and lower power (e.g., L40S).

8.1 Why Large Models Exhaust GPUs

Transformers keep model weights in memory, generate large activation maps, and have quadratic attention cost with sequence length, causing both compute and memory pressure.

8.2 Future Directions

Next‑gen GPUs will push even lower‑precision formats (FP4, INT4), increase per‑card HBM beyond 200 GB, double inter‑GPU bandwidth, and adopt chiplet‑level CPU‑GPU integration for better energy efficiency.

9. End‑to‑End Optimization Methodology

9.1 Locate the Bottleneck with Roofline

Measure memory‑bandwidth utilisation and SM occupancy. Bandwidth‑bound kernels show full bandwidth but low compute utilisation; compute‑bound kernels show high SM usage but low memory traffic.

9.2 High‑Impact Tuning Steps

Memory optimisation – coalesced accesses, shared‑memory reuse, minimise host‑device copies, restructure data layout (AoS → SoA).

Warp‑scheduling – avoid divergent branches, choose appropriate block dimensions.

Resource balancing – keep register and shared‑memory usage moderate to preserve occupancy.

Algorithmic tricks – operator fusion, recompute, quantisation, dynamic batching.

System‑level – overlap computation with data transfer, schedule multiple kernels concurrently.

High occupancy alone does not guarantee speed; a low‑occupancy kernel that fully utilises Tensor Cores can outperform a high‑occupancy scalar kernel.

9.3 Profiling Toolchain

Nsight Systems / PyTorch Profiler – global timeline, multi‑kernel and multi‑GPU stalls.

Nsight Compute / rocprof – per‑kernel metrics: warp stalls, memory transactions, resource utilisation.

nvidia‑smi / rocm‑smi – real‑time power, memory, and load monitoring.

10. Full‑Stack Deployment and Hardware Selection Pitfalls

10.1 Avoid TFLOPS‑Only Decisions

Peak TFLOPS ignore memory capacity, bandwidth, interconnect, and software ecosystem, which often become the real constraints.

10.2 Consumer vs. Data‑Center Cards

Consumer GPUs lack NVLink, ECC memory, and robust thermal design, making them unsuitable for large‑scale training.

10.3 Mixing Training and Inference Cards

Training cards prioritise inter‑GPU bandwidth; inference cards focus on per‑card memory and quantisation efficiency. Mixing them leads to sub‑optimal performance.

10.4 Reserve Memory Headroom

Models need extra space for gradients, optimizer state, and KV caches; fitting parameters exactly often triggers out‑of‑memory crashes.

Conclusion

GPU is not merely an “accelerator card”; it embodies massive parallelism, high‑bandwidth data flow, and specialised compute units. Understanding the hardware hierarchy, memory behaviour, and software stack enables developers, ops engineers, and hardware buyers to diagnose bottlenecks, apply targeted optimisations, and avoid costly mis‑steps across graphics, AI, scientific simulation, and emerging edge workloads.