This comprehensive guide covers 100 essential GPU fundamentals, from basic definitions and architecture to core technologies, performance optimization, emerging trends, and industry developments, providing a complete technical foundation for graphics, AI, and high‑performance computing applications.
This article examines how to centrally manage GPU resources across heterogeneous Kubernetes clusters using namespace‑based RBAC isolation, virtual control‑plane solutions like vcluster, and multi‑cluster tools such as Karmada, comparing their architectures, use cases, advantages, and limitations to guide enterprise‑level deployment decisions.
This comprehensive guide covers GPU definitions, evolution, core components, architectural designs, performance metrics, programming models, deep‑learning applications, comparisons with other processors, practical use cases, optimization techniques, and future trends, providing a solid foundation for anyone interested in modern graphics and compute acceleration.
This tutorial walks you through using a one‑click script to create a GPU‑enabled Kind Kubernetes cluster, evenly distribute GPU resources across nodes with nvkind, install necessary drivers and toolkits, deploy a vLLM‑served large language model, and verify its operation, all on a local or cloud environment.
In March 2024 Nvidia unveiled the Blackwell GPU family and the GB200 NVL72 architecture, featuring 3‑4 nm processes, redesigned CUDA cores, next‑gen ray‑tracing, upgraded DLSS, massive FP16/FP8 compute gains, 8 TB/s memory bandwidth, and NVLink Gen5, while also presenting complex power, cooling, and packaging challenges for large‑scale AI deployments.
The article explains why standard GEMM libraries may fall short for special matrix shapes, introduces NVIDIA’s open‑source CUTLASS library, details its hierarchical tiling architecture, and walks through a complete device‑API example that customizes tile sizes and data layouts to achieve near‑hand‑written kernel performance on modern GPUs.
This article compares NVIDIA’s China‑specific B30 and high‑end H20 GPUs, detailing their CPU/CPU architecture updates, memory technologies, architectural differences, performance metrics, power and cooling characteristics, and price positioning, to help enterprises and developers choose the most suitable accelerator for AI and deep‑learning tasks.
The Chinese AI server market is booming, with GPU servers still dominant while non‑GPU accelerators surge, IDC forecasts a compound annual growth above 20% through 2028, and leading vendors such as Inspur, H3C, and emerging Ascend‑based manufacturers reshaping the competitive landscape.
Coupang’s AI platform replaces costly data‑copy steps with a distributed cache that automatically pulls data from a central lake, boosts GPU utilization across regions, cuts storage and operational expenses, and speeds up model training by up to 40% while simplifying deployment via Kubernetes.
NVIDIA announced at GTC 2025 that CUDA will now natively support Python, allowing developers to write GPU‑accelerated code directly in Python without C/C++ knowledge, introducing new APIs, libraries, JIT compilation, performance tools, and a tile‑based programming model that aligns with Python’s array‑centric workflow.
This article traces the history of graphics GPUs from their 1980s origins to modern AI and high‑performance computing roles, examines China's emerging GPU market and its challenges, and reviews the key graphics and compute standards shaping the industry today.
This article explains the concepts of AI Pods and GPU clusters, compares vertical (scale‑up) and horizontal (scale‑out) expansion, describes XPU types, discusses internal and inter‑pod communication, and evaluates the benefits and drawbacks of each scaling approach along with relevant networking technologies.
NVLink Fusion, unveiled at Computex 2025, extends NVIDIA’s NVLink technology to enable high‑bandwidth, low‑latency connections between CPUs and GPUs or third‑party accelerators, offering up to 900 GB/s bandwidth, flexible heterogeneous configurations, ecosystem expansion, performance gains for AI training and inference, and potential cost reductions.
This article examines the impact of U.S. export restrictions on Nvidia’s China‑specific GPU lineup, detailing the specifications and architectural changes of the H20, B20, and B40 chips, while also discussing domestic alternatives and the broader implications for AI compute in China.
This article provides a comprehensive analysis of GPU core parameters—including compute units, memory systems, floating‑point performance, power consumption, and manufacturing process—while comparing leading international and domestic GPU products to help readers choose the right accelerator for AI, HPC, or graphics workloads.
A US Senate bill introduced by Senator Tom Cotton requires Nvidia, AMD, Intel and other high‑end GPU and AI processor makers to embed geolocation tracking, imposing six‑month compliance deadlines, new reporting obligations, and potentially billions of dollars in added R&D and export‑control costs.
This roundup highlights Tencent's GPU capacity for future models, Alibaba's fully disclosed Qwen3 technical report, Anaconda's unified AI platform, Parasoft's AI‑enhanced SOAtest, and Google Cloud's GA of the Apigee API Management Operator, offering a snapshot of current AI advancements.
This article analyzes NVIDIA's evolving AI supercomputer architectures—detailing the H100‑based 256‑GPU SuperPod, the GH200‑based 256‑GPU SuperPod with integrated Grace CPU, and the GB200‑based 576‑GPU SuperPod—examining their NVLink and InfiniBand topologies, bandwidth limits, and scalability challenges.
NVIDIA announced native Python support in its CUDA toolkit, enabling developers to write GPU‑accelerated code directly in Python, detailing the new programming model, JIT‑based architecture, performance benefits, and the broader impact on AI development and the developer ecosystem.
The article describes how a large‑scale dispatch system was re‑engineered with NVIDIA TritonServer to unify GPU‑accelerated operations‑research kernels and deep‑learning models, detailing a three‑stage architecture (in‑process, cross‑process, cross‑node), the performance, stability and memory challenges addressed, and future plans for heterogeneous GPU scaling.
This guide walks you through cross‑platform environment variable configuration, Docker containerization, GPU resource strategies, concurrency tuning, and security hardening for Ollama, providing practical code snippets and best‑practice tables to unleash its full potential in development and production.
This guide explains how to prepare, deploy, and verify the Qwen3‑8B large language model on an Alibaba Cloud Container Service for Kubernetes (ACK) cluster using ACS GPU resources, covering prerequisites, model download, storage setup, Kubernetes manifests, and testing the inference service.
This article provides a comprehensive analysis of modern server architecture, covering the evolution from CISC to RISC, the rise of heterogeneous computing with GPUs and accelerators, diverse form factors, core component technologies, reliability mechanisms, performance benchmarking, certification standards, and emerging trends such as liquid cooling and AI‑native designs.
This article objectively compares AMD and NVIDIA graphics cards for Linux users, covering out‑of‑the‑box driver support, Wayland compatibility, gaming performance, machine‑learning capabilities, and cost‑effectiveness to help readers choose the best GPU for their needs.
This article presents a detailed comparison of major NVIDIA GPUs—including RTX 4090, RTX 4090 D, RTX 3090, A10, A40, A100, and H100—covering memory size, bandwidth, Tensor BF16/FP16/FP32 throughput, FP16/FP32 performance, power draw and release dates, and explains how these specs affect AI workload efficiency.
This article analyzes NVIDIA's A100, H100, A800, H800, and H20 GPUs, compares their architectures, performance, and pricing, and provides a step‑by‑step guide for building a private AI compute center tailored to training, inference, and high‑performance computing workloads.
This guide explains how Alibaba Cloud's ACK One registered cluster provides multi‑region serverless GPU compute scheduling, addressing AI workload elasticity by using region‑specific labels, ResourcePolicy, and the ack‑co‑scheduler to automatically balance resources across regions.
This article analyzes the Xinchuang hardware ecosystem and server industry from 2024 to 2026, covering supply‑chain structure, shipment growth, market share, competitive tiers, downstream application demands, and the technical and ecological challenges that hinder full domestic substitution.
At the recent GTC conference Nvidia unveiled its roadmap of AI‑focused GPUs—Ampere, Hopper, Blackwell, Rubin and the upcoming Feynman—each named after a pioneering scientist, and this article explores the historical contributions of André‑Marie Ampère, Grace Hopper, David Blackwell, Vera Rubin and Richard Feynman, linking their legacies to the architectures’ innovations.
As AI models grow to trillion‑parameter scales, training them demands massive GPU clusters whose performance is increasingly limited by network bandwidth; this article examines why traditional PCIe interconnects become bottlenecks and how NVIDIA's NVLink and NVSwitch technologies dramatically improve multi‑GPU communication and overall system efficiency.
This article analyzes the AI compute chip ecosystem, covering GPU, FPGA, and ASIC categories, market share projections, key performance metrics such as TOPS, power and area, and provides a detailed overview of leading global vendors and emerging Chinese companies with their technical specifications and competitive positioning.
This article explains how to dramatically speed up PyTorch model training using code optimizations, mixed‑precision, torch.compile, distributed data parallelism, and DeepSpeed, presenting benchmark results that show up to 11.5× acceleration on multiple GPUs while maintaining high accuracy.
Training today's massive AI foundation models demands extensive GPU resources and sophisticated multi‑GPU communication, making technologies like NVLink and NVSwitch crucial for efficient distributed training, while data‑parallel and model‑parallel strategies together optimize performance across large‑scale hardware clusters.
This article presents a comprehensive benchmark of DeepGEMM, Cutlass, and Triton on NVIDIA H20 and H800 GPUs, analyzing TFLOPS, bandwidth, latency, and speedup across various matrix sizes, and concludes which library is optimal for different workload scenarios.
This article explains scientific notation, the IEEE‑754 floating‑point standard, the structure of FP32 and FP64 numbers, and how computational power is measured using FLOPS, illustrating CPU and GPU FP32 performance calculations and why FP32 is the common benchmark for AI workloads.
The article traces NVIDIA’s GPU architecture evolution from the Volta era’s pioneering Tensor Cores through Turing, Ampere, Hopper, and the latest Blackwell and Rubin designs, highlighting key innovations such as mixed‑precision support, sparsity, NVLink, and their impact on deep‑learning performance.
This article provides a comprehensive analysis of the GPU industry, covering product classifications, key characteristics, market size evolution, competitive dynamics among major players such as NVIDIA, AMD, and Huawei, policy influences, and future growth projections driven by AI and high‑performance computing demands.
This article explains the rise of AI‑specific processors, defines AI chips, compares their architectures, and examines the distinct requirements of training versus inference chips while outlining the main technology routes (GPU, FPGA, ASIC) and future outlook.
This article explains the cuda‑python ecosystem—including its core packages, installation via pip or conda, the experimental cuda.core API, a full Python‑to‑CUDA workflow with NVRTC compilation, performance comparison to C++, the covered APIs, and an overview of NVIDIA's new cuTicle programming model.
The article reviews Nvidia GTC 2025’s session on writing CUDA kernels with Python, compares tools such as Numba, CuPy, PyTorch extensions and cuda‑python, demonstrates a segmented reduction example with C++ and Python code, explains the underlying CUDA concepts, and shows how to install and use cuda‑python to simplify kernel development.
The article explains GPU fundamentals—from the end of Dennard scaling and why GPUs excel in parallel throughput, through CUDA programming basics like the SAXPY kernel and SIMT versus SIMD execution, to the evolution of the SIMT stack, modern scheduling, and a three‑step core architecture design.
The article explains the theoretical and practical relationships between GPU clock frequencies, power consumption, and FLOPS, describes key hardware metrics such as SM, memory, and video clocks, shows how to query and set these values with nvidia‑smi, and presents experiments on a Tesla P4 that reveal the non‑linear trade‑offs between performance, power, and temperature.
This article details JD Retail's AI engine that seamlessly supports both GPU and domestic NPU hardware, describing its heterogeneous cluster architecture, unified training and inference APIs, performance optimizations, extensive model coverage, and multiple production use cases across e‑commerce, logistics, and intelligent assistance.
At HPCA 2025, two standout papers—UniNDP, a unified compilation and simulation tool for near‑DRAM processing architectures, and WarpDrive, a GPU‑based fully homomorphic encryption accelerator leveraging Tensor and CUDA cores—demonstrate significant performance gains for AI workloads and privacy‑preserving computation.
This article analyzes Nvidia’s evolution from a graphics pioneer to an AI hardware leader and examines Huawei’s Ascend AI processor roadmap, detailing technical specifications, ecosystem strategies, recent product releases, and the potential impact on related technology stocks.
This article compiles a market overview of 52 DeepSeek integrated machines, detailing GPU chips, price ranges from tens of thousands to millions, major Chinese cloud vendors, and diverse application scenarios such as intelligent customer service, data processing, and smart governance.
This guide walks through checking the system environment, installing the NVIDIA Docker plugin, configuring Docker to use the NVIDIA runtime, verifying GPU access with Docker, deploying the NVIDIA device plugin on a Kubernetes cluster, and running GPU‑accelerated workloads in pods.
The article summarizes Zhihu's machine‑learning platform lead Wang Xin's presentation on the ZhiLight large‑model inference framework, covering model execution mechanisms, GPU workload analysis, pipeline and tensor parallelism, GPU architecture evolution, open‑source engine comparisons, ZhiLight's compute‑communication overlap and quantization optimizations, benchmark results, supported models, and future directions.
This tutorial explains how to set up Ollama for CPU or GPU environments, run the DeepSeek R1:7b large language model, and use the open‑source Dify platform to create and deploy a custom AI application, providing step‑by‑step commands and configuration details.
This guide walks you through using Alibaba Cloud Container Compute Service (ACS) to provision GPU resources, prepare the QwQ-32B model, configure persistent storage, deploy the model with vLLM, set up OpenWebUI, verify the service, and optionally benchmark its performance, all with detailed commands and YAML examples.
The article explains why nvidia-smi cannot display GPU processes when run inside a container, analyzes the underlying pid‑namespace isolation and kernel‑level restrictions, and provides three practical solutions—including using hostPid, custom kernel interception modules, and the nvitop tool—plus a workaround for gpu‑operator deployments.
This article explains how GPU Dynamic Voltage and Frequency Scaling (DVFS) reduces power consumption while preserving performance, describes NVIDIA GPU Boost 4.0 features, outlines a hardware‑counter‑based GPGPU power‑estimation model built with a BP‑ANN, reports sub‑5% error on benchmarks, and discusses intelligent and multi‑GPU extensions.
This article provides a comprehensive technical overview of GPU cloud server design, covering data‑processing pipelines, hardware topology, NUMA considerations, PCIe and proprietary interconnects, multi‑GPU communication strategies, virtualization approaches (BCC and BBC), DPU acceleration, and future trends for scaling up and out.
JD Retail’s Nine‑Number Algorithm Platform delivers an end‑to‑end AI engine that unifies GPU and domestic NPU resources across a thousand‑card cluster, offering zero‑cost model migration, optimized training and inference pipelines, support for over 40 LLM and multimodal models, and proven business‑level performance that reduces dependence on overseas chips.
This article explains the design and implementation of GPU cloud servers, covering data processing pipelines, hardware selection, topology, interconnect technologies, virtualization, multi‑GPU communication methods, and Baidu's practical solutions for both virtualized and bare‑metal instances to boost AI inference and training performance.
The article reviews AMD's newly announced RX 9070 and RX 9070 XT graphics cards, detailing their 4 nm RDNA 4 architecture, core specifications, gaming performance gains over the RX 7900 GRE, AI workload improvements, FSR 4 enhancements, and launch pricing compared with NVIDIA's RTX 50 series.
This article describes JD Retail's AI engine built on domestic NPU chips, covering challenges, heterogeneous GPU‑NPU scheduling, high‑performance training and inference engines, extensive model support, real‑world deployment cases, and future plans for large‑scale chip clusters and ecosystem development.
This guide explains why DeepSeek is a compelling GPT‑4‑level alternative, provides hardware recommendations for various model sizes, and walks through a three‑step Windows deployment using Ollama, including installation, environment configuration, model download, performance tuning, and common troubleshooting tips.
The article analyzes NVIDIA's Rubin288 high‑density GPU cabinet, comparing its orthogonal CLOS architecture with the older non‑orthogonal designs, and explains how the new layout improves reliability, bandwidth, scalability, and cooling for modern data‑center HPC deployments.
DeepSeek’s OpenSourceWeek introduced FlashMLA, a GPU‑optimized MLA decoding kernel for Hopper GPUs that leverages FlashAttention and CUTLASS to dramatically improve large‑model inference performance, with early adoption showing up to 30% higher compute utilization and doubled speed in some scenarios.
This tutorial walks Java developers through building, training, evaluating, and deploying a CIFAR‑10 image classifier using PyTorch, covering data loading, preprocessing, network definition, loss and optimizer setup, GPU acceleration, model saving, and per‑class accuracy analysis.
This tutorial shows how to use Alibaba Cloud DataWorks' serverless GPU resource groups together with the open‑source LLaMA‑Factory framework to fine‑tune the Qwen2‑VL‑2B multimodal model for tourism‑domain Q&A, covering environment setup, dataset preparation, parameter configuration, training, and interactive inference.
This article explains why enabling NVCC's --fatbin-options -compress-all breaks remote GPU calls, describes the fatbin file layout, shows how to extract and analyze the binary with objcopy, and provides a step‑by‑step implementation of a decompression routine for both ELF and PTX sections.
This article walks through the NVCC compilation pipeline, explaining how CUDA source files are transformed into host and device binaries, detailing file extensions, compilation stages, command‑line options, intermediate artifacts, and the role of registration functions such as __nv_cudaEntityRegisterCallback and __sti____cudaRegisterAll.
This article explains how to use Alibaba Cloud ACK Edge to manage on‑premise GPU resources and seamlessly fall back to cloud‑based ACS Serverless GPU via virtual nodes for deploying DeepSeek R1 inference, covering environment preparation, model download, storage setup, custom scheduling, and scaling strategies.
This guide explains how to deploy the DeepSeek‑R1 large language model on a Knative platform using an A10 GPU, covering preparation, service creation with appropriate annotations, YAML configuration, verification via curl, custom domain setup, and optional personal AI assistant deployment.
This article introduces PyTorch, covering installation on CPU/GPU, basic tensor operations, automatic differentiation, building and training neural networks, data loading with DataLoader, image classification on MNIST, model deployment, and useful tips for accelerating deep‑learning workflows.
This article explains how os‑criu provides transparent, OS‑level GPU checkpoint/restore, compares its performance with NVIDIA's cuda‑checkpoint, walks through building and installing the PhOS framework, demonstrates migration of a Llama2‑13b‑chat workload in Docker, and discusses current limitations and future Kubernetes integration plans.
This article provides a step‑by‑step, code‑rich tutorial for fine‑tuning the open‑source Llama 3 1B and 3B models on Google Colab using the Unsloth library and LoRA, covering environment setup, model loading, adapter insertion, dataset preparation, training configuration, inference, and model saving, all while keeping GPU memory usage low.
This article provides a detailed technical analysis of NVIDIA's A100, H100, A800, H800 and H20 GPUs, compares their architectures, performance and cost, and offers step‑by‑step guidance on building a private AI compute center, selecting hardware, software stacks and budgeting for different workloads.
This guide explains how to prepare an Alibaba Cloud GPU instance, install Docker and NVIDIA tools, pull or build a container image, and run the FP8‑quantized DeepSeek‑R1‑Distill‑Qwen‑32B model using vLLM and OpenWebUI for both offline and online inference.
This article breaks down the hardware and software expenses required to deploy a complete DeepSeek large‑language model on‑premises, revealing a total cost of roughly $110,000 and explaining why such an investment is prohibitive for most individual developers but may be justified for well‑funded research or corporate projects.
Learn step‑by‑step how to prepare a JD Cloud GPU instance, install GPU drivers, deploy Ollama, run DeepSeek‑R1 models, configure graphical clients like Chatbox on Windows and macOS, and optionally feed local data using AnythingLLM to build an offline knowledge base.
A recent Azul Systems study suggests that Java may surpass Python in enterprise AI development within the next 18‑36 months, highlighting Java's scalability, performance, and emerging GPU projects while acknowledging cultural and tooling advantages that still favor Python.
This guide explains how to deploy a production‑ready DeepSeek‑R1 inference service on Alibaba Cloud ACK using KServe, covering model preparation, storage configuration, service deployment, observability, autoscaling, model acceleration, gray‑release and GPU‑shared inference.
This guide walks you through preparing a Tencent Cloud account, creating a Cloud Studio workspace, installing Ollama, downloading and running the DeepSeek‑R1 large language model, interacting via terminal or API, and managing resources and model versions.
This article provides a comprehensive analysis of China's Xinchang server industry, covering the upstream component supply chain, mid‑stream manufacturers, downstream users, shipment statistics, market share evolution, competitive tiers, application demands, and the technical and ecological challenges facing domestic CPU and GPU development.
This guide explains how to deploy the full 671B DeepSeek R1 model on local hardware using Ollama, leveraging dynamic quantization to shrink model size, detailing hardware requirements, step‑by‑step installation, configuration, performance observations, and practical recommendations.
This article provides a comprehensive tutorial on locally deploying the full‑size DeepSeek R1 671B model using Ollama, covering dynamic quantization options, hardware specifications, detailed installation commands, configuration files, performance observations, and practical recommendations for consumer‑grade systems.
This article traces the GPU's journey from early 2D graphics cards to modern GPGPUs powering AI and HPC, explains core hardware components, compares GPU and CPU architectures, and details the 3D rendering pipeline that underlies graphics and parallel computation.
The article examines the AI compute chip ecosystem—covering GPUs, FPGAs, and ASICs like VPU/TPU—highlights market share trends, key performance metrics such as TOPS, power and area, and provides a detailed overview of leading global and Chinese manufacturers and their flagship products.
This guide walks you through the hardware and software prerequisites, Docker-based installation, environment configuration, model fine‑tuning, IDE integration, maintenance, and troubleshooting for running the DeepSeek AI programming assistant entirely on your own machine.
The article dissects NVCC’s compilation pipeline, showing how internal registration functions from host_runtime.h are injected into the host binary, how a simple CUDA demo is processed with --dryrun, and how the generated fatbin, PTX, and cubin files are linked and registered for GPU execution.
This article provides a comprehensive overview of modern GPU architecture, covering components such as SMs, GPCs, memory hierarchy, unified shader architecture, SIMT execution, warp scheduling, and compares IMR, TBR, and TBDR rendering pipelines while offering practical optimization techniques for developers.
This tutorial walks through the motivation for Nvidia GPU kernel interception, explains Linux kernel module basics and privilege rings, shows how to set up an Ubuntu environment, write and compile a minimal LKM, load and test it, then create a fake GPU character device and mount it into a Docker container for verification.
This article introduces a comprehensive set of Python performance‑enhancing tools and libraries—including NumPy, SciPy, PyPy, Cython, Numba, GPU‑based solutions, and various wrappers—explaining how they accelerate code execution, reduce memory usage, and enable efficient single‑ and multi‑processor programming.
The 2024 AI server market is booming with global shipments surpassing 1.2 million units in 2023 and projected to reach 1.67 million in 2024, driven by rapid growth in China’s AI compute capacity, distinct training and inference server designs, and facing challenges in GPU quality, high‑speed interconnects, and cooling solutions.
This guide explains how to use Alibaba Cloud's ACK Edge to create a secure, high‑availability Kubernetes cluster that unifies management and scheduling of ECS instances across multiple VPCs, regions, and accounts, with detailed scenarios, advantages, step‑by‑step procedures, and sample YAML deployments.
The article details Nvidia’s upcoming RTX 50 consumer GPUs, the Blackwell‑based Grace NVLink72 data‑center super‑chip, and the pocket‑sized Project DIGITS AI system, highlighting specifications, performance claims, pricing expectations, and the broader impact on the GPU market.
This guide walks through installing the NVIDIA container toolkit, configuring Docker to use the NVIDIA runtime, verifying GPU access, deploying the NVIDIA device plugin in Kubernetes, labeling GPU nodes, and running a GPU‑accelerated FFmpeg pod to confirm successful GPU integration.
This guide walks through checking system and software environments, installing and configuring the NVIDIA Docker plugin, verifying GPU access in Docker containers, deploying the NVIDIA device plugin on a Kubernetes cluster, creating GPU‑enabled pods, and troubleshooting common issues, all with concrete commands and configuration examples.
At CES 2025, Nvidia CEO Jensen Huang announced the RTX 50 series GPUs built on the Blackwell architecture, the Thor automotive processor, the Project Digits personal AI supercomputer, new AI agents and robotics initiatives, detailing pricing, performance specs, and partnerships across automotive and AI ecosystems.
The article provides a detailed technical analysis of Nvidia’s new GB300 and B300 GPUs, comparing their performance, memory architecture, and power consumption to previous generations, and examines how these changes affect AI inference workloads, NVL72 accelerator systems, and the supply‑chain strategies of major cloud providers.
This article dissects the GPUMounter project's implementation of dynamic GPU device mounting to a pod, detailing the roles of cgroups (v1 and v2) and Linux namespaces, and provides step‑by‑step command‑line examples and a CLI tool for practical use.
Broadcom’s market‑cap breakthrough past $1 trillion highlights its strategic push into AI ASICs, revealing how ASIC‑FPGA trade‑offs, collaborations with Google, and competition with Nvidia’s GPU ecosystem are reshaping the high‑performance computing landscape.
This talk presents the Ant Group team's recent work on large‑model inference memory optimization, covering GPU memory challenges, virtual memory management (VMM), the Virtual Tensor framework, LayerKV techniques, performance comparisons with Page Attention and FlashAttention, and extensive experimental results demonstrating reduced latency and higher QPS.
This article evaluates NVIDIA's China‑specific H20 and L20 inference chips, comparing their compute and memory‑bandwidth characteristics against A100, H100 and H200, and shows how they achieve superior throughput in large‑model inference despite reduced specifications.
This article explains the fundamentals of GPU architecture, the need for GPU virtualization, and walks through user‑level, kernel‑level, hardware‑level, and full GPU virtualization techniques, illustrating each layer with diagrams and code examples while highlighting practical deployment scenarios.
This article presents the YiNian LLM framework, detailing how KVCache, prefill/decoding separation, continuous batching, PageAttention, and multi‑hardware scheduling are used to speed up large language model inference while managing GPU memory and latency.
This article provides an in‑depth overview of NVIDIA’s latest GPU families—including A100/A800, H100/H200/H800, B100/B200, and L40S—detailing their release backgrounds, key specifications, typical application scenarios, and pricing to help readers understand their performance and market positioning.
