The article explains how John von Neumann’s 1945 decision to store programs in memory created the universal von Neumann architecture, why this simple design outlasted more optimal alternatives, and how his ideas also spawned parallel computing and game theory, shaping modern computers, AI, and distributed systems.
A new team member transformed a painfully slow Java data‑processing routine—originally taking 26,856 ms—by refactoring nested loops, extracting repeated calculations, and introducing a thread‑pool for parallel execution, reducing runtime to just 748 ms, and the article walks through the before‑and‑after code and key techniques.
The SSD framework and its SAGUARO optimization, developed by researchers from Stanford, Princeton, and Together AI, parallelize drafting and verification in speculative decoding, eliminating serial dependencies and achieving up to 2× faster inference than the world’s strongest engines and up to 5× speedup over standard autoregressive generation, while addressing challenges such as prediction accuracy, acceptance‑rate trade‑offs, and fallback strategies.
This article explains how AI model training has evolved from single‑GPU workloads to massive distributed training using MPI for CPU‑centric communication and NCCL for GPU‑centric communication, covering their histories, core concepts, programming interfaces, topology discovery, protocol choices, and performance testing on multi‑GPU clusters.
Nvidia leverages its GPUs to solve quantum computers' fragile error‑correction problem, introducing ultra‑fast NVQLink interconnect and the CUDA‑Q programming platform, creating a feedback loop that secures its dominance in both traditional and emerging quantum markets.
This article reviews eleven high‑performance Python libraries—Polars, Numba, orjson, PyO3, Blosc, Awkward Array, Dask, Vaex, Modin, scikit‑learn‑intelex, uvloop and PyPy—showing how they achieve multi‑fold speedups through Rust, JIT, SIMD, lazy evaluation and parallel execution, and offers guidance on when to choose each tool.
This article explains why GPUs have become the preferred platform for high‑performance computing, covering Dennard scaling, GPU speed advantages, theoretical FLOPS calculations, CUDA programming examples like SAXPY, the SIMT execution model, instruction pipelines, and modern techniques for handling branch divergence and register bank conflicts.
This article explains how NVIDIA transitioned from gaming graphics cards to general‑purpose GPUs with the first data‑center Tesla GPU, detailing the unified shader architecture, the internal components of TPCs and SMs, CUDA 1.0 programming basics, and performance calculations that illustrate the massive computational advantage over contemporary CPUs.
This tutorial demonstrates how to accelerate Python's compute‑intensive tasks by rewriting critical functions in Rust, using the pyo3 library to create Python extension modules, and compares single‑thread, multithreaded, and multiprocessing performance on Linux.
This article explains how modern GPUs evolved from graphics rendering to general‑purpose computing, details the CPU‑GPU heterogenous architecture, walks through a CUDA demo that adds two billion‑element arrays, compares CPU and GPU performance, and describes the compilation, loading, and execution pipeline of CUDA kernels.
This comprehensive guide covers 100 essential GPU fundamentals, from basic definitions and architecture to core technologies, performance optimization, emerging trends, and industry developments, providing a complete technical foundation for graphics, AI, and high‑performance computing applications.
This comprehensive guide covers GPU definitions, evolution, core components, architectural designs, performance metrics, programming models, deep‑learning applications, comparisons with other processors, practical use cases, optimization techniques, and future trends, providing a solid foundation for anyone interested in modern graphics and compute acceleration.
This article provides an in‑depth overview of Python’s multiprocessing module, covering its fundamentals, process creation, inter‑process communication methods such as Queue, Pipe, shared memory, process pools, synchronization techniques, and practical best‑practice guidelines for effective parallel programming.
In this candid interview, 83‑year‑old Turing‑Award winner Jeffrey Ullman reflects on his decades‑long impact on compilers, databases, and algorithms, discusses the unpredictable nature of technological revolutions, explores the rise of large language models, parallel computing, prompt engineering, and the challenges of adapting education and software engineering to rapid AI‑driven change.
Training today's massive AI foundation models demands extensive GPU resources and sophisticated multi‑GPU communication, making technologies like NVLink and NVSwitch crucial for efficient distributed training, while data‑parallel and model‑parallel strategies together optimize performance across large‑scale hardware clusters.
The article explains GPU fundamentals—from the end of Dennard scaling and why GPUs excel in parallel throughput, through CUDA programming basics like the SAXPY kernel and SIMT versus SIMD execution, to the evolution of the SIMT stack, modern scheduling, and a three‑step core architecture design.
DeepSeek’s recent Open‑Source Week unveiled a suite of AI‑focused tools—including the DualPipe pipeline parallelism algorithm, the EPLB expert load balancer, detailed training‑inference framework data, the high‑performance 3FS parallel file system, and the Smallpond data‑processing framework—each with GitHub links and performance highlights.
This article traces the GPU's journey from early 2D graphics cards to modern GPGPUs powering AI and HPC, explains core hardware components, compares GPU and CPU architectures, and details the 3D rendering pipeline that underlies graphics and parallel computation.
This article walks through a practical APOD workflow for CUDA development—assessing bottlenecks, parallelizing with cuBLAS/cuFFT/Thrust, optimizing iteratively, and deploying—while covering nvcc compilation flags, PTX virtual ISA, nvprof profiling, core terminology (SP, SM, warp, grid, block, thread), indexing patterns, and unified memory references.
This article presents a concise tutorial on essential Python data science libraries, covering data cleaning with Pandas, numerical analysis with NumPy and SciPy, visualization with Matplotlib and Seaborn, machine learning with scikit‑learn, NLP with NLTK and spaCy, time‑series modeling, image processing, database access, and parallel computing, each illustrated with ready‑to‑run code examples.
The paper describes transforming a tightly coupled, multi‑stage video search ranking pipeline into a modular, end‑to‑end large‑model architecture that decouples recall, employs a graph‑engine parallel framework and elastic compute allocation, thereby boosting performance, flexibility, personalization and lowering long‑term operational costs.
The article outlines how video search ranking has shifted from a tightly‑coupled multi‑stage cascade to an extensible, end‑to‑end, model‑centric framework called Rankflow, leveraging large‑model inference, decoupled recall, fine‑grained parallelism, and elastic compute allocation to boost performance, flexibility, and maintainability while paving the way for future retrieval‑augmented generation integration.
This article presents a concise, step‑by‑step derivation of FlashAttention, explaining the prerequisite linear‑algebra concepts, the softmax simplifications, and the parallel computation workflow—including the LSE‑enhanced version—so readers can grasp the algorithm’s elegance without heavy mathematics.
This article explains the Fork/Join model and Java's ForkJoinPool, covering divide‑and‑conquer theory, task splitting, core APIs, code examples, common pitfalls, performance testing, and best‑practice recommendations for high‑concurrency computing.
Tencent Cloud's TDSQL shattered world records in both TPC‑DS (OLAP) and TPC‑C (OLTP) benchmarks, achieving a 7260 M QphDS score at a cost of 37.52 CNY/kQphDS, and the article explains the three self‑developed technologies—MPP execution, parallel execution framework, and columnar‑vectorized engine—that made this performance possible.
This article examines the challenges of scaling large AI models across multiple GPUs, explores data, pipeline, and tensor parallelism, analyzes collective communication patterns and data‑channel technologies such as PCIe, NVLink and RDMA, and offers concrete optimization recommendations to boost training efficiency.
The article examines Go's coroutine architecture and its GMP (Goroutine-Machine-Processor) model, tracing coroutine history, comparing kernel, user, and hybrid thread models, and detailing G, M, and P components, scheduling principles, work-stealing, preemption, and runtime implementation that give Go high concurrency and low latency.
The article explains OpenAI's 2020 Scaling Laws that show larger transformer models, more data, and greater compute consistently improve performance, introduces the concept of emergent abilities at critical size thresholds, and outlines the core principles of parallel computing such as multi‑processor usage, task decomposition, concurrent execution, and inter‑processor communication.
This article explains what a GPU is, how it differs from a CPU, its internal architecture, and why its massive parallel processing makes it essential for graphics rendering, scientific computation, and AI inference, illustrated with examples such as NVIDIA RTX 3090.
This article explains the pivotal role of GPUs in today’s generative AI era, describes their architecture and applications, compares them with CPUs, ASICs, and FPGAs, and offers guidance on selecting the right processor for AI workloads while also noting related reference resources.
This article examines multithreaded architecture, explaining its core principles, highlighting efficiency and resource utilization benefits, discussing synchronization, communication, and management challenges along with mitigation strategies, and exploring its future impact on cloud computing, big data, AI, and overall system performance.
This article introduces Python's multiprocessing module and the Celery task queue, explains their core concepts, and provides practical code examples for multi‑process parallel computation, inter‑process communication, asynchronous execution, scheduled jobs, result callbacks, retries, and distributed task orchestration.
This article introduces Python's multiprocessing module and the Celery distributed task queue, explains their core features, and provides ten practical code examples demonstrating multi‑process parallel computation, inter‑process communication, asynchronous tasks, scheduling, retries, and distributed processing for real‑world applications.
This article demonstrates how to use Python's concurrent.futures module to parallelize image resizing, turning a single‑process script into a multi‑core solution with just three additional lines of code, achieving up to a six‑fold speed‑up on typical CPUs.
This article explains what a GPU (Graphics Processing Unit) is, how it differs from a CPU, its architecture and performance characteristics, and why it powers everything from real‑time rendering to AI inference, using examples like the NVIDIA RTX 3090.
Baidu’s AI infrastructure combines a massive InfiniBand‑linked GPU cluster, Kunlun chips, the PaddlePaddle framework, and the Wenxin model suite with 4D hybrid parallelism, elastic fault tolerance, and a two‑stage training pipeline to overcome computation, memory, and communication walls, delivering world‑leading MLPerf performance for large‑scale LLMs.
This article explains Python's multiprocessing module, shows how to create and manage processes and process pools with clear code samples, and outlines common use cases, best practices, and pitfalls for efficiently handling CPU‑bound tasks.
This article provides a comprehensive overview of high‑performance computing, covering system architectures, hardware components, performance metrics, network topologies, common parallel file systems, cluster management functions, mainstream job‑scheduling systems, and MPI‑based parallel programming models.
This article details Du Xiaoman's development of the ATLAS automated machine learning platform, covering business scenarios, AI algorithm deployment challenges, the end‑to‑end production workflow, platform components such as annotation, data, training and deployment, as well as optimization techniques like AutoML, meta‑learning, NAS, and large‑scale parallelism, concluding with lessons learned and future directions.
The guide defines backend coding standards and best practices, specifying clear naming conventions for classes, methods, variables and constants, enforcing consistent code style via .editorconfig, recommending unchecked exceptions for simpler error handling, promoting structured asynchronous logging with trace IDs, and outlining performance optimizations such as loop refinement, CompletableFuture concurrency, and proper resource management to prevent memory leaks.
The article explains how massive parallel pixel calculations demand GPUs, whose high cost and inflexibility are solved by Tencent Cloud’s elastic, virtualized GPU services—including vGPU, qGPU, TACO abstraction, and spot instances—delivering up to 16 EFLOPS for AI, scientific, graphics, and video workloads.
Numba is a Python just‑in‑time compiler that transforms functions into fast native machine code, enabling near C‑level performance without rewriting code; by adding simple decorators like @jit or @njit, you can accelerate loops, NumPy operations, and even leverage parallel or GPU execution.
This article explains the concept and challenges of data lakes, describes the “impossible triangle” of storage, compute, and cost, and introduces the open‑source SPL engine that provides multi‑source, file‑based, high‑performance computing to overcome those limitations.
This article provides a comprehensive overview of high‑performance computing (HPC), covering its definition, hardware architectures, performance metrics, cluster components, parallel file systems, management and scheduling tools, as well as common MPI implementations and links to further technical resources.
The article narrates a CPU's journey from a naïve element‑wise increment loop to the adoption of SIMD, MMX, SSE, and AVX instruction sets, illustrating the motivations, challenges, and architectural decisions behind parallelizing integer and floating‑point operations.
This article explains how to overcome Python's Global Interpreter Lock by installing ipyparallel, configuring parallel profiles, and using engines, DirectView, and LoadBalancedView to run both synchronous and asynchronous tasks, with code examples and performance comparisons.
This article explains how computer systems combine hardware and system software, describes the memory hierarchy, operating‑system abstractions, Amdahl's law, and various forms of parallelism, and shows why mastering these fundamentals can dramatically improve program efficiency and reliability.
The article explains high‑performance computing (HPC) concepts, including serial and parallel processing, supercomputer performance measured in FLOPS, real‑world scientific applications such as drug discovery and weather forecasting, and the hardware architectures that enable these massive computational capabilities.
The article presents a domain‑specific language that mimics Excel formulas, a stack‑based parser and recursive engine for evaluating calculations, and a multi‑layer architecture—including a dynamic priority scheduler—to efficiently resolve field dependencies, improve maintainability, and enable monitoring across large data systems.
This article walks through implementing a lightweight single‑machine MapReduce framework inspired by the original MapReduce paper, covering the abstract Map/Reduce model, task scheduling between master and workers, core Go code for map, reduce, worker, and coordinator, and a brief reflection on its limitations.
This article explains how IPython and Jupyter notebooks support multi‑language execution, integrate Fortran via F2PY, and enable parallel and distributed computing with ipyparallel, illustrating practical magic commands, cluster setup, and performance considerations for scientific Python workflows.
This article traces the historical evolution of computers from first‑generation vacuum‑tube machines to modern VLSI systems, explains fundamental performance limits such as the von Neumann bottleneck and power consumption, and introduces parallel optimization concepts like Amdahl’s Law and future computing paradigms.
This article presents a series of Java Stream API performance experiments—comparing serial and parallel streams on primitive, object, and reduction tasks—showing that while serial streams are slower than traditional loops for simple operations, parallel streams can significantly outperform both in multi‑core environments.
This article explains what supercomputers are, outlines major ranking systems such as TOP500 and Green500, describes their wide‑range research and application value, and details the three‑layer architecture, parallel computing principles, and emerging trends like exascale, AI integration, quantum and bio‑computing.
This article provides a concise overview of GPU computing fundamentals, covering GPU hardware components, memory hierarchy, parallel execution models, and the CUDA programming framework, illustrating how CPUs and GPUs cooperate in heterogeneous computing environments.
This tutorial demonstrates how to initialize a SparkContext in PySpark, perform simple parallel computations such as temperature conversion and reduction, create a SparkSession to read CSV data, and apply common DataFrame operations like selecting columns, adding new columns, filtering, grouping, and aggregating.
This article explains how the Mars framework enables parallel and distributed execution of core Python data‑science libraries—Numpy, Pandas, and Scikit‑Learn—while integrating with RAPIDS for GPU acceleration, and demonstrates its performance advantages through code examples and benchmark results.
The article explains the concepts, principles, advantages, and key differences between parallel computing and distributed computing, highlighting their roles within high‑performance computing and when each approach is most appropriate.
The talk explores PostgreSQL’s unique community-driven development, its commercial and innovative capabilities, new version features such as enhanced partitioning and parallel computing, and how its open‑source licensing, extensible architecture, and cloud integrations make it a compelling, enterprise‑ready database solution.
This article explains the history, architecture, and operation of GPUs, and introduces major acceleration frameworks such as CUDA and OpenCL, highlighting their roles in parallel computing and modern graphics processing for scientific and AI workloads.
The article explains how heterogeneous parallel computing—distributing tasks across CPUs, GPUs, FPGAs and other accelerators—has become essential after Moore’s law plateau, detailing its principles, hardware and software perspectives, classification of architectures, processing stages, user‑guided versus compiler‑guided methods, and its relevance to AI, cloud and industry workloads.
This article explains the history and origin of GPUs, compares CPU and GPU architectures, describes the GPU processing pipeline, and introduces acceleration technologies such as CUDA and OpenCL, highlighting their programming models, supported languages, and key performance metrics.
The article explains the latest TOP500 supercomputer rankings, emphasizes that architecture—not just CPU count—is the core technology behind high‑performance computing, describes the challenges of networking, software, and power, and illustrates diverse applications such as nuclear simulation, climate forecasting, and video rendering.
This article explains the fundamental differences between CPUs and GPUs, outlines key GPU specifications such as CUDA cores, memory capacity, bandwidth, and floating‑point precision, and reviews NVIDIA's major GPU series and architectural evolution for high‑performance and AI workloads.
This article traces the evolution of GPUs from early graphics chips to modern parallel processors, explains their internal pipeline, compares CPU and GPU architectures, and introduces key acceleration frameworks like CUDA and OpenCL for general‑purpose computing.
This article explains how to speed up TensorFlow deep‑learning model training using single‑GPU acceleration, multi‑GPU parallelism, and distributed TensorFlow on Kubernetes, covering device placement, session parameters, synchronous vs asynchronous training modes, and practical code examples to improve performance and scalability.
Meituan built a custom deep‑learning platform that combines data‑parallel and hybrid parallelism across multi‑GPU/cluster hardware, uses coarse‑grained scheduling and Kaldi‑derived acoustic algorithms, and supports fast NLU model hot‑updates, achieving near‑linear GPU scaling and 6–7× speedups over traditional solutions.
This article explains the concepts, principles, and key distinctions between parallel computing and distributed computing, describing their objectives, basic conditions, advantages, and typical use cases within high‑performance computing, and highlights how they differ from grid and cloud computing.
Meituan’s in‑store dining team demonstrates how R’s open‑source packages, powerful data manipulation, rich visualization libraries, and reproducible reporting can be engineered into scalable, parallelized workflows that turn secondary data processing into fast, interactive dashboards and analytics, proving R’s enterprise‑grade performance and adoption.
This article uses a human‑computer analogy and a playing‑card counting example to explain the fundamentals of distributed computing, why single machines cannot handle massive data, and how the MapReduce model’s four steps—split, transform, shuffle, and merge—solve big‑data problems.
Learn how to dramatically improve Python performance by choosing optimal data structures, minimizing redundant data, using copy wisely, leveraging dict/set lookups, generators, efficient loops, string joining, proper formatting, fast variable swapping, concise comparisons, C extensions, multiprocessing, PyPy, and profiling tools, all backed by real benchmarks.
The Metric Logic Tree automates business metric anomaly analysis by integrating heterogeneous data sources (Kylin, MySQL, Elasticsearch, Druid) with a three‑layer architecture—metric calculation, algorithmic analysis (waterfall and Gini‑coefficient methods), and a master‑worker computation service—that parallelizes queries, delivers immediate conclusions, and shortens decision cycles, as demonstrated in Meituan‑Dianping’s hotel‑travel operations.
This article explains how Alibaba's Cainiao network built the STARK VRP engine—an AI‑driven, distributed vehicle‑routing solver that supports dozens of VRP variants, leverages metaheuristics, parallel island models, and deep reinforcement learning to dramatically reduce fleet size and travel distance in Chinese logistics.
Exploring how a securities firm processes billions of daily stock‑market indicators in real time, this article compares an in‑process Redis + Lua solution with an out‑of‑process Goroutine‑based architecture, detailing data flow, performance trade‑offs, and scalability considerations for high‑frequency time‑series workloads.
The article argues that, despite the hype around ever‑changing programming languages, mastering core algorithms and computer science theory remains essential for building efficient, scalable solutions across fields—from search engines and parallel computing to scientific research—because algorithms are the enduring foundation of technology.
This article introduces the MapReduce parallel processing model, illustrates its core map and reduce operations with a library‑shelf analogy and a classic word‑count example, and walks through each processing stage using clear diagrams to show how massive data is aggregated efficiently.
Former professor and hedge‑fund founder D.E. Shaw leveraged his expertise in massive parallel computing to create the Anton supercomputer, a purpose‑built machine that accelerates molecular dynamics simulations thousands of times faster than traditional supercomputers, reshaping computational chemistry and high‑frequency trading.
