OpenAI’s latest Codex update adds locked‑screen system control and lets any device with the client join a distributed network, enabling users to repurpose idle computers as a coordinated AI compute fleet while raising security and platform‑compatibility questions.
The article details MaxCompute's AI‑driven evolution, covering its multilayer architecture, multimodal storage management, SQL AI functions, the Python‑based MaxFrame framework, and several industry case studies that demonstrate performance gains and flexible resource scheduling for large‑scale AI workloads.
This article details a step‑by‑step, distributed pipeline built on Alibaba Cloud PAI using Data‑Juicer and Ray that transforms raw egocentric hand videos into LeRobot v2.0‑compatible Vision‑Language‑Action (VLA) training data, covering video splitting, frame extraction, camera calibration, 3D hand reconstruction, pose estimation, action captioning, and export, with code snippets, performance numbers, and references.
When attribution analysis on over 900 million rows slowed to more than 40 seconds and threatened cluster stability, Ctrip's smart attribution team rebuilt the architecture with Ray and DuckDB, achieving sub‑15‑second query times, 160 % performance gain, and complete resource isolation.
This guide reviews eight leading AI development tools for Java in 2025, explaining how each library or framework—such as DJL, TensorFlow Java, Hugging Face, LangChain, Apache Kafka, Ray, Deeplearning4j, and Neo4j—enables Java developers to build, train, and deploy intelligent applications without switching languages.
This article details Ant Group's use of the Ray distributed computing framework to accelerate massive data indexing, migrate a C++ engine to Ray, implement elastic resource scheduling, improve long‑tail task efficiency, and build a robust RAG operator system with comprehensive governance, achieving up to 2× speed gains and 99.9% success rates.
This article explains Ray Data’s execution pipeline, detailing the LogicalOptimizer’s architecture, core abstractions, rule‑based optimization process, and both logical and physical rule sets, with concrete code examples and practical illustrations of each optimization technique.
This article examines the shortcomings of traditional big‑data engines for AI workloads, presents a Ray‑based heterogeneous fusion architecture that unifies CPU/GPU scheduling, Python ecosystems, and streaming‑batch processing, and details fault‑tolerance, checkpointing, compute‑storage separation, resource‑utilization, scalability, and observability improvements that enable thousands of nodes and dramatically higher GPU efficiency.
This article explains how Apache Ignite's memory‑first architecture, distributed compute grid, and event‑driven streaming enable sub‑100 ms decision making, high throughput, and strong consistency for real‑time bidding platforms, with practical code examples, Spring Boot integration, monitoring tips, and security considerations.
This article details a scalable, distributed pipeline that uses LAS AI Data Lake, Daft on Ray, and advanced video‑processing techniques—scene detection, splitting, frame sampling, filtering, and caption generation—to transform tens of millions of hours of robot‑captured video into high‑quality, searchable semantic data while dramatically boosting CPU and GPU utilization.
This article analyzes the technical foundations, development frameworks, real‑time inference optimizations, typical industry deployments, and future research directions of multi‑agent systems in 2025, highlighting protocols like FIPA‑ACL and MCP, tools such as LangGraph and ADP3.0, and edge‑computing breakthroughs.
This guide explains Ray's fundamental building blocks—including Tasks, Actors, remote Objects, Placement Groups, and environment dependencies—showing how to define, schedule, and retrieve distributed workloads with code examples and command‑line utilities.
The article examines the emerging AI Super Node ecosystem, detailing its core concepts, four‑layer architecture, key enabling technologies, current challenges such as compatibility and energy consumption, and future directions like quantum‑classic hybrids and green low‑carbon designs, illustrating how it overcomes scaling bottlenecks in modern AI deployments.
This article examines how WeChat’s Astra platform leverages the Ray distributed framework to manage million‑node AI workloads, addressing challenges of scale, heterogeneous GPU resources, operational complexity, and cost, and outlines the architecture that unifies Ray services across multiple Kubernetes clusters.
This article examines how Tencent's WeChat team leveraged the Ray distributed computing framework within the Astra platform to tackle massive AI workloads, addressing challenges of scale, GPU diversity, operational complexity, and cost while outlining their architecture and practical insights.
This article details how Pinterest’s senior staff engineer Dr. Luo leveraged the open‑source Ray framework to overcome LLM data‑preprocessing bottlenecks, describing the system’s architecture, key features such as map_batches, Carry‑Over Columns and Accumulators, and the dramatic performance and cost improvements achieved.
This article examines how Tencent leverages the Ray distributed framework within the Astra platform to handle WeChat's massive AI workloads, addressing challenges of scale, heterogeneous GPU resources, operational complexity, and cost while outlining the architecture and practical benefits.
This article explains how Ant Group’s Ragent framework leverages Ray to create scalable, multi‑tenant AI agents, detailing its background, motivation, and design while outlining the core modules—Profile, Memory, Planning, and Action—that power large‑language‑model agents.
This article explores how the LAS team's AI‑driven data lake solution, built on Daft for lake computing and Lance for lake storage, tackles the emerging challenges of multimodal data handling, offering faster I/O, heterogeneous CPU‑GPU scheduling, and seamless integration for AI workloads.
This article explains how Tencent's WeChat team built the Astra platform on Ray to manage millions of AI compute nodes, addressing challenges of massive scale, heterogeneous GPU resources, low‑priority node instability, deployment complexity, and cost, while detailing architecture, scheduling strategies, and practical usage examples.
This article outlines MaxCompute's decade‑long evolution—from the early PyODPS SDK to the native Distributed Python Engine—highlights the challenges big‑data platforms face in the AI era, and showcases Data+AI solutions and real‑world case studies across multimodal processing, massive text deduplication, and autonomous‑driving data pipelines.
This article breaks down the hardware bottlenecks of large‑scale LLM training, explains the Roofline performance model, arithmetic intensity, and how computation and communication costs interact on GPUs and TPUs, offering concrete formulas and examples for efficient scaling.
In the era of data‑driven AI, Ray offers an open‑source unified compute framework that abstracts distributed system complexity, enabling developers to seamlessly scale Python code from a laptop to large GPU clusters, and provides the Ray AI Runtime (AIR) with libraries such as Ray Data, Train, Tune, and Serve to accelerate LLM training, hyper‑parameter tuning, and model serving.
This article explores how Apache SeaTunnel addresses modern data integration challenges by providing a high‑performance, distributed, plugin‑based platform that decouples connectors from execution engines, enabling seamless batch and streaming synchronization across heterogeneous sources such as databases, message queues, and data lakes.
This article explains EasyRec's end‑to‑end recommendation system, covering its training‑inference architecture, a series of CPU/GPU and distributed optimizations, and a real‑time online‑learning pipeline that together improve throughput, latency, and model freshness.
This deep‑dive explains how Ray Data transforms user‑level Dataset APIs into a logical plan, optimizes it, converts it into a physical streaming execution graph, and runs it on a cluster using task and actor pools, detailing each component from read sources to write sinks with code examples.
This guide outlines a comprehensive Hadoop learning roadmap, covering essential prerequisites, core concepts such as HDFS, MapReduce, and YARN, hands‑on projects, advanced ecosystem tools like Hive, Pig, HBase and Spark, plus curated resources and community channels for aspiring big‑data engineers.
This article explains how to build a flexible machine‑learning infrastructure on Alibaba Cloud ACK using Ray and KubeRay, covering Ray's core components, AI libraries, deployment options on VMs and Kubernetes, code examples for data processing, model serving, and advanced scheduling and quota management techniques.
This article, based on a meetup presentation, details Alibaba Cloud's MaxCompute platform—its evolution, serverless architecture, AI integration, distributed Python framework, Object Table, near‑real‑time processing, and intelligent warehouse features—addressing the challenges of data warehouses in the Data+AI era.
This article introduces MaxFrame, a cloud‑native distributed Python compute service built on MaxCompute, detailing its architecture, seamless integration with the Python ecosystem, and real‑world use cases ranging from large‑scale data analysis and machine learning to offline LLM inference and custom image deployments.
This article introduces DeepSeek Smallpond, a lightweight yet high‑performance AI data‑processing engine built on Ray and DuckDB, explains its dual Dataframe and LogicalPlan APIs, showcases integration with Volcano Engine's AI Data Lake LAS, and provides practical code examples for distributed processing, multimodal storage, and RAG pipelines.
The article analyzes how the rapid rise of open‑source large‑model AI in 2025 is reshaping the data development profession, urging developers to transition from specialized data‑engineer roles to full‑stack AI data engineering skills such as distributed computing, lake‑house architectures, and model tuning.
This article summarizes a meetup talk by Alibaba Cloud expert Yu Deshui, detailing MaxCompute’s evolution, serverless architecture, AI‑enabled features, and the platform’s comprehensive solutions—including OpenLake, MaxFrame, Object Table, near‑real‑time computing, and AI Functions—to address the challenges of modern data‑centric AI workloads.
This article explains how ClickHouse’s columnar storage, compression, and bitmap functions enable fast, scalable processing of billions of user tags and groups in a CDP, covering data storage design, bitmap generation, and distributed query optimization.
The article details Bilibili's comprehensive redesign of its tag system—including background challenges, architectural layers, technical upgrades like Iceberg integration and shard‑based ClickHouse writes, crowd selection methods, online service guarantees, performance metrics, and future plans—showcasing a data‑driven solution that boosts stability, speed, and business coverage.
This article introduces Flex, a Flink‑compatible stream‑batch vectorized engine built on Velox and Gluten, explains the SIMD‑based execution model, details native operator optimizations, fallback mechanisms, correctness and usability improvements, and presents performance results and future development plans.
The article details ByteDance's use of Ray and RayData to construct scalable audio and video data processing pipelines for multimodal AI models, addressing challenges of massive data volume, resource constraints, and fault tolerance through pipeline design, RayCore enhancements, and custom scheduling optimizations.
As AI accelerates, large‑model compute infrastructure becomes a cornerstone of digital transformation, with specialized accelerators, heterogeneous architectures, massive distributed clusters, intelligent scheduling, soaring costs, energy concerns, software‑hardware co‑design challenges, and data‑privacy issues shaping its future development.
A former Nvidia engineer, working through the GIMPS distributed project and leveraging thousands of GPUs across dozens of data centers, confirmed that 2^136279841−1—a 41,024,320‑digit Mersenne prime—is the largest known prime ever found, surpassing the previous record by over 1.6 million digits.
This article details the AI computing challenges faced by WeChat, explains why the Ray distributed engine was chosen, and describes the design and large‑scale deployment of the AstraRay platform—including scheduling, resource management, and multi‑model support—to achieve low‑cost, high‑efficiency AI services.
The article details the design and deployment of 360's AI Compute Center, covering GPU server selection, high‑performance networking, Kubernetes‑based cluster management, advanced scheduling, training and inference acceleration techniques, and a comprehensive AI development platform with visualization and fault‑tolerance features.
This article examines Unity's cloud rendering technology, detailing its distributed architecture, workflow steps, key innovations such as low‑latency transmission and real‑time rendering, and explores how these capabilities enable large‑scale, immersive experiences in the emerging metaverse.
This article provides a comprehensive overview of Apache Spark SQL, covering its fundamental concepts such as TreeNode, AST, and QueryPlan, the distinction between logical and physical plans, the rule‑execution framework, core components like SparkSqlParser and Analyzer, as well as the Spark Session, Dataset/DataFrame, and various writer APIs, supplemented by a detailed Q&A session.
This article introduces the fundamentals of distributed computing, compares major frameworks such as Hadoop, Spark, and Flink, and outlines their key components, performance characteristics, and typical application scenarios including big‑data analytics, cloud services, real‑time streaming, and scientific computing.
This article provides a comprehensive overview of Apache Spark, covering its origins, key characteristics, core concepts such as RDD, DAG, partitioning and dependencies, the internal architecture including SparkConf, SparkContext, SparkEnv, storage and scheduling systems, as well as deployment models and the company behind the product.
This article explains a distributed‑computing approach for generating a globally unique integer index from massive string datasets, replacing single‑reducer sorting with hash‑bucket partitioning and parallel processing to cut runtime from 30 minutes to just 2 minutes.
Tencent’s Game Social Algorithm Team presents a Spark‑based distributed network embedding framework that recursively partitions hundred‑billion‑edge game graphs into manageable subgraphs, runs node2vec locally, and fuses results, enabling efficient link prediction and node classification across multiple games within hours.
The rapid growth of AI model sizes has created unprecedented demands on network bandwidth, latency, stability, and automation, making efficient RDMA‑based interconnects, advanced congestion control, and intelligent deployment essential for scaling distributed training clusters to thousands of GPUs.
The article describes how JD’s advertising team tackled the high‑concurrency, low‑latency challenges of online recommendation inference by designing a distributed graph heterogeneous computing framework, optimizing GPU kernel launches with TensorBatch, deep‑learning compiler techniques, and a multi‑stream GPU architecture, achieving significant throughput and latency improvements.
This article details JD Retail's ad‑tech team's deep‑compute optimizations—including a distributed graph‑based heterogeneous framework, GPU‑focused inference engine enhancements, TensorBatch request aggregation, deep‑learning compiler bucket pre‑compilation, asynchronous compilation, and multi‑stream GPU processing—to overcome high‑concurrency, low‑latency online recommendation challenges.
The article introduces PAI‑ChatLearn, a flexible and high‑performance framework developed by Alibaba Cloud's PAI team that supports full‑pipeline RLHF training for large models, explains the evolution of parallel training strategies, details the framework’s architecture and configuration, and showcases performance results and practical usage examples.
This article describes how JD Retail's advertising technology team tackled the high‑compute demands of modern recommendation models by designing a distributed graph‑partitioned heterogeneous computing framework, introducing TensorBatch request aggregation, leveraging deep‑learning compiler bucketing and asynchronous compilation, and implementing a multi‑stream GPU architecture to dramatically improve online inference throughput and latency.
The article outlines the accelerating demand for high‑performance computing driven by AI, AR/VR, biotech and other workloads, examines the limits of Moore's law, and presents emerging solutions such as advanced chip architectures, chiplet integration, near‑memory/in‑memory computing, and distributed xPU‑based systems for scalable, efficient compute.
This article explains why Ray is popular among AI researchers, how ByteDance uses KubeRay to host Ray applications, and how Kueue manages and schedules RayJob workloads, covering Ray's architecture, KubeRay components, real-world use cases, and job scheduling strategies.
This article explains how to use Celery together with Redis to manage and execute periodic and asynchronous tasks in Python, covering basic concepts, architecture, configuration steps, single‑worker and multi‑worker setups, distributed processing strategies, and practical considerations for reliable task execution.
This article explains core concepts and practical solutions for handling massive datasets that cannot fit into memory, covering batch processing, distributed sorting, bitmap indexing, hash‑based lookups, top‑K extraction, and deduplication techniques with code examples and multi‑machine strategies.
Baidu's search vertical offline computing system evolved through four stages—from a fragmented pre‑2018 processing setup to a unified business framework, then serverless functions, and finally a data‑intelligent architecture with multi‑layer abstraction, graph and multi‑language engines, achieving 5‑10× efficiency gains and dramatically reducing failures.
This article explores how Tencent integrates big data with cloud‑native technologies, detailing the evolution, opportunities, challenges, the peak‑range (FengLuan) architecture, engine and scheduling layers, mixed‑workload strategies, runtime optimizations, and future directions for large‑scale data platforms.
This article explains how the Ray distributed computing engine simplifies the design, deployment, and operation of complex cloud‑native distributed systems—illustrated through an AutoML service example—by detailing system complexity, Ray’s core concepts, resource customization, runtime environments, monitoring, and ecosystem integrations.
This article discusses the challenges of loop detection in financial risk control, presents distributed graph computing optimization techniques—including pruning, multi‑graph handling, and memory‑efficient algorithms—shows experimental results, and shares real‑world ArcNeural engine case studies and future directions.
Fanglue is an interactive, web‑based rule‑development platform that integrates expert domain knowledge with distributed AI algorithms to efficiently generate and evaluate decision rules for anti‑fraud scenarios, leveraging Ray for real‑time processing and achieving VLDB‑2023 acceptance.
This article examines common performance issues in big data SQL queries, such as data skew and data explosion, and provides systematic troubleshooting steps and practical optimization techniques across the Map, Reduce, and Join stages, including partition merging, column pruning, predicate pushdown, and join strategies.
This article explains the challenges of large‑model offline (batch) inference, such as GPU memory limits and distributed scheduling, and shows how Ray’s cloud‑native architecture, model partitioning, and Ray Datasets can be used to build efficient, elastic inference frameworks deployed with KubeRay.
This article presents a comprehensive interview with several big‑data platform experts, outlining the core components such as data integration, storage and computation, distributed scheduling, and query analysis, while also highlighting current challenges, best‑practice tools, and future trends in big‑data architecture.
Tencent’s Star Network delivers a 1.6 Tbps Ethernet‑RDMA fabric, fat‑tree topology supporting up to 4 K GPUs, multi‑track traffic aggregation and adaptive heterogeneous links plus a custom TCCL library, cutting AllReduce overhead from 35 % to 3.7 %, speeding AI training iterations by 32 % while automating deployment and providing sub‑second self‑healing.
This article introduces parallel programming concepts, memory architectures, execution models, Python threading versus multiprocessing performance, and reviews several Python parallel libraries such as Ray, Dask, Dispy, ipyparallel, and Joblib for building scalable concurrent applications.
MPP (Massive Parallel Processing) databases, designed for large‑scale analytical workloads, use distributed, shared‑nothing architectures with multiple control and compute nodes, offering high scalability, diverse data‑sharding strategies, and powerful SQL compatibility, as illustrated by vendors like Teradata, Vertica, Greenplum, and emerging open‑source solutions.
The article surveys unconventional offline data‑task optimizations—such as distribution‑by, seeded random shuffling, explode‑based skew mitigation, hash bucketing, task‑parallelism tuning, and multi‑insert materialization—organized by point, line, and surface perspectives, and stresses that effective performance gains require both technical tricks and business‑driven pipeline adjustments.
PGLBox, Baidu’s GPU‑based large‑scale graph training framework, delivers up to 27× speedup over CPU clusters by fully GPU‑accelerating storage, sampling, and training, supporting billions of nodes, advanced GNN algorithms, multi‑level storage, and seamless integration of massive pretrained models.
Distributed computing splits massive tasks across multiple servers, and this article explains the classic MapReduce batch engine and the modern Spark framework, covering their architectures, strengths, limitations, and evolution, while highlighting key features like fault tolerance, in‑memory processing, and real‑time streaming capabilities.
While distributed clusters are popular for big‑data processing, they are not a universal solution; tasks that are hard to partition or involve heavy cross‑node communication often perform better on a well‑optimized single machine, making a careful analysis of workload characteristics essential before scaling out.
An expert interview series examines the architecture of big data platforms, detailing core modules such as data integration, storage, computation, scheduling, and query analysis, while highlighting current challenges, best‑practice tools, and future trends like cloud‑native, object storage, and real‑time processing.
This article presents an end‑to‑end privacy‑preserving machine‑learning solution for AI and big‑data workloads built on Intel SGX, the open‑source TEE OS Occlum, and BigDL PPML, detailing its architecture, key features, deployment steps, and real‑world applications.
An extensive survey ranks and compares a wide range of machine learning libraries and frameworks—both deep and shallow learning—detailing their languages, types, GPU acceleration, distributed computing capabilities, and typical academic and industrial applications, based on Google search popularity as of May.
This article introduces causal inference methodologies tailored for game scenarios, discusses the challenges of offline inference on massive data, and presents three distributed solutions—low‑complexity propensity‑score matching, robust double‑robust estimation, and panel difference‑in‑differences—along with their implementation details and performance insights.
This article provides a comprehensive overview of Hadoop, covering its historical development, key characteristics, the HDFS storage framework, the MapReduce processing engine, YARN resource manager, and a wide range of real-world application scenarios, as well as the broader Hadoop ecosystem and its major components.
This article provides a comprehensive overview of Hadoop, covering its history, core features, the HDFS storage framework, MapReduce computation engine, YARN resource manager, real‑world application scenarios, and the surrounding ecosystem of tools such as Hive, Spark and Kafka.
This article introduces GraphLearn, a large‑scale distributed graph learning platform designed for industrial GNN workloads, details its architecture, sampling implementation, training pipeline, system optimizations such as GPU‑accelerated sampling, and showcases real‑world applications in recommendation and risk control.
The Huawei Cloud HCDE summit highlighted the rise of distributed cloud native, detailing global scheduling, SaaS tenant isolation, AI creation tool workflows, and the integration of Kubernetes and Dapr to address cross‑cloud, cross‑region application challenges.
This article explains the fundamentals of graph computing, compares it with traditional processing, outlines industry challenges such as partitioning and load imbalance, and details HugeGraph’s self‑developed architecture, key technical solutions, and how developers can create and deploy graph algorithms.
The article describes JD Search's machine‑learning alert system that combines offline and real‑time training, FFT‑based periodic detection, Prophet forecasting, and DBSCAN anomaly clustering, and explains architectural design, data preprocessing, model optimization, and distributed deployment to improve alert accuracy and response speed.
The article introduces SuperSQL, a federated SQL middleware that unifies heterogeneous data sources across multiple data centers, leverages Apache Calcite for cost‑based optimization, pushes down operators to various engines, manages metadata with a Trie model, and demonstrates significant performance gains over traditional solutions.
The article analyzes how the AI era’s demand for ultra‑low latency and high throughput exposes fundamental limits of the traditional TCP/IP stack, and explains why RDMA’s kernel‑bypass, zero‑copy design, and emerging congestion‑control algorithms are becoming the preferred network fabric for modern data‑center workloads.
This article explains how to overcome Python's Global Interpreter Lock by installing ipyparallel, configuring parallel profiles, and using engines, DirectView, and LoadBalancedView to run both synchronous and asynchronous tasks, with code examples and performance comparisons.
This article explains JD’s self‑developed data‑skew mitigation solution for Spark, detailing the problem of uneven key distribution, the limitations of the open‑source AQE implementation, and JD’s OptimizeSkewedJoinV2 algorithm that dramatically reduces stage latency in large‑scale join workloads.
The article details the design, architecture, core features, scheduling logic, and performance gains of Vipshop’s self‑built Hera data service, which unifies data‑warehouse access, supports multiple engines, adapts SQL execution, and dramatically improves SLA for both B‑to‑B and B‑to‑C workloads.
This article explains the fundamentals of Remote Procedure Call (RPC), its historical evolution, core concepts such as remote procedures and calls, practical examples, analogies, and the challenges it introduces—like latency, address‑space isolation, partial failures, and concurrency—while highlighting RPC’s pivotal role in enabling scalable distributed architectures.
iQIYI’s unified big‑data + AI platform combines a hybrid‑cloud model, storage‑compute separation via its QBFS virtual file system, a reusable feature‑store and operator DAGs, and multi‑tenant YARN scheduling to overcome legacy Hive/Spark bottlenecks, accelerate large‑scale model training, improve data quality, and prepare for future real‑time, privacy‑preserving AI workloads.
Gorse is an open‑source Go recommendation system that automates model selection, supports distributed training and prediction, offers a RESTful API and dashboard, and stores data in MySQL/MongoDB with Redis caching, enabling fast integration of personalized suggestions into online services.
FAE, Alibaba’s high‑performance distributed MPP engine, stores billions of user‑behavior events in a time‑ordered AFile model and uses stateless masters, importers, mergers and workers with Redis and MySQL metadata to deliver sub‑second, 10‑100× faster ad‑attribution queries across ad‑hoc, offline and near‑real‑time scenarios such as frequency, path and funnel analysis.
The article introduces Alibaba's Fast Attribution Engine (FAE), describing the technical challenges of user behavior and advertising attribution, its data model (AFile), system architecture, performance advantages over traditional OLAP solutions, and a range of application scenarios such as frequency analysis, crowd flow modeling, path, retention, funnel analysis, and visitor selection.
The article details the background, architecture, core features, scheduling mechanisms, Lisp‑based query DSL, and Alluxio integration of Vipshop's self‑developed Hera data service, illustrating how it unifies multi‑engine data access, improves SLA, and accelerates large‑scale crowd computing tasks.
This article explains the growing analytical demands in the big‑data era, the limitations of traditional OLAP, and how Kyligence’s distributed OLAP engine addresses data‑island issues, multi‑dimensional and many‑to‑many analysis, unified security, and performance optimization with MDX on Spark, delivering a seamless Excel‑like experience.
This article introduces the third edition of Stanford’s Data Mining Tutorial, highlighting its panoramic roadmap of data‑mining techniques for massive datasets, core features, comprehensive topic coverage, target audience, and supplementary resources while noting its popularity among students and professionals.
This article explains the evolution of cloud computing, the distinction between serial, parallel, and distributed computing models, and details the four classic computer architecture classifications (SISD, SIMD, MISD, MIMD) along with shared‑memory and distributed‑memory MIMD systems and their role in modern distributed system layers.
This article presents a detailed overview and ranking of numerous machine learning tools and libraries, distinguishing deep and shallow learning approaches, highlighting language support, GPU acceleration, and distributed computing capabilities, and provides insights into their academic and industrial usage.
This article systematically introduces NetEase Kyuubi, an open‑source high‑performance JDBC and SQL execution engine built on Apache Spark, covering its background, core architecture, service discovery, session and operation management, startup processes, and key source‑code implementations with detailed code examples.
Jet, an open‑source in‑memory distributed batch and stream processing engine, introduces dynamic ClassDefinition for SQL, JDBC sink batch size option, SQL aggregation memory optimizations, improved sliding‑window pickAny performance, numerous extension updates, critical bug fixes, and a major API change to DAG.toDotString for enhanced parallelism control.
This article details the design of a red packet calculation and distribution system for a Spring Festival promotion, addressing mixed-type packet computation, seamless transition to awarding, distributed task processing, stability guarantees, and successful implementation results.
This article explains SparkSQL's overall workflow, introduces the basic elements of joins, and details the physical execution processes for various join types—including sort‑merge, broadcast, and hash joins—along with their implementation conditions and optimization considerations.
