AlphaPROBE reformulates alpha‑factor discovery as a strategy‑navigation problem on a directed acyclic graph, combining a Bayesian factor retriever with a DAG‑aware generator to achieve superior prediction accuracy, stable returns, and faster training across three major Chinese stock markets.
In 2024 DGraph upgraded its architecture by splitting single clusters into multiple business‑specific clusters, adopting a sharded active‑active topology, and replacing its 1:N thread‑pool with an M:N grouped execution model that uses atomic scheduling, while parallelizing FlatBuffer encoding, streamlining SDK conversions, adding DAG debugging, timeline analysis, and dynamic sub‑graph templates to boost scalability, stability and developer productivity.
This article explains the DPP platform’s overall architecture, its key features for rapid iteration, and the three‑stage evolution of its orchestration engine—from the fixed DPP‑Engine to the flexible BizEngine and finally the graph‑based DagEngine—detailing design decisions, protocols, challenges, and future directions.
This tutorial guides you through installing Apache Airflow, initializing its database, starting the web server and scheduler, creating a Python DAG that reads, cleans, groups, and saves CSV data, configuring the DAG directory, and monitoring the pipeline via the Airflow web UI.
This article explains how Flink applications are transformed through StreamGraph, JobGraph, and ExecutionGraph stages, and presents a low‑code canvas approach that lets users assemble DAGs, persist them in a MySQL adjacency list, and generate zero‑code Flink programs using BFS traversal.
Netflix has open‑sourced its Maestro workflow orchestrator, a highly scalable, DAG‑based system built on Git, Java, Gradle and Docker that handles millions of daily jobs for data scientists, enabling ETL, ML pipelines, A/B testing and more, while meeting strict SLOs.
The article details the design, core technical solutions, high‑availability architecture, performance optimizations, and open capabilities of Buffalo, JD.com’s distributed DAG‑based job scheduling platform that supports massive task volumes, complex dependencies, and flexible resource management.
RiskFactor unifies iQIYI’s legacy real‑time and offline feature platforms onto Opal’s DAG‑plus‑SQL engine, accelerating feature production fifteen‑fold, cutting latency from hours to minutes, streamlining development, lowering costs, and delivering more reliable, versioned risk‑control capabilities against sophisticated online threats.
This tutorial explains how to use the Hera Python SDK to create, submit, and manage Argo Workflows on an ACK One Serverless Argo cluster, covering installation, DAG diamond and MapReduce examples, and practical commands for token generation and workflow execution.
Bilibili’s video submission platform uses a layered micro‑service architecture with a DAG‑based scheduler, extensive observability, and HA tactics such as sharding, 64‑bit ID migration, full‑link stress tests, chaos engineering, and multi‑active data‑center deployment, while tooling like trace correlation and automated alerts ensures stability and guides future hybrid‑cloud migration.
This article explains how a data warehouse dependency model was transformed from coarse project-level dependencies to fine-grained task-level DAGs, introduces virtual tasks for external dependencies, describes offset handling, and outlines the technical implementation and future automation plans for large‑scale scheduling systems.
This article introduces Apache Airflow, explains its core concepts such as DAGs, tasks, operators, executors, and the web UI, and provides multiple practical Python code examples for Bash commands, Python functions, SQL queries, task dependencies, sensors, dynamic DAGs, SubDAGs, XCom, email alerts, and error handling.
Exgraph, Baidu Search’s graph execution engine, uses a human‑readable description language and a robust execution core with dependency injection, object pooling, and interruption handling to orchestrate complex, parallel or conditional tasks, improving code readability and unifying diverse execution scenarios in search architecture.
This article presents a metadata‑driven approach to improve data warehouse timeliness by extracting upstream lineage, identifying over‑layered, duplicate, and critical‑path tasks, and applying targeted scheduling and code‑level optimizations, demonstrated with a hotel order wide‑table case study.
This article explains the definition and properties of directed acyclic graphs (DAG), describes the basic topological sorting algorithm steps, and provides a complete Java implementation using Guava's MutableGraph class, illustrating the process with an example and its execution result.
This article explains how to parse OceanBase logs to trace the tenant freeze dump workflow, detailing the roles and log sequences of the freeze check thread, LSFreeze, Flush, DagScheduler, and MiniMerge threads, and illustrating each step with actual log excerpts and code snippets.
The article presents Baidu’s Aladdin vertical search product and introduces the tanGo framework, which abstracts search pipelines into resources, cards, and scenes, enabling configuration‑driven, graph‑based resource scheduling for single results, demand clusters, and groups, while measuring scale, efficiency, and user satisfaction.
This article explains how Volcano Engine identified the tangled authorization, data‑fetching, and performance problems of its advertising AB‑testing platform and refactored it by splitting services, redesigning the data model with MySQL and ClickHouse, applying DAG scheduling, time‑wheel algorithms, Domain‑Driven Design, and rigorous unit testing to achieve a more stable, extensible backend solution.
This article explains how directed acyclic graphs (DAGs) are applied in frontend development for resource loading optimization, component library construction, and task flow orchestration, and demonstrates a real-world implementation in Kuaishou's e‑commerce advertising engine with detailed architecture and code examples.
This article explains how to analyze and automatically optimize complex DAG‑based data platform task chains, identify bottlenecks, adjust upstream task timings, and reduce critical‑path execution time by up to 30 minutes while preventing resource contention and peak overloads.
This guide explains how to combine ByteHouse's cloud‑native data warehouse with Apache Airflow to build scalable, automated, and easy‑to‑manage data pipelines, covering business scenarios, data flow, and step‑by‑step installation and DAG creation.
This article explains the principles of probabilistic programming languages and walks through a step‑by‑step implementation of a minimal PPL in Python, covering model definition, variable representation, DAG traversal, log‑density computation, and a posterior grid illustration.
This article explains the principles of probabilistic programming languages and walks through constructing a basic PPL in Python, covering model definition with latent and observed variables, distribution handling, DAG traversal for log‑density computation, and demonstrates evaluation with example code and visualizations.
The article details Bilibili’s Comet workflow engine—a low‑code, plugin‑extensible platform built since 2019 that uses visual DAG templates, graph‑based legality checks, and asynchronous execution to automate diverse business processes such as SRE automation, permission requests, and push‑task approvals, improving operational efficiency across mobile and web services.
Apache Airflow’s architecture consists of schedulers, executors, workers, a web server, and a metadata database, enabling scalable workflow orchestration, while essential terminology such as DAGs, operators, and sensors defines how tasks are organized, executed, and monitored within data pipelines.
This tutorial walks through creating a simple “Hello World” Apache Airflow DAG by setting up the Python file, importing modules, defining the DAG object, adding a PythonOperator task, writing the callable function, and running the DAG with Airflow’s webserver and scheduler.
Apache Airflow is an open‑source platform for programmatically authoring, scheduling, and monitoring workflows using Directed Acyclic Graphs (DAGs), featuring components such as Scheduler, Web Server, Database, and various Executors, and offering easy‑to‑use, extensible, scalable, and robust integrations for data pipeline management.
This article explains what Apache Airflow DAGs are, describes their architecture and how they model data pipelines as directed acyclic graphs, and provides practical best‑practice guidelines for writing clean, reproducible, and resource‑efficient workflows.
This article explains how to abstract repetitive data‑report tasks into a standardized workflow, describes the core capabilities of scheduling and configuration, shows how to implement DAG‑based visual editors, and compares similar platforms such as n8n and Orange, offering practical code examples and design insights.
This article introduces Huya's offline job scheduling platform, covering its positioning, evolution, system architecture, baseline scheduling techniques, cost‑optimization strategies, resource‑balancing methods, and future intelligent data‑warehouse directions, illustrating how data‑driven automation improves YARN utilization and SLA compliance.
This article explains Apache Airflow's core concepts, including DAG definitions, scheduling intervals, task dependencies, various operators such as BashOperator, PythonOperator, Branch operators, sensors, and custom operators, and provides code examples and configuration details for building robust data pipelines.
This article explains how data scheduling systems manage task execution, ETL processes, and cross‑period dependencies by linking task versions, data partitions, and time parameters, and introduces the offset‑and‑cnt model to express dynamic dependencies in big‑data pipelines.
This article explains how probabilistic programming languages work and walks through building a minimal PPL in Python, covering model definition, latent and observed variables, DAG traversal, log‑density computation, and extensions such as posterior grid approximation.
This article introduces the fundamentals of directed acyclic graphs (DAGs), explains adjacency representations, details the Dagre layout algorithm’s concepts, computation steps, and constraints, and presents the Perfect‑Process front‑end library that implements these techniques for interactive pipeline diagram rendering and editing.
The article describes the background, challenges, architecture, operator design, DAG processing, tag persistence, and business applications of a real-time dynamic tag processing platform (CDP) built to improve revenue growth and cost reduction for Trip.com's international operations.
This article explains how to design and implement a DAG‑based task orchestration framework in Java, covering graph representations, dependency management, executor integration, state tracking, and how to persist workflows and tasks in a relational database for platform‑level usage.
This article explains the principles, steps, and various algorithms behind hierarchical (Sugiyama) graph layout, covering node layering, crossing reduction, coordinate computation, drawing, and practical implementation details with JavaScript libraries such as dagre and d3‑dag.
This article explains how to build a task‑orchestration framework using directed acyclic graphs (DAG), covering graph representations, Java data structures, dependency management, concurrent execution with thread pools, and persisting workflow state to a relational database for platform‑level use.
This article explains how to design and implement a task‑orchestration workflow using a directed acyclic graph (DAG) in Java, covering graph representation, core data structures, concurrency handling with executors, persistence to a database, and platform‑level visualisation.
This article details Baidu's multimodal retrieval architecture, explaining the separation of online and offline services, the design of the Imazon image processing and ingestion platform, its technical indicators, large‑scale streaming and batch pipelines, optimization practices for high throughput, and the underlying content‑relationship engine.
The Imazon platform unifies Baidu’s image acquisition, feature extraction, and ANN‑based multimodal retrieval into a cloud‑native, real‑time pipeline that ingests billions of images daily, optimizes storage and GPU usage, reduces message‑queue costs, and ensures high‑throughput, low‑latency search across text, visual, and voice queries.
This article provides a comprehensive overview of GitLab CI/CD, covering its core concepts—pipelines, stages, jobs, and runners—along with .gitlab-ci.yml configuration, variables, triggers, DAG pipelines, runner types, cloud‑native capabilities, efficiency management, and practical demo examples to help teams implement robust DevOps workflows.
This article explains the concepts of task scheduling dependencies and workflow, introduces graph theory basics such as vertices, edges, and DAGs, and provides a complete Go implementation—including graph structures, BFS traversal, topological sorting, and concurrent execution—to efficiently manage dependent tasks in distributed systems.
This article introduces AntV X6, a versatile JavaScript graph‑editing engine, covering its core capabilities such as node and edge creation, custom HTML/React nodes, rich connection styles, grid and background options, interactive tools like ports, grouping, selection, undo/redo, and provides links to tutorials, demos, and extensibility mechanisms.
The article describes how Liulishuo automates data‑warehouse task scheduling by parsing SQL to build DAGs, introduces systematic governance to improve reliability, reduce execution time, and handle unreliable data sources, ultimately enhancing data‑engineering efficiency.
This article explains what Git is, why version control is essential, compares local, centralized, and distributed systems, describes Git's storage mechanisms and object types, and outlines the typical Git workflow with practical examples.
Alibaba’s Fuxi DAG 2.0 redesign separates logical and physical graphs, introduces dynamic scheduling, unified offline and near‑real‑time execution, and a flexible bubble mode, enabling massive big‑data jobs to run up to five times faster while dramatically reducing resource waste.
EasyScheduler is an open‑source big‑data workflow scheduler that uses a decentralized architecture with Master and Worker nodes coordinated via ZooKeeper, supporting DAG‑based task definitions, various task types, fault tolerance, priority handling, distributed locks, and remote logging, all illustrated with detailed component diagrams.
This article introduces EasyScheduler, an open‑source big data workflow scheduling system, explaining its core terminology, decentralized architecture, distributed lock implementation, thread‑shortage handling, fault‑tolerance mechanisms, task‑retry and priority designs, as well as its logging solution using Logback and gRPC.
Alibaba has released the comprehensive Cluster Data V2018 dataset, detailing eight days of operation for 4,000 servers and their mixed online and offline workloads, including DAG information, enabling researchers to study large‑scale data‑center performance, resource utilization, scheduling algorithms, and derive new insights.
The article describes the design goals, architecture, and key components of Jarvis, an internal DAG‑driven job scheduling platform for big‑data pipelines, covering timed‑shard and workflow schedulers, high‑availability mechanisms, task development for Hive and data‑transfer jobs, dependency handling, APIs, monitoring, and future enhancements.
This article outlines the evolution of 360’s Titan big‑data processing platform, describing the challenges of traditional script‑based development, the three architectural stages (pre‑Titan, Titan 1.0, Titan 2.0), the functional modules, the DITTO component framework, and key takeaways for building flexible, self‑service data pipelines.
This article explains the fundamentals of blockchain technology, introduces IOTA’s Tangle as a DAG‑based alternative without miners, and discusses its claimed benefits such as decentralised security, quantum resistance, scalability and suitability for IoT micro‑payments.
This article explains blockchain fundamentals, clears common misconceptions, describes its basic components, outlines the technology’s evolution from 1.0 to 3.0, and highlights current and future application scenarios across various industries.
The article describes how a high‑traffic e‑commerce transaction system improves availability through rate limiting, circuit breaking, gray‑release, JVM warm‑up, an online diff testing tool, the ARES regression platform, and a DAG‑driven asynchronous execution framework to boost throughput and reduce latency.
The article introduces Weiflow, a dual‑layer DAG‑based machine‑learning workflow framework designed for Sina Weibo, and explains how its modular XML configuration, Scala implementation, and integration with Spark, TensorFlow, Hive, Storm, and Flink improve development efficiency, scalability, and execution performance across the entire ML pipeline.
Voidbox integrates Docker containers with YARN to simplify distributed application development, improve deployment, boost cluster efficiency, and provide fault‑tolerant, DAG‑based execution modes, enabling seamless resource management for Hadoop‑based big data jobs.
This article chronicles Baidu’s eight‑year journey from early Hadoop adoption to advanced MPI, DAG engines, and real‑time streaming platforms, detailing architectural milestones, performance optimizations, and practical lessons for large‑scale offline and online data processing.
