Tagged articles
DAG
63 articles · Page 1 of 1
Ant Digital launches LingDT-Decision-1.0, a 4B/27B parameter decision model for enterprise AI that introduces native multi-select and sequential DAG decision capabilities, outperforming Microsoft-Decision-1 and Jev 1.13 on 23 benchmarks while cutting token costs over 90% and boosting response speed 10x in real-world customer service, risk control, and finance tasks.
Ant DigitalDAGDecision Model
0 likes · 8 min read

The article explains how workflow orchestration transforms repetitive manual tasks into automated pipelines by using DAG‑based nodes, directed edges, and three fundamental programming constructs—sequential execution, conditional branching, and loops—illustrated with a daily data‑fetch‑to‑Excel‑to‑email example and the WorkBuddy platform.
DAGLoopWorkBuddy
0 likes · 9 min read

The article analyzes why traditional script‑based API tests fail in microservice environments and proposes a Skill‑oriented, DAG‑driven distributed automation engine that unifies HTTP calls, database verification, message validation, retries, and observability into a scalable, governable platform.
DAGJavaKubernetes
0 likes · 37 min read

Graph Engineering isn’t a brand‑new technology but a shift from single‑agent loops to a network of specialized nodes, edges, and shared state that lets multiple AI agents collaborate, run in parallel, and avoid context decay, with practical LangGraph examples, pros, cons, and when to adopt it.
AI workflowDAGGraph Engineering
0 likes · 23 min read

AlphaPROBE reformulates alpha‑factor discovery as a strategy‑navigation problem on a directed acyclic graph, combining a Bayesian factor retriever with a DAG‑aware generator to achieve superior prediction accuracy, stable returns, and faster training across three major Chinese stock markets.
AlphaPROBEBayesian RetrievalDAG
0 likes · 22 min read

This article outlines production-tested Flink operator naming conventions emphasizing business-logic clarity, concise action-object-result formats, type-specific patterns for map/filter/join/window/sink, DAG-stage prefixes, and pairing name() with uid() for state recovery, while avoiding technical jargon and dynamic variables.
Best PracticesDAGFlink
0 likes · 9 min read

In 2024 DGraph upgraded its architecture by splitting single clusters into multiple business‑specific clusters, adopting a sharded active‑active topology, and replacing its 1:N thread‑pool with an M:N grouped execution model that uses atomic scheduling, while parallelizing FlatBuffer encoding, streamlining SDK conversions, adding DAG debugging, timeline analysis, and dynamic sub‑graph templates to boost scalability, stability and developer productivity.
Backend EngineeringDAGRecommendation System
0 likes · 13 min read

This article explains the DPP platform’s overall architecture, its key features for rapid iteration, and the three‑stage evolution of its orchestration engine—from the fixed DPP‑Engine to the flexible BizEngine and finally the graph‑based DagEngine—detailing design decisions, protocols, challenges, and future directions.
DAGDPPorchestration
0 likes · 16 min read

This tutorial guides you through installing Apache Airflow, initializing its database, starting the web server and scheduler, creating a Python DAG that reads, cleans, groups, and saves CSV data, configuring the DAG directory, and monitoring the pipeline via the Airflow web UI.
Apache AirflowDAGETL
0 likes · 6 min read

This article explains how Flink applications are transformed through StreamGraph, JobGraph, and ExecutionGraph stages, and presents a low‑code canvas approach that lets users assemble DAGs, persist them in a MySQL adjacency list, and generate zero‑code Flink programs using BFS traversal.
DAGFlinkLow-Code Development
0 likes · 6 min read


21CTOAug 4, 2024 · Operations
Netflix has open‑sourced its Maestro workflow orchestrator, a highly scalable, DAG‑based system built on Git, Java, Gradle and Docker that handles millions of daily jobs for data scientists, enabling ETL, ML pipelines, A/B testing and more, while meeting strict SLOs.
DAGMaestroNetflix
0 likes · 5 min read

The article details the design, core technical solutions, high‑availability architecture, performance optimizations, and open capabilities of Buffalo, JD.com’s distributed DAG‑based job scheduling platform that supports massive task volumes, complex dependencies, and flexible resource management.
BackendDAGDistributed Scheduling
0 likes · 13 min read

RiskFactor unifies iQIYI’s legacy real‑time and offline feature platforms onto Opal’s DAG‑plus‑SQL engine, accelerating feature production fifteen‑fold, cutting latency from hours to minutes, streamlining development, lowering costs, and delivering more reliable, versioned risk‑control capabilities against sophisticated online threats.
DAGRisk Controlbig data
0 likes · 14 min read

This tutorial explains how to use the Hera Python SDK to create, submit, and manage Argo Workflows on an ACK One Serverless Argo cluster, covering installation, DAG diamond and MapReduce examples, and practical commands for token generation and workflow execution.
Argo WorkflowsDAGHera SDK
0 likes · 11 min read

Bilibili’s video submission platform uses a layered micro‑service architecture with a DAG‑based scheduler, extensive observability, and HA tactics such as sharding, 64‑bit ID migration, full‑link stress tests, chaos engineering, and multi‑active data‑center deployment, while tooling like trace correlation and automated alerts ensures stability and guides future hybrid‑cloud migration.
BilibiliDAGHigh Availability
0 likes · 35 min read

This article explains how a data warehouse dependency model was transformed from coarse project-level dependencies to fine-grained task-level DAGs, introduces virtual tasks for external dependencies, describes offset handling, and outlines the technical implementation and future automation plans for large‑scale scheduling systems.
DAGTask Schedulingautomation
0 likes · 13 min read

This article introduces Apache Airflow, explains its core concepts such as DAGs, tasks, operators, executors, and the web UI, and provides multiple practical Python code examples for Bash commands, Python functions, SQL queries, task dependencies, sensors, dynamic DAGs, SubDAGs, XCom, email alerts, and error handling.
Apache AirflowDAGPython
0 likes · 7 min read

Exgraph, Baidu Search’s graph execution engine, uses a human‑readable description language and a robust execution core with dependency injection, object pooling, and interruption handling to orchestrate complex, parallel or conditional tasks, improving code readability and unifying diverse execution scenarios in search architecture.
DAGGo DevelopmentSearch Architecture
0 likes · 10 min read

This article presents a metadata‑driven approach to improve data warehouse timeliness by extracting upstream lineage, identifying over‑layered, duplicate, and critical‑path tasks, and applying targeted scheduling and code‑level optimizations, demonstrated with a hotel order wide‑table case study.
DAGLineageOptimization
0 likes · 7 min read

This article explains the definition and properties of directed acyclic graphs (DAG), describes the basic topological sorting algorithm steps, and provides a complete Java implementation using Guava's MutableGraph class, illustrating the process with an example and its execution result.
DAGGraph AlgorithmsGuava
0 likes · 4 min read

This article explains how to parse OceanBase logs to trace the tenant freeze dump workflow, detailing the roles and log sequences of the freeze check thread, LSFreeze, Flush, DagScheduler, and MiniMerge threads, and illustrating each step with actual log excerpts and code snippets.
DAGFreeze ProcessOceanBase
0 likes · 16 min read

The article presents Baidu’s Aladdin vertical search product and introduces the tanGo framework, which abstracts search pipelines into resources, cards, and scenes, enabling configuration‑driven, graph‑based resource scheduling for single results, demand clusters, and groups, while measuring scale, efficiency, and user satisfaction.
BackendConfigurationDAG
0 likes · 10 min read

This article explains how Volcano Engine identified the tangled authorization, data‑fetching, and performance problems of its advertising AB‑testing platform and refactored it by splitting services, redesigning the data model with MySQL and ClickHouse, applying DAG scheduling, time‑wheel algorithms, Domain‑Driven Design, and rigorous unit testing to achieve a more stable, extensible backend solution.
AB TestingAdvertisingBackend
0 likes · 16 min read

This article explains how directed acyclic graphs (DAGs) are applied in frontend development for resource loading optimization, component library construction, and task flow orchestration, and demonstrates a real-world implementation in Kuaishou's e‑commerce advertising engine with detailed architecture and code examples.
DAGFrontend Developmentcomponent library
0 likes · 14 min read

This article explains how to analyze and automatically optimize complex DAG‑based data platform task chains, identify bottlenecks, adjust upstream task timings, and reduce critical‑path execution time by up to 30 minutes while preventing resource contention and peak overloads.
DAGResource OptimizationTask Scheduling
0 likes · 15 min read

This guide explains how to combine ByteHouse's cloud‑native data warehouse with Apache Airflow to build scalable, automated, and easy‑to‑manage data pipelines, covering business scenarios, data flow, and step‑by‑step installation and DAG creation.
Apache AirflowByteHouseCloud Data Warehouse
0 likes · 10 min read

This article explains the principles of probabilistic programming languages and walks through a step‑by‑step implementation of a minimal PPL in Python, covering model definition, variable representation, DAG traversal, log‑density computation, and a posterior grid illustration.
Bayesian inferenceDAGPPL
0 likes · 12 min read

This article explains the principles of probabilistic programming languages and walks through constructing a basic PPL in Python, covering model definition with latent and observed variables, distribution handling, DAG traversal for log‑density computation, and demonstrates evaluation with example code and visualizations.
Bayesian inferenceDAGPPL
0 likes · 13 min read

The article details Bilibili’s Comet workflow engine—a low‑code, plugin‑extensible platform built since 2019 that uses visual DAG templates, graph‑based legality checks, and asynchronous execution to automate diverse business processes such as SRE automation, permission requests, and push‑task approvals, improving operational efficiency across mobile and web services.
DAGGoSRE
0 likes · 18 min read

Apache Airflow’s architecture consists of schedulers, executors, workers, a web server, and a metadata database, enabling scalable workflow orchestration, while essential terminology such as DAGs, operators, and sensors defines how tasks are organized, executed, and monitored within data pipelines.
Apache AirflowDAGETL
0 likes · 8 min read

This tutorial walks through creating a simple “Hello World” Apache Airflow DAG by setting up the Python file, importing modules, defining the DAG object, adding a PythonOperator task, writing the callable function, and running the DAG with Airflow’s webserver and scheduler.
Apache AirflowDAGData Engineering
0 likes · 9 min read

Apache Airflow is an open‑source platform for programmatically authoring, scheduling, and monitoring workflows using Directed Acyclic Graphs (DAGs), featuring components such as Scheduler, Web Server, Database, and various Executors, and offering easy‑to‑use, extensible, scalable, and robust integrations for data pipeline management.
Apache AirflowDAGExecutor
0 likes · 10 min read

This article explains what Apache Airflow DAGs are, describes their architecture and how they model data pipelines as directed acyclic graphs, and provides practical best‑practice guidelines for writing clean, reproducible, and resource‑efficient workflows.
Apache AirflowBest PracticesDAG
0 likes · 10 min read

This article explains how to abstract repetitive data‑report tasks into a standardized workflow, describes the core capabilities of scheduling and configuration, shows how to implement DAG‑based visual editors, and compares similar platforms such as n8n and Orange, offering practical code examples and design insights.
BackendDAGScheduling
0 likes · 20 min read

This article introduces Huya's offline job scheduling platform, covering its positioning, evolution, system architecture, baseline scheduling techniques, cost‑optimization strategies, resource‑balancing methods, and future intelligent data‑warehouse directions, illustrating how data‑driven automation improves YARN utilization and SLA compliance.
Cost OptimizationDAGYARN
0 likes · 12 min read

This article explains Apache Airflow's core concepts, including DAG definitions, scheduling intervals, task dependencies, various operators such as BashOperator, PythonOperator, Branch operators, sensors, and custom operators, and provides code examples and configuration details for building robust data pipelines.
Apache AirflowDAGScheduling
0 likes · 15 min read

This article explains how data scheduling systems manage task execution, ETL processes, and cross‑period dependencies by linking task versions, data partitions, and time parameters, and introduces the offset‑and‑cnt model to express dynamic dependencies in big‑data pipelines.
DAGData SchedulingETL
0 likes · 14 min read

This article explains how probabilistic programming languages work and walks through building a minimal PPL in Python, covering model definition, latent and observed variables, DAG traversal, log‑density computation, and extensions such as posterior grid approximation.
Bayesian inferenceDAGPPL
0 likes · 11 min read

This article introduces the fundamentals of directed acyclic graphs (DAGs), explains adjacency representations, details the Dagre layout algorithm’s concepts, computation steps, and constraints, and presents the Perfect‑Process front‑end library that implements these techniques for interactive pipeline diagram rendering and editing.
AlgorithmDAGfrontend
0 likes · 13 min read

The article describes the background, challenges, architecture, operator design, DAG processing, tag persistence, and business applications of a real-time dynamic tag processing platform (CDP) built to improve revenue growth and cost reduction for Trip.com's international operations.
DAGTag ManagementTiDB
0 likes · 16 min read

This article explains how to design and implement a DAG‑based task orchestration framework in Java, covering graph representations, dependency management, executor integration, state tracking, and how to persist workflows and tasks in a relational database for platform‑level usage.
DAGExecutorJava
0 likes · 11 min read

This article explains the principles, steps, and various algorithms behind hierarchical (Sugiyama) graph layout, covering node layering, crossing reduction, coordinate computation, drawing, and practical implementation details with JavaScript libraries such as dagre and d3‑dag.
DAGSugiyama algorithmgraph layout
0 likes · 16 min read

This article explains how to build a task‑orchestration framework using directed acyclic graphs (DAG), covering graph representations, Java data structures, dependency management, concurrent execution with thread pools, and persisting workflow state to a relational database for platform‑level use.
DAGJavaTask Scheduling
0 likes · 11 min read

This article explains how to design and implement a task‑orchestration workflow using a directed acyclic graph (DAG) in Java, covering graph representation, core data structures, concurrency handling with executors, persistence to a database, and platform‑level visualisation.
BackendConcurrencyDAG
0 likes · 11 min read

This article details Baidu's multimodal retrieval architecture, explaining the separation of online and offline services, the design of the Imazon image processing and ingestion platform, its technical indicators, large‑scale streaming and batch pipelines, optimization practices for high throughput, and the underlying content‑relationship engine.
DAGMultimodal RetrievalSystem Optimization
0 likes · 13 min read

The Imazon platform unifies Baidu’s image acquisition, feature extraction, and ANN‑based multimodal retrieval into a cloud‑native, real‑time pipeline that ingests billions of images daily, optimizes storage and GPU usage, reduces message‑queue costs, and ensures high‑throughput, low‑latency search across text, visual, and voice queries.
DAGMultimodal Retrievalcloud native
0 likes · 13 min read

This article provides a comprehensive overview of GitLab CI/CD, covering its core concepts—pipelines, stages, jobs, and runners—along with .gitlab-ci.yml configuration, variables, triggers, DAG pipelines, runner types, cloud‑native capabilities, efficiency management, and practical demo examples to help teams implement robust DevOps workflows.
CI/CD pipelinesDAGDevOps
0 likes · 19 min read

This article explains the concepts of task scheduling dependencies and workflow, introduces graph theory basics such as vertices, edges, and DAGs, and provides a complete Go implementation—including graph structures, BFS traversal, topological sorting, and concurrent execution—to efficiently manage dependent tasks in distributed systems.
ConcurrencyDAGGo
0 likes · 10 min read

This article introduces AntV X6, a versatile JavaScript graph‑editing engine, covering its core capabilities such as node and edge creation, custom HTML/React nodes, rich connection styles, grid and background options, interactive tools like ports, grouping, selection, undo/redo, and provides links to tutorials, demos, and extensibility mechanisms.
AntV X6DAGER diagram
0 likes · 11 min read

The article describes how Liulishuo automates data‑warehouse task scheduling by parsing SQL to build DAGs, introduces systematic governance to improve reliability, reduce execution time, and handle unreliable data sources, ultimately enhancing data‑engineering efficiency.
DAGData EngineeringSQL parsing
0 likes · 12 min read

This article explains what Git is, why version control is essential, compares local, centralized, and distributed systems, describes Git's storage mechanisms and object types, and outlines the typical Git workflow with practical examples.
DAGDelta StorageGit
0 likes · 21 min read

Alibaba’s Fuxi DAG 2.0 redesign separates logical and physical graphs, introduces dynamic scheduling, unified offline and near‑real‑time execution, and a flexible bubble mode, enabling massive big‑data jobs to run up to five times faster while dramatically reducing resource waste.
DAGbig datadistributed-systems
0 likes · 38 min read


21CTOApr 29, 2019 · Big Data
EasyScheduler is an open‑source big‑data workflow scheduler that uses a decentralized architecture with Master and Worker nodes coordinated via ZooKeeper, supporting DAG‑based task definitions, various task types, fault tolerance, priority handling, distributed locks, and remote logging, all illustrated with detailed component diagrams.
DAGbig datadistributed-systems
0 likes · 17 min read

This article introduces EasyScheduler, an open‑source big data workflow scheduling system, explaining its core terminology, decentralized architecture, distributed lock implementation, thread‑shortage handling, fault‑tolerance mechanisms, task‑retry and priority designs, as well as its logging solution using Logback and gRPC.
DAGLoggingfault tolerance
0 likes · 14 min read

Alibaba has released the comprehensive Cluster Data V2018 dataset, detailing eight days of operation for 4,000 servers and their mixed online and offline workloads, including DAG information, enabling researchers to study large‑scale data‑center performance, resource utilization, scheduling algorithms, and derive new insights.
DAGDatasetResource Utilization
0 likes · 7 min read

The article describes the design goals, architecture, and key components of Jarvis, an internal DAG‑driven job scheduling platform for big‑data pipelines, covering timed‑shard and workflow schedulers, high‑availability mechanisms, task development for Hive and data‑transfer jobs, dependency handling, APIs, monitoring, and future enhancements.
DAGHigh Availabilityjob scheduling
0 likes · 17 min read

This article outlines the evolution of 360’s Titan big‑data processing platform, describing the challenges of traditional script‑based development, the three architectural stages (pre‑Titan, Titan 1.0, Titan 2.0), the functional modules, the DITTO component framework, and key takeaways for building flexible, self‑service data pipelines.
DAGDITTOTitan
0 likes · 14 min read

This article explains the fundamentals of blockchain technology, introduces IOTA’s Tangle as a DAG‑based alternative without miners, and discusses its claimed benefits such as decentralised security, quantum resistance, scalability and suitability for IoT micro‑payments.
Blockchain AlternativeDAGIoT
0 likes · 6 min read

This article explains blockchain fundamentals, clears common misconceptions, describes its basic components, outlines the technology’s evolution from 1.0 to 3.0, and highlights current and future application scenarios across various industries.
BlockchainBlockchain 2.0Consensus Mechanism
0 likes · 18 min read

The article describes how a high‑traffic e‑commerce transaction system improves availability through rate limiting, circuit breaking, gray‑release, JVM warm‑up, an online diff testing tool, the ARES regression platform, and a DAG‑driven asynchronous execution framework to boost throughput and reduce latency.
Async ProgrammingBackendDAG
0 likes · 6 min read

The article introduces Weiflow, a dual‑layer DAG‑based machine‑learning workflow framework designed for Sina Weibo, and explains how its modular XML configuration, Scala implementation, and integration with Spark, TensorFlow, Hive, Storm, and Flink improve development efficiency, scalability, and execution performance across the entire ML pipeline.
DAGScalaSpark
0 likes · 16 min read

Voidbox integrates Docker containers with YARN to simplify distributed application development, improve deployment, boost cluster efficiency, and provide fault‑tolerant, DAG‑based execution modes, enabling seamless resource management for Hadoop‑based big data jobs.
Cluster ComputingDAGDocker
0 likes · 17 min read

This article chronicles Baidu’s eight‑year journey from early Hadoop adoption to advanced MPI, DAG engines, and real‑time streaming platforms, detailing architectural milestones, performance optimizations, and practical lessons for large‑scale offline and online data processing.
BaiduDAGHadoop
0 likes · 21 min read
