Tagged articles
data pipelines
51 articles · Page 1 of 1
Snowflake reveals a five-month production run where AI agents automatically investigate data pipeline failures, propose fixes, and submit pull requests within 16 minutes on average, with hundreds of PRs merged and 500+ pipelines improved, using a job-based architecture that isolates each investigation for parallel scalability.
AI AgentsSnowflakeagent architecture
0 likes · 16 min read

The article argues that in ontology-driven enterprise systems, performance bottlenecks lie in data integration pipelines and business databases, not the graph database layer, using Palantir Foundry's architecture and Airbus Skywise's 1,000+ daily pipelines as evidence, and shows Memgraph's specs far exceed typical ontology loads.
Airbus SkywiseMemgraphPalantir
0 likes · 13 min read

This article reveals the deep structural isomorphism between manufacturing assembly lines and data pipeline orchestration, mapping five shared design philosophies, a 15-dimension logical correspondence, bidirectional lessons, and six practical mental models for data engineers to adopt factory-floor thinking.
Data EngineeringIdempotencyassembly line
0 likes · 22 min read

This article explains the fundamentals of batch processing, outlines its non‑real‑time nature, lists typical scenarios and pros‑cons, provides practical implementation guidelines, and presents three detailed industry case studies illustrating how large‑scale offline jobs are built and maintained.
Backendbatch processingcase study
0 likes · 11 min read

The article argues that while embodied data can generate revenue, merely selling raw data or collection hours fails to create sustainable high‑value business; true profitability requires integrating data collection, semantic enrichment, model training, evaluation, and real‑world robot feedback into a closed loop.
Embodied AIRobotics Datadata pipelines
0 likes · 16 min read

This article explains how the pure‑Python functional library toolz can transform tangled automation scripts into clear, composable data pipelines, reducing code size, improving testability, and eliminating hidden technical debt through concrete examples and step‑by‑step refactoring.
Pythonautomationdata pipelines
0 likes · 15 min read

Python is praised for its clarity but often deemed slow; this article reveals twelve overlooked, sometimes unconventional techniques—from using enumerate instead of range loops to leveraging Numba, Polars, and mypyc—that can dramatically accelerate data pipelines, APIs, and scientific workloads without rewriting code in another language.
PolarsPythondata pipelines
0 likes · 9 min read

This article explains why robust AI infrastructure is essential, outlines its key components—from specialized hardware and orchestration platforms to security and governance—and provides a step‑by‑step roadmap, real‑world case studies, and best‑practice recommendations for constructing and continuously optimizing AI systems.
DevOpsdata pipelines
0 likes · 17 min read

The article explains why data quality is critical for modern businesses, outlines the challenges data engineers face when monitoring hundreds of models, and provides a step‑by‑step guide to constructing an automated, AI‑driven data quality monitoring system using Strands Agents, Amazon Bedrock, dbt and Amazon Redshift, including code examples, workflow orchestration, security hardening, containerized deployment and observability.
AI AgentsAmazon RedshiftStrands Agents
0 likes · 14 min read

The article introduces SPL’s latest operators—pack-fields, log-to-metric, and metric-to-metric—explaining their smart field aggregation, trimming, type inference, wildcard matching, and label manipulation capabilities, and demonstrates through code examples and performance benchmarks how they dramatically improve data processing efficiency and observability in cloud‑native log services.
SPLcloud-nativedata pipelines
0 likes · 10 min read

This article explains the need for lightweight batch processing, outlines a layered architecture and robustness strategies, and demonstrates how Spring Batch implements these concepts with clear interfaces, job management, and support for ignore, retry, and restart mechanisms.
JavaRobustnessSpring Batch
0 likes · 10 min read

Argo Workflows, the top‑rated CNCF project, extends Kubernetes to orchestrate AI, ML, and data pipelines with a scalable, cloud‑native architecture, offering powerful scheduling, Python SDK support, and new plugins for Spark, Ray, and PyTorch.
AIArgo WorkflowsKubernetes
0 likes · 9 min read

This article explains why Kafka is essential for modern data engineering, highlighting its widespread adoption, high throughput, scalability, durability, integration with streaming ecosystems, and common real‑time use cases, while also providing a concise list of interview topics for aspiring engineers.
data pipelinesreal-time processingstreaming
0 likes · 6 min read

Chronon is an open‑source ML feature platform that lets engineers declaratively define, compute, and serve both batch and real‑time features with built‑in observability, data‑quality checks, and a low‑latency retrieval API, ensuring online‑offline consistency while simplifying pipeline management and enabling future automation.
Chronondata pipelinesobservability
0 likes · 13 min read

The DeepSeek open‑source week introduced seven breakthrough technologies—FlashMLA, DeepGEMM, DeepEP, DualPipe, EPLB, 3FS, and Smallpond—that together overhaul data flow, algorithmic complexity, hardware utilization, MoE communication, and resource balancing, dramatically improving large‑model training efficiency and lowering entry barriers for the AI industry.
AI hardwareDeepSeekdata pipelines
0 likes · 17 min read

Apache SeaTunnel is an open‑source, distributed data integration platform that enables efficient real‑time data synchronization across diverse sources and destinations, supporting both streaming and batch processing, with detailed architecture, connector plugins, CDC handling, transform capabilities, and deployment strategies for large‑scale data pipelines.
Apache SeaTunnelCDCReal-Time Data Integration
0 likes · 34 min read

This article introduces the open‑source Kestra workflow engine, outlines its key features for building scalable data pipelines, provides step‑by‑step Docker installation and YAML workflow examples, and showcases its visual UI for monitoring and managing complex ETL and automation tasks.
DockerKestraWorkflow Orchestration
0 likes · 6 min read

The article details ByteDance's use of Ray and RayData to construct scalable audio and video data processing pipelines for multimodal AI models, addressing challenges of massive data volume, resource constraints, and fault tolerance through pipeline design, RayCore enhancements, and custom scheduling optimizations.
AIByteDanceMultimodal Models
0 likes · 16 min read

The article discusses the challenges of training ever‑larger AI models on internet‑scale data, critiques traditional batch ETL pipelines, and proposes a streaming data‑flow architecture with dynamic data selection and a shared‑memory/Alluxio middle layer to decouple data processing from model training, improving efficiency and scalability.
AI infrastructureMultimodal Datadata pipelines
0 likes · 20 min read

CryptoHouse is a free, real‑time blockchain analytics platform built on ClickHouse and Goldsky, offering SQL access to Solana and Ethereum data, with materialized views, fair‑use quotas, deduplication via ReplacingMergeTree, a lightweight UI, and detailed engineering insights.
Blockchain analyticsClickHouseGoldsky
0 likes · 18 min read

This article traces the evolution of streaming processing languages, compares iLogtail's native and extended pipeline modes, and demonstrates how the new SPL syntax in iLogtail 2.0 delivers high‑performance, flexible log and time‑series data processing with unified, SQL‑like commands and interactive debugging tools.
Log AnalyticsSPLdata pipelines
0 likes · 13 min read

This article introduces Apache Airflow, explains its core concepts such as DAGs, tasks, operators, executors, and the web UI, and provides multiple practical Python code examples for Bash commands, Python functions, SQL queries, task dependencies, sensors, dynamic DAGs, SubDAGs, XCom, email alerts, and error handling.
Apache AirflowDAGPython
0 likes · 7 min read

To support rapid overseas expansion, the article outlines a comprehensive backend architecture—including management, data ingestion, device tracking, attribution, and offline tasks—while detailing fine-grained user permission controls, automated product onboarding, batch ad creation, and server‑side attribution workflows, plus future enhancements.
Advertising PlatformBackenddata pipelines
0 likes · 12 min read

This article explains how Tencent leverages A/B testing across its overseas games, detailing market differences, experimental methodology, multi‑cloud platform compliance, data architecture, and case studies that illustrate how targeted experiments improve user onboarding, gameplay settings, and email‑based re‑engagement.
A/B testingGame Analyticsdata pipelines
0 likes · 12 min read

This article traces the evolution of data architecture from early BI systems through traditional big‑data stacks, streaming, Lambda and Kappa designs, and explains how a unified stream‑batch model simplifies development while keeping logic consistent across data‑analysis and pipeline applications.
BI systemsKappa architectureLambda architecture
0 likes · 16 min read


SQB BlogJul 20, 2023 · Artificial Intelligence
This article explains the design, implementation, and iterative optimization of Boss Circle's recommendation engine, covering the initial simple ranking, the introduction of Elasticsearch‑based scoring, multi‑pool data sources, machine‑learning experiments, real‑time feature handling, and future personalization challenges.
ElasticsearchPersonalizationRanking
0 likes · 17 min read

This guide introduces StreamPipes, an end‑to‑end industrial IoT toolbox for non‑technical users, outlines its key features, shows how to connect data sources, build pipelines, visualize data, and provides step‑by‑step Docker‑Compose installation, configuration, and development instructions.
Docker ComposeIndustrial IoTStreamPipes
0 likes · 8 min read

This article explains how Apache Airflow's Celery Executor works, covering its key features, installation steps, configuration details, architectural components, and the complete task execution process that enables scalable, distributed workflow orchestration for data pipelines.
Apache AirflowCelery ExecutorTask Scheduling
0 likes · 15 min read

This guide explains how to install Apache Airflow in a virtual environment, set up the Airflow home, create an admin user, understand role‑based access control, and run essential Airflow CLI commands for managing DAGs and tasks.
Airflow RolesApache AirflowCLI
0 likes · 6 min read

Apache Airflow is an open‑source platform for programmatically authoring, scheduling, and monitoring workflows using Directed Acyclic Graphs (DAGs), featuring components such as Scheduler, Web Server, Database, and various Executors, and offering easy‑to‑use, extensible, scalable, and robust integrations for data pipeline management.
Apache AirflowDAGExecutor
0 likes · 10 min read

This article introduces EasyCV, an open‑source all‑in‑one visual algorithm platform that abstracts diverse data sources, provides SOTA self‑supervised models, and offers ready‑to‑download datasets for image classification, object detection, segmentation, and pose estimation, complete with configuration examples.
Computer VisionEasyCVObject Detection
0 likes · 9 min read

The talk by Tang Gengyang from Citic Baixin Bank details the challenges faced in risk feature engineering, presents two solution frameworks (1.0 and 2.0) for accelerating deployment, improving reuse, handling offline/online consistency, and outlines future enhancements for a more efficient, automated feature pipeline.
Asynchronous ProcessingFlinkdata pipelines
0 likes · 14 min read

This article introduces the concept of Change Data Capture (CDC), explains why traditional batch reporting strains resources, describes how CDC captures only data changes to keep source databases performant, and outlines modern CDC architectures, production‑ready considerations, and best‑practice guidelines for building reliable data pipelines.
CDCchange data capturedata integration
0 likes · 16 min read

This article explains Apache Airflow's core concepts, including DAG definitions, scheduling intervals, task dependencies, various operators such as BashOperator, PythonOperator, Branch operators, sensors, and custom operators, and provides code examples and configuration details for building robust data pipelines.
Apache AirflowDAGScheduling
0 likes · 15 min read

This article provides a detailed tutorial on Apache Airflow fundamentals, Docker-based installation on Windows, Dockerfile creation, container deployment via Docker run and Docker Compose, Airflow configuration, and practical usage of DAGs, tasks, connections, and UI features for data pipeline orchestration.
Apache AirflowDockerDocker Compose
0 likes · 14 min read

This article explains ByteDance's baseline monitoring system for data pipelines, detailing its motivation, core concepts, architecture, instance generation, alert types, and handling of complex task dependencies to reduce operational costs and improve SLA compliance across hundreds of projects.
alertingbaseline monitoringbig data
0 likes · 21 min read


21CTONov 1, 2021 · Big Data
This guide outlines the fast‑growing data engineering career path, covering essential Linux fundamentals, programming languages, testing, database concepts, data warehouses, processing frameworks, messaging systems, cluster computing, workflow scheduling, monitoring, infrastructure as code, and CI/CD tools.
Data Engineeringbig datadata pipelines
0 likes · 5 min read

The article examines Apache Kafka's strengths and shortcomings, explores the operational complexities of managing a Kafka deployment, and encourages organizations to reassess its value versus emerging alternatives by weighing maturity, scalability, and total cost of ownership.
Alternative PlatformsKafkaOperational Challenges
0 likes · 12 min read

Apache Kafka 3.0.0 introduces a host of enhancements—including deprecated Java 8/Scala 2.12 support, Raft metadata snapshots, stronger producer guarantees, MirrorMaker 2 upgrades, and Kafka Streams improvements—while continuing to serve real‑time data pipelines and streaming applications.
Apache KafkaKafka 3.0big data
0 likes · 3 min read

At the 16th GOPS Global Operations Conference, Shen Hui of DingMao Technology explained how time‑series data analysis underpins AIOps, outlining its four‑step workflow, key challenges, and the company’s three‑pipeline solution that enables trend forecasting, fault prediction, and a robust AI‑driven operational platform.
AIAIOpsOperations
0 likes · 7 min read

Tubi’s Popper platform combines a Scala‑based experiment engine, reproducible JSON‑stored configurations, a React UI, and data pipelines using Spark and Akka to enable fast, cross‑team A/B testing, automated analysis, health checks, and data‑driven decision making across mobile and OTT services.
A/B testingAkkaExperimentation platform
0 likes · 15 min read

This article examines the evolution of user data, computing power, and models, and presents the design principles, key architectural factors, and practical implementation techniques for building a full‑stack machine learning platform that supports large‑scale data processing, distributed training, and low‑latency online serving.
Machine Learning Platformbig-data-integrationdata pipelines
0 likes · 15 min read

The article describes an intelligent duty robot that uses a sense‑think‑act framework and job dependency graphs to automatically monitor, diagnose, and remediate data pipeline jobs in a real‑estate platform, reducing operational pressure and achieving over 98% notification accuracy.
Real Estate Dataautomationdata pipelines
0 likes · 9 min read

The article explains what back pressure is in Flink streaming jobs, why it occurs when data generation outpaces downstream consumption, how Flink monitors it via stack‑trace sampling, configurable parameters, Web UI visualization, and compares the approach with Spark Streaming's back pressure mechanism.
FlinkSparkdata pipelines
0 likes · 5 min read


AntTechMay 7, 2019 · Artificial Intelligence
SQLFlow is an open‑source project that connects diverse SQL engines (MySQL, Hive, SparkSQL, etc.) with AI frameworks (TensorFlow, PyTorch, XGBoost, etc.) through extended SQL syntax, enabling analysts to train and predict models using only a few SQL statements while aiming for high scalability and performance.
AI IntegrationGoSQL
0 likes · 13 min read

This article introduces TensorFlow Extended (TFX), illustrating practical TensorFlow examples such as ship trajectory classification, insurance premium adjustments, and car auction pricing, then explains TFX’s data validation, schema generation, model analysis, and deployment options to streamline machine‑learning pipelines.
AITFXTensorFlow
0 likes · 12 min read


58 TechJan 11, 2019 · Artificial Intelligence
This article describes the design and implementation of a comprehensive efficiency‑optimization platform at 58.com, detailing its end‑to‑end workflow—from log aggregation and feature extraction through machine learning model training and online experimentation—highlighting modular, configurable, and scalable solutions for multi‑business, multi‑product ranking.
click-through rateconversion ratedata pipelines
0 likes · 25 min read


21CTOFeb 20, 2018 · Big Data
This article explains how real-time streaming has evolved from batch Hadoop systems through Lambda architecture to modern Kappa-style pipelines, highlighting its growing importance for developers, enterprises, and the integration of streaming with microservices, AI, and cloud-native technologies.
AI IntegrationKappa architectureLambda architecture
0 likes · 8 min read

Alibaba’s Double 11 live‑data dashboards required ultra‑high‑precision, low‑latency real‑time processing of billions of events, and the article explains the end‑to‑end architecture—including DRC, TimeTunnel, Galaxy, OTS, XTool, and OneService—used to achieve million‑plus QPS, fault‑tolerance, and flexible data collection.
AlibabaTimeTunnelbig data architecture
0 likes · 14 min read


21CTOSep 28, 2015 · Artificial Intelligence
This article details Meituan's end‑to‑end recommendation system, covering its four‑layer architecture, data sources, candidate‑generation strategies, fusion methods, and both linear and non‑linear re‑ranking models, while highlighting practical optimizations like AB testing and online learning.
MeituanRecommendation Systemdata pipelines
0 likes · 15 min read

At Airbnb’s inaugural OpenAir conference, the company unveiled three open‑source big‑data tools—Airpal, a Presto‑based visual SQL query engine; Aerosolve, an interpretable machine‑learning engine for pricing recommendations; and Airflow, an internal platform for orchestrating and monitoring data pipelines.
AirbnbOpenAirbig data
0 likes · 4 min read
