Qunce Intelligence processes 300K hours of embodied intelligence data (10TB/day) using MaxCompute Serverless and MaxFrame, enabling video decoding, alignment, model inference, and quality checks in a unified pipeline without self-managed peak clusters, handling PB-scale multimodal sensor data with fault tolerance.
This article demonstrates replacing custom Python scrapers with Node.js native fetch for concurrent API calls across platforms, reducing maintenance overhead and legal risks while accelerating AI-assisted development workflows.
DataWorks AI Data Generation turns synthetic data creation into a pipeline: a JSON config lets LLMs orchestrate business scenarios while a rule engine enforces cross-field consistency, delivering simulation-grade wide tables in minutes for testing, demos, and model training across industries like IoV, finance, and healthcare.
This article details 360AI's TLP annotation platform, which transforms chaotic manual labeling into an engineered data production line by integrating project management, a real-time workbench with rejection tracking, model-assisted labeling, compute scheduling, role-based collaboration, and a closed-loop where production bad cases flow back into new annotation tasks, turning data into a compounding asset.
This article explains exactly-once processing semantics in data engineering, contrasting it with at-most-once and at-least-once, detailing why retries cause duplicates, the roles of checkpoints, idempotency, and transactions, why technical exactly-once doesn't guarantee business correctness, and the four critical failure tests needed before production deployment.
This article systematically addresses seven technical doubts about the OntoL 8.0 ontology reasoning platform, detailing architectural mechanisms for data pipelines, acceptance workflows, evidence-based traceability, in-process replica semantics, consistency validation, Memgraph licensing, and video verification.
This article details Feeder pipelines that transport diverse data types—user features, content tags, embeddings, popularity rankings, and operational rules—from offline computation to online serving, covering batch atomic switches, NRT incremental updates, cursor-based recovery, scheduling platform design, and latency observability practices.
This article explains why enterprises need professional data integration platforms, detailing sync strategies (full, incremental, CDC), visual ETL workflows with a sales analysis example, real-time computing options, data quality dimensions, and operational practices for scheduling, monitoring, and failure recovery.
This article clarifies the distinctions between data lakes, data warehouses, and lakehouse architectures, explaining their core philosophies, ideal use cases, common pitfalls like data swamps, and why stable data pipelines matter more than architectural buzzwords.
China Postal Savings Bank transformed its serial number data upload pipeline from a four-tier architecture to a flat two-tier system using centralized control, direct device connections, and intelligent traffic scheduling, reducing processing time from 20 to 5 minutes and cutting operational overhead by 50%.
China Postal Savings Bankarchitecture transformationcentralized control
This article outlines the eight core modules of enterprise data architecture—data source, integration, storage, processing, modeling, governance, service, and operations—explaining how each layer ensures data flows reliably from source systems to business consumption while maintaining quality, consistency, and operational stability.
This article introduces Apache Kafka as a distributed streaming platform, detailing its core advantages—high throughput via sequential writes and zero-copy, horizontal scalability through partitions, durability with 7-day retention and replication—and key use cases including log aggregation, event-driven architectures, real-time analytics, and big data integration with tools like Flink and Spark Streaming.
The article details a unified data production platform where Forge Agent orchestrates the development lifecycle, Sindri translates domain DSL into executable plans with validation gates, and ADF manages DAG scheduling and multi-engine execution, while a quality system provides end-to-end verification, lineage, and monitoring to enable trustworthy automation.
The article presents a unified data development platform where Forge Agent acts as an AI facade, Sindri handles deterministic planning via DSL, ADF manages execution and scheduling, and quality systems ensure trustworthiness, moving beyond chat interfaces to controlled, automated data production.
Volcano Engine's LAS video fine-grained understanding operator enables automated, scalable video data pipelines for embodied intelligence and autonomous driving, achieving 94.41% dual-arm accuracy, 100k hours/day processing, 90%+ data usability, and cutting annotation effort by 200+ person-days daily while reducing iteration cycles from weeks to days.
This article explains the core differences between ETL, ELT, and ETLT data integration patterns, emphasizing that the key distinction is where transformation occurs—before or after loading—and provides five decision criteria for choosing the right approach based on data latency, raw data retention, compute costs, and business logic volatility.
This article details a three-stage big data pipeline—collection, computation, and application—used by a ride-hailing platform to build user and vehicle profiles, enable intelligent order matching that boosts click-through rates by 30%, and create a self-reinforcing data flywheel that continuously improves service quality.
This article explains how to capture and stream row‑level changes from PostgreSQL using its Write‑Ahead Log, compares the approach with MySQL binlog CDC, and provides a step‑by‑step guide—including configuration, logical decoding, snapshot handling, replication slots, and production‑grade correctness considerations—to build a reliable CDC pipeline.
EgoSteer is an open‑source full‑stack framework that transforms massive first‑person human‑hand videos into high‑quality training data via the EgoSmith pipeline, trains a flow‑matching VLA augmented with a lightweight world‑model expert, and integrates a unified action space and real‑time chunking to enable dual dexterous hands to follow natural language commands, achieving up to 75 % success across 40 diverse manipulation tasks.
The article explains why a production‑grade enterprise knowledge base requires more than just dumping PDFs into a vector store, detailing a distributed, event‑driven architecture with separate collection, processing, retrieval, and governance layers that handle high‑concurrency crawling, real‑time cleaning, versioned indexing, permission filtering, and feedback‑driven updates.
The article outlines an end‑to‑end industrial pipeline for large‑model training data, detailing six concrete steps for raw web text processing, multimodal VQA/caption/interleave/audio generation methods, and a data‑engineering backbone that uses agentic pipelines, unified storage, and a post‑training production platform to ensure high‑quality, verifiable data.
At the 2026 Volcano Engine FORCE conference, TalkingData outlined industry AI agent trends, high‑quality data set construction, and data compliance, then announced a deep three‑pronged collaboration with Volcano Engine—covering AI‑plus‑cloud large‑model bases, custom data agents, and AI‑driven data pipelines—backed by metrics showing over 30% cost reduction, 100% resource‑scaling improvement, and a 215% boost in data processing speed.
The article explains why a closed‑loop feedback system is essential for recommendation engines, describes the end‑to‑end architecture from client‑side event collection through gateways, message queues, preprocessing, data warehousing, and traceability, and details how the resulting signals drive user profiling, feature generation, sample creation and model training while addressing practical engineering challenges.
The article presents DataFlow, an open‑source, GPU‑centric data‑engineering framework that tackles LLM data‑preparation bottlenecks by defining a two‑level operator taxonomy, a LLM‑driven WebAgent for automatic crawling, a PDF‑to‑Markdown MinerU, a Ray‑based distributed runtime, and extensive multimodal extensions, and validates the design with quantitative experiments showing significant quality gains across math, code, and reasoning benchmarks.
This article walks through setting up the foundational environment for a real‑time data visualization pipeline using RocketMQ, Flink, Elasticsearch and Kibana, covering code configuration, index verification, Kibana dashboard creation, and practical production tips.
In the AI boom, product managers who can coordinate front‑end, back‑end, algorithm, data cleaning and compute resources and master reverse‑engineering, rapid execution, and patient problem‑solving command six‑figure salaries, as illustrated by refund‑strategy redesign, custom AI客服 deployment, and complex 3D point‑cloud labeling pipelines.
OpenHuman’s Memory Tree architecture separates hot and cold data paths, uses content‑addressed IDs, and builds layered summaries, offering low‑latency queries and robust idempotency for AI agents that need continuous background learning.
Fresha replaced its Postgres‑Snowflake‑MSK pipeline with an AutoMQ‑based Diskless Kafka message layer and StarRocks for real‑time analytics, cutting storage costs 17‑20×, dropping query latency from seconds to sub‑second, and migrating ~1,000 topics in a week with zero downtime.
At QECon Shenzhen 2026, Xiaohongshu's tech team will present five technical talks that showcase how AI agents are applied to architecture risk analysis, change automation, large‑model load‑testing data construction, end‑to‑end testing, and client‑side performance, illustrating concrete engineering solutions and measurable productivity gains.
KaptureCX migrated its core analytics from ClickHouse to StarRocks, introduced RisingWave and Kafka for CDC, and achieved millisecond‑level query latency, a reporting cycle cut from weeks to one day, and a solid data foundation for AI‑driven services.
The article dissects a credit data system architecture, detailing six logical layers—from multi-source data collection and feature engineering (including graph features and feature stores) to model training, real‑time stream processing, decision engine integration, and privacy‑preserving computation—while explaining the trade‑offs, tools, and performance targets needed for accurate, low‑latency risk assessment.
The article details KAI, a 173 cm, 115‑DOF humanoid robot with tactile skin and a custom battery, and explains how its ultra‑human form, massive first‑person data collection, and three‑stage training pipeline are intended to enable a world‑model‑driven embodied AI system, while also acknowledging the engineering and market challenges ahead.
This article details a step‑by‑step, distributed pipeline built on Alibaba Cloud PAI using Data‑Juicer and Ray that transforms raw egocentric hand videos into LeRobot v2.0‑compatible Vision‑Language‑Action (VLA) training data, covering video splitting, frame extraction, camera calibration, 3D hand reconstruction, pose estimation, action captioning, and export, with code snippets, performance numbers, and references.
This article details a step‑by‑step guide for constructing a high‑performance multimodal data pipeline—covering video segmentation, duration filtering, frame extraction, safety and aesthetic scoring, and caption generation—using Alibaba Cloud PAI, Paimon, DataJuicer, and distributed frameworks like Ray and Daft, with real‑world performance metrics.
This article presents a comprehensive, step‑by‑step framework for turning laboratory AI models into production‑ready systems, covering capability mapping, technology stack choices, model selection, prompt engineering, data pipelines, training strategies, and cross‑team collaboration to ensure stability, observability, and trustworthiness.
This article outlines a comprehensive, six‑part model engineering framework that transforms AI capabilities into reusable business functions, defines a stable technical stack, establishes model selection and architecture guidelines, implements rigorous control, data, and training processes, and explains how these layers synergize for reliable, scalable deployment.
Despite ever‑more capable large language models, AI agents frequently stumble because enterprise data is messy, pipelines introduce errors, RAG lacks timeliness and conflict resolution, and context assembly requires dedicated ingestion, resolution, selection, decay, and inference layers, plus a harness to manage execution and governance.
This article introduces the Flow PHP data‑processing framework, highlights its ultra‑low memory footprint and extensible pipeline capabilities, and provides step‑by‑step installation and code examples for handling in‑memory arrays and CSV files in ETL workflows.
The article outlines a full‑stack data architecture for 2026, covering edge‑preprocessed collection, unified observability with OpenTelemetry, CDC‑based real‑time sync, a Kafka‑Flink‑Iceberg processing stack, Data Mesh governance, and AI‑Agent powered insight, while offering practical adoption paths and pitfalls to avoid.
The article explains how to decide between incremental and full data synchronization based on business patterns, describes three table types, outlines critical incremental‑sync details such as checkpointing, transaction integrity, large‑batch handling and log retention, and shows how to merge increments into full snapshots in a data‑warehouse pipeline.
The article presents CharacterFlywheel, a 15‑generation flywheel methodology that iteratively improves social‑dialogue LLMs in production using data‑driven reward models, rejection sampling, and a mix of SFT, DPO, and RL, with detailed experiments and best‑practice insights.
Starbucks China’s logging team migrated several petabytes of logs from legacy VM‑based Elasticsearch clusters to a cloud‑native bare‑metal Kubernetes platform, upgrading ES from 7.x to 8.x, containerizing components, optimizing storage and Kafka, and achieving up to 80% query speed gains, 30% CPU reduction, and 200% write‑throughput improvement.
Although Spark excels at large‑scale, complex computations and offers high‑level APIs like DataFrame and SQL, real‑world projects often require additional tools for data ingestion, task scheduling, and simple streaming, so combining Spark with a data‑integration platform such as FineDataLink yields a more efficient and maintainable pipeline.
Facing Postgres overload and costly Snowflake queries, Fresha rebuilt its analytics platform by introducing StarRocks as a unified SQL entry point, combining federated lakehouse queries with high‑performance internal tables, which reduced homepage query latency to around 200 ms and achieved minute‑level data freshness across real‑time, historical, and search workloads.
This article explains five streaming ETL patterns—filter, route, transform (extract, normalize, anonymize), aggregate, and trigger—showing how each can be implemented with SQL statements and illustrating their use cases with concrete code examples.
UNO‑Bench, an open‑source benchmark from Meituan’s LongCat team, provides the first high‑quality, low‑redundancy unified evaluation framework for omni‑modal large language models, featuring 1,250 manually annotated cross‑modal samples and 2,480 enhanced single‑modal samples covering 44 fine‑grained tasks and five modality combinations.
This article introduces DMS Airflow, an enterprise‑level data workflow orchestration platform built on Apache Airflow, covering its advanced DAG capabilities, deep DMS integration, scheduling, task dependency management, dynamic task generation, resource scaling, security features, and practical code examples for SQL, Spark, DTS, and Notebook tasks.
This article explores Python's __missing__ dunder method, showing how it surpasses traditional approaches like if/else, .get() and defaultdict by enabling dynamic, self‑healing dictionary behavior, and demonstrates advanced real‑world applications such as smart counters, infinite nested dicts, API caching, and automatic data pipelines.
In this detailed presentation, Baidu’s engineering manager Yang Jingwei explains the current landscape, emerging trends, key challenges, data pipelines, model training, prompt engineering, multi‑platform support, and future outlook of Baidu’s intelligent code assistant and AI IDE, illustrating practical solutions and real‑world impact.
Zepto transformed its brand‑analytics platform from a Postgres MVP into a production‑grade, sub‑second real‑time analytics solution by adopting StarRocks, redesigning its data pipeline with Databricks, Kafka, and Flink, and choosing a storage‑compute architecture that supports massive joins and rapid insights.
This case study details how Hypergryph leveraged Alibaba Cloud EMR Serverless StarRocks, Flink, and Kafka to replace a ClickHouse data warehouse with a high‑performance, elastic, and easy‑to‑operate real‑time analytics platform that dramatically improved query speed, stability, operational efficiency, and cost for their gaming business.
This article explores the convergence of big data engineering and platform engineering, detailing how platform engineering methodologies solve big data's complexity challenges through unified resource pooling, standardized pipelines, observability, and governance, delivering faster deployment, 30-50% resource utilization gains, and reduced MTTR.
This article explains how to define and objectively evaluate the quality of AI‑generated product images for e‑commerce by decoupling assessment from the generation pipeline, selecting quantifiable metrics such as CLIPScore and Inception Score, building a lightweight evaluation system, cleaning and labeling data, and validating the approach with real‑world business and model datasets.
This article introduces Vector, an open‑source, Rust‑based observability data pipeline that outperforms traditional tools like Logstash and Fluentd, covering its core features, concepts, installation script, minimal configuration example, and how it handles events, logs, metrics, and traces.
The latest SPL update introduces powerful operators like pack-fields, log-to-metric, and metric-to-metric, delivering dramatic performance gains, richer data transformation capabilities, and enhanced observability for cloud‑native log processing pipelines.
This article explains how Observability 2.0 reshapes log, metric and trace management by unifying health views, introduces the evolution of Alibaba Cloud's SLS data pipeline, compares its three service modes, and demonstrates performance, cost and integration benefits for large‑scale, real‑time log processing.
This article details a closed-loop Data Lifecycle Management (DLM) framework covering six stages—requirement confirmation, model design, development, testing, monitoring, and archiving—with governance practices, toolchains, and a bank case study showing reduced rework, storage costs, and model drift.
This article analyzes DataOps as a deep fusion of data analytics, lean thinking, agile practices, and DevOps culture, contrasting it with DevOps, detailing its four genetic components with case studies, explaining its emergence from technical, business, and organizational drivers, examining current hiring fragmentation and talent gaps, and predicting explosive job growth driven by policy, tooling maturity, and talent supply reforms.
Observability 2.0 unifies logs, metrics and traces into a single platform, introduces event‑centric Wide Events, and drives a complete redesign of Alibaba Cloud's SLS data pipeline that delivers higher performance, lower latency, richer low‑code SPL processing, and up to a 66.7% reduction in processing costs.
This guide outlines a comprehensive data development roadmap, covering infrastructure setup, governance frameworks, automated pipelines, BI and analytics tools, AI/ML integration, cultural adoption, and continuous performance monitoring to enable intelligent business transformation.
The article outlines the design and implementation of a proactive alerting system for Taobao Special Edition, covering five alert categories—slot expiration, rights issues, configuration platforms, experiment audience expiration, and public‑opinion problems—detailing data‑driven rule engines, flexible integration, and successful 24‑hour inventory alerts while planning minute‑level rapid‑consumption warnings.
This article details how the supply‑chain big‑screen dashboard for Double‑11 maintains high stability by mapping the full data‑flow, identifying risk points across ingestion, processing, storage and service layers, and applying comprehensive technical safeguards such as high‑availability design, fault‑tolerance, monitoring, and coordinated operational procedures.
The Bilibili TTV team optimized OpenSora and CogVideoX text‑to‑video models by redesigning data storage with Alluxio, parallelizing VAE encoding, applying dynamic sequence‑parallel and DeepSpeed‑Ulysses attention, adapting GPU code for NPU execution, leveraging profiling‑driven kernel fusion, FlashAttention, and expandable memory to dramatically increase training efficiency and frame throughput, while outlining future pipeline‑parallel and ZeRO‑3 scaling plans.
This article explains why traditional databases like MySQL struggle with massive datasets, introduces ElasticSearch’s inverted‑index architecture, and details a practical pipeline using Hive, wide tables, binlog, Canal, and Otter to achieve near real‑time indexing for petabyte‑level data.
This article explains the concepts, models, architecture, data management, project configuration, engine design, and error‑handling procedures needed to implement an automated, systematic reconciliation system that can handle large‑scale online transaction volumes with high accuracy and efficiency.
This tutorial guides you through installing Apache Airflow, initializing its database, starting the web server and scheduler, creating a Python DAG that reads, cleans, groups, and saves CSV data, configuring the DAG directory, and monitoring the pipeline via the Airflow web UI.
This article details Bilibili's end‑to‑end front‑end error monitoring solution, covering the custom SDK, error capture and classification, unique ID generation, filtering, white‑screen detection, data pipelines, APM visualisation, lifecycle plugins, one‑click alerts, and future roadmap, all backed by real‑world metrics and code examples.
The article recounts how the author built a billion‑scale AI image‑asset library by replacing a week‑long import with a clustered‑table, sharded pipeline, MD5‑based unique keys, a custom DataWorks task scheduler, and multi‑engine query layers, sharing practical engineering practices learned through successive iterations.
This article details the end‑to‑end process of pre‑training, fine‑tuning, and evaluating a 1‑billion‑parameter Chinese LLM named Steel‑LLM on limited hardware, covering data collection, pipeline design, training framework choices, architectural tweaks, performance results, and practical lessons for resource‑constrained developers.
The author builds a fully AI‑driven pipeline that scrapes JD.com comments about an Elasticsearch book, processes the data through cleaning and preprocessing, indexes it into Elasticsearch, and creates a series of Kibana visualizations, while reflecting on model selection and practical challenges.
The article presents a decoupled business operation log management architecture that uses Alibaba’s Canal to capture MySQL binlog changes, streams them through Kafka, and stores structured before‑and‑after records in Elasticsearch with nested mappings, enabling multi‑table correlation via transaction IDs, visual querying, and reliable rollback without modifying application code.
This article walks through the design and implementation of a decoupled, high‑performance business operation log solution that captures MySQL binlog changes via Canal, streams them through Kafka, and stores and queries them in Elasticsearch, addressing challenges such as batch operations, multi‑table transactions, and non‑business data filtering.
This article details the design and deployment of an intelligent alert attribution system for Ctrip Hotel's front‑end, describing the background challenges, the unified data pool, weighted alert rules, three attribution algorithms, achieved improvements in accuracy and troubleshooting speed, and future enhancement plans.
This article introduces Logstash’s core configuration syntax, explains key filter plugins such as grok, mutate, date, ruby, and aggregate, demonstrates conditional processing and multi‑event handling, and provides practical code examples to help readers efficiently parse, transform, and route log data.
This article details the three‑stage evolution of Ctrip's vacation product log system—from a simple single‑table DB approach, through a platform‑based ES + HBase solution, to a scalable V3.0 architecture that improves storage, search, and business empowerment while handling billions of log entries.
This guide walks through the challenges of ingesting high‑volume sales‑point data into ClickHouse, explains why a buffering layer like Kafka is essential, compares three integration approaches—ClickHouse Kafka engine, Kafka Connect, and an external writer—and provides concrete SQL and Terraform examples to help you choose the right solution.
This article introduces Baidu's log platform that handles billions of daily events, explains UBC logging concepts and monitoring requirements, and details a low‑cost, high‑accuracy architecture using real‑time streaming, dimension mapping, watermarking, and time‑window aggregation to achieve reliable, scalable event monitoring.
The article analyzes the rapid rise of open‑source large language models, explains how Llama 3 serves as a strong base for domain‑specific models, details a data‑driven pipeline, fine‑tuning, reinforcement learning, engineering optimizations, and a comprehensive evaluation framework, and showcases the XuanYuan series that outperforms GPT‑4 on several finance benchmarks.
This article details the design and implementation of a high‑availability, real‑time logistics supply‑chain dashboard, covering Flink‑based data pipelines, ClickHouse OLAP storage, metric consistency, stability measures, extensible configuration, and comprehensive monitoring to ensure accurate, scalable performance during major promotions.
This article describes the design and implementation of a high‑availability, real‑time logistics supply‑chain dashboard using Flink and ClickHouse, covering data processing pipelines, metric consistency, stability mechanisms, extensible configurations, and monitoring techniques to guide similar large‑screen projects.
This article presents the design and implementation of a high‑availability, real‑time logistics supply‑chain monitoring dashboard, covering its data processing pipeline with Flink, storage choices between Elasticsearch and ClickHouse, multi‑layer architecture, metric consistency, stability mechanisms, extensibility configurations, and monitoring practices.
The Feature Center in iQIYI’s Opal platform centralizes feature creation, storage, and real‑time access through a drag‑and‑drop DAG workflow and DSL‑driven transformations, handling massive QPS and low‑latency demands while enabling fast business iteration, cross‑team reuse, and monitoring for advertising, recommendation, and risk‑control applications.
This article explains Baidu's log platform architecture, the UBC event‑tracking protocol, monitoring requirements, and the low‑cost, high‑accuracy solutions—including dimension mapping, watermark handling, data trimming, and time‑window aggregation—that enable real‑time, customizable monitoring of petabyte‑scale log streams.
This article details the background, architecture, data preparation methods, massive‑data handling strategies, verification processes, and error‑handling mechanisms of ZuanZuan's channel reconciliation system, highlighting design choices such as binlog ingestion, task‑driven bill downloads, sharding with Hive archiving, and MQ‑based reconciliation to ensure financial data consistency and safety.
This article details how Alluxio was adopted to replace NAS in autonomous driving model training, describing the data closed‑loop workflow, the challenges of the previous system, Alluxio's architectural benefits, deployment strategies across single and multiple data centers, functional and performance testing, operational tuning, and the resulting cost and efficiency gains.
This article walks through the end‑to‑end lifecycle of a SQL task in a big‑data environment, covering creation, scheduling metadata, instance generation, resource allocation, ODPS execution, and final processing on the Fuxi distributed engine.
The article details NetEase Cloud Music’s end‑to‑end, online‑configurable data‑consumption service and schema‑driven visualization platform that transform raw client logs into ClickHouse records, automatically generate tables and dashboards, and provide observability, dramatically reducing manual effort while supporting over twenty performance metrics for frontend monitoring.
This article presents a comprehensive overview of risk control model building for online small‑loan scenarios, covering pre‑loan, in‑loan and post‑loan stages, the associated data pipelines, model deployment strategies, optimization attempts, and monitoring frameworks to ensure accuracy, stability and effectiveness.
The article describes how an advertising retrieval system evolved from serial to parallel full builds and finally to a hybrid incremental indexing approach that records direct entity relationships during assembly, enabling fast reverse‑lookup of changed units via inverted indexes, reducing database load, latency, and rebuild overhead.
This article outlines the importance of event tracking for data, describes iQIYI's five‑stage tracking system evolution, analyzes the challenges of the self‑service phase, presents the middle‑platform improvements, explains the migration strategy, and details the downstream data lake, real‑time stream, and data‑warehouse processing workflows.
This article details ByteDance's end‑to‑end event‑tracking cost governance, covering background, strategies, large‑scale data pipelines, resource challenges, control mechanisms, automated and supervised governance modes, and the substantial savings achieved through point filtering, tiered prioritization, and sampling.
Shuhe Technology replaced a cumbersome Kafka‑to‑ECS/K8s log‑processing pipeline with Alibaba Cloud Function Compute, achieving faster, more elastic, and cost‑effective handling of massive application logs while reducing operational overhead and simplifying maintenance.
This article explores how Kafka serves as a versatile backbone for messaging, durable storage, log aggregation, monitoring, commit logs, recommendation pipelines, stream processing, CDC, system migration, and event sourcing, highlighting its performance, reliability, and practical deployment patterns.
This article presents a metadata‑driven approach to improve data warehouse timeliness by extracting upstream lineage, identifying over‑layered, duplicate, and critical‑path tasks, and applying targeted scheduling and code‑level optimizations, demonstrated with a hotel order wide‑table case study.
This article details Agoda's evolution of Apache Kafka usage—from a two‑step logging architecture that separates developer concerns, through cluster layout, scaling metrics, monitoring and audit pipelines, to cost attribution, authentication, ACLs, and automation tools—highlighting trade‑offs and operational lessons learned.
This article presents a comprehensive overview of Zhihu's event‑tracking system, covering its evolution from early Hadoop‑based pipelines to cloud‑native architectures, detailing toolsets for requirement management, validation, data collection, querying, and service design, and concluding with a practical Q&A on best practices and optimization.
This report details the migration of Dada's unified data‑collection platform across 43 sites, outlining the achieved cost reductions, data‑analysis benefits, migration workflow, architectural design, technical highlights, and the challenges and solutions encountered during the project.
This article details Huolala's user event tracking (埋点) system, covering its background, challenges, the construction of a four‑module management platform, backend SDK design, monitoring and quality assurance mechanisms, and future plans for service integration, data lineage, and governance optimization.
This article explains how Volcano Engine identified the tangled authorization, data‑fetching, and performance problems of its advertising AB‑testing platform and refactored it by splitting services, redesigning the data model with MySQL and ClickHouse, applying DAG scheduling, time‑wheel algorithms, Domain‑Driven Design, and rigorous unit testing to achieve a more stable, extensible backend solution.
This article presents a comprehensive overview of the challenges, architectural evolution from version 1.0 to 3.0, core components, key technologies, and future directions of JD's real‑time risk insight platform, highlighting data integration, streaming processing, plugin mechanisms, and intelligent anomaly detection.