BestHub
Sign in
Tagged articles

Data Pipeline

258 articles · Page 3 of 3
Qunar Tech Salon
Qunar Tech Salon
Feb 21, 2020 · Artificial Intelligence

Building an End‑to‑End Data‑Model Loop for Alibaba XiaoMi AI Services

The article describes how Alibaba's XiaoMi AI platform constructs a closed‑loop pipeline—from data collection and annotation to model training, evaluation, and real‑time deployment—using multi‑dimensional data processing, visualization, and Spark‑based engines to accelerate iterative improvements and address operational pain points.

AIData PipelineSpark
0 likes · 9 min read
Building an End‑to‑End Data‑Model Loop for Alibaba XiaoMi AI Services
DataFunTalk
DataFunTalk
Feb 17, 2020 · Artificial Intelligence

Building a Closed‑Loop AI System: From Data Collection to Model Deployment in Alibaba’s XiaoMi

This article explains how Alibaba’s XiaoMi team constructs a full‑cycle AI pipeline—covering real‑time and offline data processing, high‑dimensional visualization, model training, iterative feedback, and Spark‑based deployment—to accelerate intelligent product iteration while addressing common engineering pain points.

AIData PipelineSpark
0 likes · 10 min read
Building a Closed‑Loop AI System: From Data Collection to Model Deployment in Alibaba’s XiaoMi
dbaplus Community
dbaplus Community
Jan 12, 2020 · Big Data

How Xiaomi Achieved Real‑Time MySQL‑to‑Kudu Sync with Binlog and Talos

Facing MySQL performance bottlenecks at massive scale, Xiaomi built the LCSBinlog service that captures MySQL binlog events, streams them through the Talos platform, and writes to Kudu for real‑time BI, detailing architecture, job scheduling, consistency guarantees, use cases, and troubleshooting lessons.

BinlogCDCData Pipeline
0 likes · 13 min read
How Xiaomi Achieved Real‑Time MySQL‑to‑Kudu Sync with Binlog and Talos
dbaplus Community
dbaplus Community
Nov 21, 2019 · Databases

How to Build a Real‑Time MySQL Statistics Platform with ClickHouse

This article explains how a growing company designed, optimized, and deployed a comprehensive MySQL monitoring and analysis pipeline—moving from Flume‑HDFS‑Hive to ClickTail‑ClickHouse, enriching SQL parsing, and applying practical methods for state statistics, trend analysis, permission management, and data‑skew detection.

DBAData PipelineDatabase Monitoring
0 likes · 16 min read
How to Build a Real‑Time MySQL Statistics Platform with ClickHouse
DataFunTalk
DataFunTalk
Nov 7, 2019 · Big Data

Real-Time Computing Engine at Beike: Architecture, Practices, and Future Plans

This article details Beike's real‑time computing engine, covering its background, streaming platform built on Spark Streaming and Flink, data ingestion via Kafka, metadata handling, SQL‑based task development, monitoring, storage solutions, and future roadmap for resource management and AI‑enhanced monitoring.

Data PipelineFlinkKafka
0 likes · 14 min read
Real-Time Computing Engine at Beike: Architecture, Practices, and Future Plans
Big Data Technology & Architecture
Big Data Technology & Architecture
Oct 30, 2019 · Big Data

Building a Real‑Time Data Processing Pipeline with Apache Kafka, Spark Streaming, and Cassandra

This tutorial explains how to create a highly scalable, fault‑tolerant real‑time data processing platform by configuring a Kafka topic, a Cassandra keyspace, adding Spark and connector dependencies, developing a Java‑based Spark Streaming pipeline, enabling checkpoints, and deploying the application with spark‑submit.

CassandraData PipelineJava
0 likes · 8 min read
Building a Real‑Time Data Processing Pipeline with Apache Kafka, Spark Streaming, and Cassandra
Alibaba Cloud Developer
Alibaba Cloud Developer
Oct 30, 2019 · Big Data

How Real-Time Big Data Pipelines Detect E‑Commerce Ad Misplacements

This article explains how a large‑scale e‑commerce search advertising system uses real‑time big‑data pipelines, log synchronization, NoSQL storage, and proactive verification to automatically discover and correct ad placement errors across the entire data processing chain, protecting both advertisers and the platform.

Data Pipelinead verificationbig data
0 likes · 13 min read
How Real-Time Big Data Pipelines Detect E‑Commerce Ad Misplacements
58 Tech
58 Tech
Sep 6, 2019 · Big Data

Architecture and Technical Implementation of the WMDA Data Analytics Platform

The article details WMDA's end‑to‑end data analytics architecture, covering zero‑event data collection, real‑time and offline processing pipelines built on Spark Streaming, Druid, Hadoop, Kettle, and TaskServer, and explains how these components collaborate to deliver comprehensive user behavior analysis.

Data PipelineDruidETL
0 likes · 11 min read
Architecture and Technical Implementation of the WMDA Data Analytics Platform
Xueersi Online School Tech Team
Xueersi Online School Tech Team
Sep 6, 2019 · Big Data

Real-Time Data Architecture, Evolution, and Applications at an Online School

The article details the six‑layer big‑data architecture of an online school, chronicles its migration from Storm to Spark Streaming and finally to Flink, and showcases concrete real‑time applications such as gateway monitoring, user‑profile tagging, renewal reporting, and advertising analysis, while outlining future development directions.

AnalyticsBig Data ArchitectureData Pipeline
0 likes · 14 min read
Real-Time Data Architecture, Evolution, and Applications at an Online School
Ctrip Technology
Ctrip Technology
Sep 4, 2019 · Artificial Intelligence

Design and Implementation of Ctrip's User Precise Marketing System

This article details the design goals, architecture, core functionalities, and optimization strategies of Ctrip's user precise marketing system, which leverages RESTful integration, flexible rule-based and machine‑learning models, real‑time monitoring, and AB testing to improve traffic utilization and conversion rates.

AB TestingCtripData Pipeline
0 likes · 11 min read
Design and Implementation of Ctrip's User Precise Marketing System
Xianyu Technology
Xianyu Technology
Aug 28, 2019 · Big Data

Unified Search System Architecture and Automation for Multiple Business Scenarios

To avoid building separate search services for each Xianyu business, the team created a unified, generic search architecture based on Alibaba’s HA3 engine and a control layer that automates data dumping, indexing, query translation, and result ranking across five subsystems, enabling new services to be onboarded in minutes instead of weeks.

Data PipelineIndexingautomation
0 likes · 18 min read
Unified Search System Architecture and Automation for Multiple Business Scenarios
Youzan Coder
Youzan Coder
Aug 14, 2019 · Big Data

Comprehensive Guide to Data Collection, Event Modeling, and Tracking in Big Data Platforms

The guide explains how comprehensive data collection in big‑data platforms relies on a standardized event model, passive and code‑based embedding, multi‑platform SDKs, a log‑middleware layer, precise location tracking, and an embedding management platform that supports workflow, testing, quality monitoring, and scalable infrastructure for future enhancements.

AnalyticsData PipelineLog Processing
0 likes · 19 min read
Comprehensive Guide to Data Collection, Event Modeling, and Tracking in Big Data Platforms
ITPUB
ITPUB
Jul 2, 2019 · Databases

How ClickHouse Powers Ctrip’s Hotel Data Platform for Billions of Daily Updates

This article explains how Ctrip’s hotel data intelligence platform handles over ten billion daily data updates and nearly a million queries by adopting ClickHouse, detailing the system's background, the reasons for choosing ClickHouse over other solutions, the data ingestion pipelines, monitoring strategies, operational practices, and performance outcomes.

ClickHouseData PipelineReal-time Analytics
0 likes · 13 min read
How ClickHouse Powers Ctrip’s Hotel Data Platform for Billions of Daily Updates
Ctrip Technology
Ctrip Technology
Jun 26, 2019 · Databases

Applying ClickHouse for a High‑Performance Hotel Data Intelligence Platform

This article describes how Ctrip Hotel's data intelligence platform leverages ClickHouse to achieve real‑time analytics on billions of daily updates and millions of queries, detailing the system architecture, data ingestion pipelines, monitoring, and operational lessons learned for large‑scale, high‑availability data services.

Data PipelineReal-time Analyticsdata warehouse
0 likes · 12 min read
Applying ClickHouse for a High‑Performance Hotel Data Intelligence Platform
58 Tech
58 Tech
May 31, 2019 · Artificial Intelligence

Summary of 58 Group Technical Salon: Recommendation System Architecture and Search Ranking Algorithm Practices

The article summarizes the 58 Group technical salon where experts presented the microservice‑based recommendation system architecture, data and strategy layers, and the internally built search ranking platform covering sampling, feature engineering, and model training, highlighting practical implementations and lessons learned.

AIData Pipelinemachine learning
0 likes · 7 min read
Summary of 58 Group Technical Salon: Recommendation System Architecture and Search Ranking Algorithm Practices
NetEase Media Technology Team
NetEase Media Technology Team
May 16, 2019 · Backend Development

Design and Implementation of a Configurable, Extensible Content Processing System (Apollo)

Apollo is a configurable, extensible content‑processing platform that models each step as a node defined in a configuration file, supports multiple implementations for A/B testing, decouples producers and consumers via Kafka, ensures fault‑tolerant retries and replay, captures fine‑grained metrics through Canal‑to‑TiDB pipelines, and cuts new‑type development effort to roughly ten percent of the original cost while delivering high‑quality data to downstream teams.

Data PipelineKafkaTiDB
0 likes · 9 min read
Design and Implementation of a Configurable, Extensible Content Processing System (Apollo)
HomeTech
HomeTech
Jan 18, 2019 · Big Data

Data Mill: A Real‑Time Spark Streaming Framework for DSP Business Support

Data Mill is a Spark‑Streaming‑based real‑time computation framework that abstracts tasks as DataFrames, enables SQL‑driven development, and supports DSP business requirements by reducing latency to 15‑30 minutes while providing a scalable architecture, caching strategy, and automated fault handling.

DSPData PipelineReal-Time Computing
0 likes · 10 min read
Data Mill: A Real‑Time Spark Streaming Framework for DSP Business Support
DataFunTalk
DataFunTalk
Dec 20, 2018 · Artificial Intelligence

How to Build World-Class Visual AI Technology

This presentation outlines the fundamentals of computer vision, discusses key factors such as algorithm research, large‑scale training platforms, intelligent data processing, and hardware optimization, and shares practical experiences from DeepGlint on building a world‑class visual AI system and its real‑world applications.

Data Pipelinecomputer visionhardware optimization
0 likes · 23 min read
How to Build World-Class Visual AI Technology
JD Tech
JD Tech
Oct 10, 2018 · Backend Development

Design and Architecture of JD's Virtual Order Center (Hamal)

The article explains the architecture and core mechanisms of JD's Virtual Order Center, describing how the Hamal service leverages MySQL binlog listening, Zookeeper coordination, fast TCP‑based consumption, read‑write separation, and multi‑level search to reliably process billions of virtual orders.

BackendBinlogData Pipeline
0 likes · 7 min read
Design and Architecture of JD's Virtual Order Center (Hamal)
Big Data and Microservices
Big Data and Microservices
Sep 3, 2018 · Big Data

From Raw Data to Business Impact: A Complete Data Analyst Skill Guide

The article outlines a comprehensive data‑analyst competency framework, covering data collection, storage, extraction, mining, analysis, visualization, and practical application, and provides concrete questions, techniques, and tool recommendations to help analysts turn raw data into actionable business insights.

Business IntelligenceData AnalysisData Pipeline
0 likes · 9 min read
From Raw Data to Business Impact: A Complete Data Analyst Skill Guide
Ctrip Technology
Ctrip Technology
Jul 24, 2018 · Backend Development

Design and Implementation of CTran V3: A Multilingual Translation Platform for Ctrip International Business

This article presents a comprehensive case study of CTran V3, a redesigned multilingual translation platform for Ctrip's international business, detailing its architecture, data flow, job scheduling, translation engine, real‑time services, and lessons learned to guide similar large‑scale content localization projects.

BackendData PipelineJob Scheduling
0 likes · 21 min read
Design and Implementation of CTran V3: A Multilingual Translation Platform for Ctrip International Business
Dada Group Technology
Dada Group Technology
Jul 24, 2018 · Operations

Building a Scalable Growth Operations Platform: User Grouping, Dynamic Queries, and Automation

The article describes how a growth operations team can improve efficiency by designing a flexible user‑grouping system, dynamic query generation, and automated rule execution, while addressing data latency, real‑time processing, and scalability challenges through a Lambda‑style architecture.

Data PipelineDynamic QueryLambda Architecture
0 likes · 14 min read
Building a Scalable Growth Operations Platform: User Grouping, Dynamic Queries, and Automation
Efficient Ops
Efficient Ops
Jun 6, 2018 · Big Data

How Tencent’s Multi‑Dimensional Monitoring Turns Big Data Into Real‑Time Business Insights

This article explains how Tencent’s ZhiYun multi‑dimensional monitoring system evolves from the Mobile Monitor platform, outlines its design principles, data‑factory capabilities, storage choices, and intelligent features, and demonstrates how it enables real‑time, multi‑dimensional analysis and alerting for large‑scale business operations.

Data PipelineDruidStorm
0 likes · 11 min read
How Tencent’s Multi‑Dimensional Monitoring Turns Big Data Into Real‑Time Business Insights
Java Captain
Java Captain
May 24, 2018 · Big Data

Debugging a Kafka Data Drop: A Step‑by‑Step Troubleshooting Case Study

After a recent feature release caused a sharp decline in a key data metric, the team followed a systematic, fourteen‑step troubleshooting process—including verification, code review, DBA involvement, local debugging, environment comparison, logging, packet capture, service restarts, request mode changes, load testing, and partition resizing—to identify and resolve a Kafka‑related throughput bottleneck.

Data PipelineKafkaLoad Testing
0 likes · 8 min read
Debugging a Kafka Data Drop: A Step‑by‑Step Troubleshooting Case Study
21CTO
21CTO
Apr 9, 2018 · Artificial Intelligence

How E‑Commerce Platforms Build Effective Product Recommendation Systems

This article explains the fundamentals and advanced techniques of e‑commerce product recommendation systems, covering conventional and personalized approaches, user profiling, data collection, storage, modeling, the three‑stage pipeline of preprocessing, recall and ranking, as well as system architecture, challenges, and key algorithms such as LR and GBDT.

Data Pipelinee-commercemachine learning
0 likes · 17 min read
How E‑Commerce Platforms Build Effective Product Recommendation Systems
Snowball Engineer Team
Snowball Engineer Team
Mar 23, 2018 · Big Data

Redesigning Snowball's Log Collection Architecture During Hadoop Cluster Expansion

The article details Snowball's challenges with a saturated CDH Hadoop cluster, outlines the limitations of the original Kafka‑based log pipeline, and explains how a comprehensive redesign using FlumeNG, Spillable Memory Channels, and custom HDFS sinks resolves latency, data loss, and high‑load issues while supporting future growth.

Cluster MigrationData PipelineFlumeNG
0 likes · 6 min read
Redesigning Snowball's Log Collection Architecture During Hadoop Cluster Expansion
Suning Technology
Suning Technology
Mar 9, 2018 · Big Data

How Suning Built a Scalable Real-Time Log Analysis Platform with Spark Streaming

Suning’s real‑time log analysis system integrates Flume, Kafka, Storm and Spark Streaming to collect, cleanse, and compute metrics like NDCG, ensuring low latency, high throughput, exact‑once processing, and robust data safety while supporting multi‑dimensional analytics on massive online‑offline traffic.

Data PipelineData QualityNDCG
0 likes · 12 min read
How Suning Built a Scalable Real-Time Log Analysis Platform with Spark Streaming
Meitu Technology
Meitu Technology
Dec 19, 2017 · Industry Insights

Inside Meitu’s In‑House Log Collection System Arachnia: Design, Challenges, and Core Mechanisms

This article introduces Meitu’s self‑developed log collection system Arachnia, explaining why a custom solution was needed for massive server‑side user‑behavior logs, the key requirements such as reliability and real‑time throughput, and the core architectural mechanisms that address those challenges.

ArachniaData PipelineLog Collection
0 likes · 2 min read
Inside Meitu’s In‑House Log Collection System Arachnia: Design, Challenges, and Core Mechanisms
Efficient Ops
Efficient Ops
Dec 18, 2017 · Operations

How WiFi Key Built a Million‑User Monitoring Platform: Architecture and Best Practices

This article describes how WiFi 万能钥匙 designed and implemented the Roma monitoring platform to handle billions of daily requests, covering background challenges, architectural principles, component design, data collection, transmission, storage, alerting, and future directions for large‑scale observability.

Data Pipelinearchitecturemicroservices
0 likes · 16 min read
How WiFi Key Built a Million‑User Monitoring Platform: Architecture and Best Practices
Meituan Technology Team
Meituan Technology Team
Dec 1, 2017 · Big Data

Metric Logic Tree: Automated Anomaly Analysis for Business Metrics

The Metric Logic Tree automates business metric anomaly analysis by integrating heterogeneous data sources (Kylin, MySQL, Elasticsearch, Druid) with a three‑layer architecture—metric calculation, algorithmic analysis (waterfall and Gini‑coefficient methods), and a master‑worker computation service—that parallelizes queries, delivers immediate conclusions, and shortens decision cycles, as demonstrated in Meituan‑Dianping’s hotel‑travel operations.

Data Pipelinealgorithmanomaly detection
0 likes · 7 min read
Metric Logic Tree: Automated Anomaly Analysis for Business Metrics
Efficient Ops
Efficient Ops
Nov 15, 2017 · Big Data

How Tencent Built a 10 TB‑Per‑Day Full‑Link Log Monitoring Platform

This article explains how Tencent's ZhiYun full‑link log monitoring platform handles massive daily logs, overcomes challenges of diverse log formats, high throughput, fault‑tolerant design, and provides scalable storage, query, and alerting capabilities for distributed micro‑service environments.

Data PipelineLog Monitoringbig data
0 likes · 10 min read
How Tencent Built a 10 TB‑Per‑Day Full‑Link Log Monitoring Platform
ITPUB
ITPUB
Sep 30, 2017 · Big Data

Designing Scalable Open‑Source ETL Systems: Lessons from Baidu Waimai

This talk details Baidu Waimai's end‑to‑end ETL design, covering demand sources, data flow patterns, multi‑stage system evolution, storage choices, scheduling architecture, configuration‑driven processing, quality monitoring, and how data lineage enables transparent, self‑service data delivery.

Data PipelineData QualityETL
0 likes · 25 min read
Designing Scalable Open‑Source ETL Systems: Lessons from Baidu Waimai
ITPUB
ITPUB
Sep 29, 2017 · Big Data

Designing an Open ETL System: Baidu Waimai’s Scalable Data Pipeline Practices

In this talk, a Baidu Waimai engineer explains the motivations, requirements, and architectural choices behind their open‑source ETL platform, covering data flow patterns, logical mappings, storage options, scheduling, metadata management, and quality monitoring to achieve scalable, transparent, and explainable data delivery.

Data PipelineETLbig data
0 likes · 26 min read
Designing an Open ETL System: Baidu Waimai’s Scalable Data Pipeline Practices
Meituan Technology Team
Meituan Technology Team
Sep 21, 2017 · Big Data

Feature Production Scheduling: Architecture Evolution and Core Technologies

Using Meituan‑Dianping’s hospitality online feature system as a case study, the article describes how feature production scheduling evolved from offline batch ETL to automated, metadata‑driven pipelines and sub‑second streaming, detailing the underlying architecture, incremental updates, storage abstraction, write‑shaving, atomicity, and recovery mechanisms.

Data Pipelinebig datafeature engineering
0 likes · 23 min read
Feature Production Scheduling: Architecture Evolution and Core Technologies
Architecture Digest
Architecture Digest
Sep 15, 2017 · Artificial Intelligence

Overview of Recommendation Systems: Goals, Methods, Architecture, and Practical Considerations

This article explains the objectives of recommendation systems, compares popular recommendation approaches, details the components and algorithms of personalized recommendation pipelines, and discusses practical challenges such as real‑time processing, freshness, cold‑start, diversity, content quality, and surprise handling.

Data Pipelinecold startevaluation
0 likes · 15 min read
Overview of Recommendation Systems: Goals, Methods, Architecture, and Practical Considerations
Architecture Digest
Architecture Digest
Sep 2, 2017 · Big Data

Designing a High‑Availability, High‑Efficiency Distributed Scheduling Platform for Big Data

This article examines the principles, features, and implementation details of distributed scheduling for big‑data ETL pipelines, covering decentralised schedulers, host selection strategies, fault‑tolerance, operator abstraction, elasticity, trigger mechanisms, visual monitoring, alarm handling, data fan‑in/fan‑out, parameter consistency, real‑time quality checks, lineage tracking, and field‑level traceability.

Data PipelineETLbig data
0 likes · 23 min read
Designing a High‑Availability, High‑Efficiency Distributed Scheduling Platform for Big Data
21CTO
21CTO
Aug 18, 2017 · Big Data

How Ctrip Builds a Scalable User Profile Platform for Personalized Travel

This article explains why Ctrip creates user profiles, describes the product and technical architectures, and details the data collection, computation, storage, high‑availability querying, and monitoring components that power its personalized travel recommendations and services.

CtripData Pipelinepersonalization
0 likes · 8 min read
How Ctrip Builds a Scalable User Profile Platform for Personalized Travel
High Availability Architecture
High Availability Architecture
Aug 8, 2017 · Big Data

Practical Big Data Architecture Evolution and Lessons Learned

The article reviews the evolution of big‑data architectures from a simple RDB‑centric pipeline to a SaaS‑based solution, highlighting common bottlenecks such as scaling, integration, cost, and operational complexity, and shares practical experiences and best‑practice recommendations for building efficient, maintainable data platforms.

Data PipelineSaaSarchitecture
0 likes · 12 min read
Practical Big Data Architecture Evolution and Lessons Learned
StarRing Big Data Open Lab
StarRing Big Data Open Lab
Jul 28, 2017 · Big Data

How Transwarp Transporter Enables Near‑Real‑Time ETL in Big Data Pipelines

The article introduces Transwarp Transporter, a near‑real‑time ETL tool for TDH 5.x, explains its architecture, visual dashboard, drag‑and‑drop data‑flow design, debugging features, parameter management, and highlights how it empowers business users to achieve fast, reliable data migration in big‑data environments.

Data IntegrationData PipelineETL
0 likes · 7 min read
How Transwarp Transporter Enables Near‑Real‑Time ETL in Big Data Pipelines
Architecture Digest
Architecture Digest
Jul 18, 2017 · Backend Development

Design and Implementation of Ctrip Real‑Time User Data Collection System

This article describes the design, technology selection, and performance evaluation of Ctrip's real‑time user behavior data collection platform, covering Netty‑based network handling, Kafka/Hermes messaging, encryption, compression, Avro backup, and related analytics products, with detailed feasibility analysis and benchmark results.

Data PipelineKafkaNetty
0 likes · 17 min read
Design and Implementation of Ctrip Real‑Time User Data Collection System
Qunar Tech Salon
Qunar Tech Salon
Jul 4, 2017 · Big Data

Design and Evolution of Airbnb's Log Data Storage and Query Platform

The article describes how Airbnb's data infrastructure team built a next‑generation log storage and query platform to improve data quality, timeliness, flexibility, and anomaly detection, outlining the system architecture, key requirements, five improvement areas, and the resulting benefits.

AirbnbData Pipelinelog platform
0 likes · 7 min read
Design and Evolution of Airbnb's Log Data Storage and Query Platform
Architects' Tech Alliance
Architects' Tech Alliance
May 7, 2017 · Big Data

Building a Complete Big Data Platform: From Hadoop Basics to Real‑Time Analytics

This guide walks beginners through the entire big‑data ecosystem—explaining the 4V characteristics, listing essential open‑source components, teaching Hadoop setup, Hive and SparkSQL usage, data ingestion with Sqoop, Flume and Kafka, task scheduling with Oozie, and real‑time processing with Storm and Spark Streaming.

Data PipelineHadoopHive
0 likes · 20 min read
Building a Complete Big Data Platform: From Hadoop Basics to Real‑Time Analytics
Meituan Technology Team
Meituan Technology Team
Mar 2, 2017 · Big Data

Meituan Waimai Feature Archive Platform: Architecture, Tag System, and Data Processing

Meituan Waimai’s Feature Archive platform processes billions of daily orders by managing ~200 user and 400 merchant tags through a three‑layer architecture—Hive, Elasticsearch, HBase, and MySQL—offering visual tag selection, instant self‑service queries, full data extraction, and a predicate‑logic query language, while supporting future extensibility.

Data PipelineElasticsearchHBase
0 likes · 14 min read
Meituan Waimai Feature Archive Platform: Architecture, Tag System, and Data Processing
21CTO
21CTO
Nov 6, 2016 · Artificial Intelligence

How to Build a Scalable AI-Powered Recommendation System with SOA

This article outlines a service‑oriented architecture for a high‑availability personalized recommendation platform, detailing the front‑end, back‑end, crawler, user‑profile modeling, data collection from logs and client events, and processing pipelines using technologies such as Node.js, Python, RabbitMQ/Kafka, MongoDB and TensorFlow.

Data PipelineFull-StackSOA
0 likes · 5 min read
How to Build a Scalable AI-Powered Recommendation System with SOA
Meituan Technology Team
Meituan Technology Team
Aug 5, 2016 · Big Data

Design and Implementation of a Large-Scale User Behavior Analytics Platform

The article outlines Meituan‑Dianping’s “Sensors Analytics” platform, a privately‑deployed, open‑PaaS solution that collects full‑stack user events from iOS, Android, Web and WeChat, maps IDs in near real‑time, stores detailed records in Kudu (real‑time) and Parquet (offline), and serves low‑latency queries via Impala, addressing the architectural and operational challenges of high‑throughput ingestion and data‑security requirements.

Data PipelineImpalaKafka
0 likes · 8 min read
Design and Implementation of a Large-Scale User Behavior Analytics Platform
Baidu Maps Tech Team
Baidu Maps Tech Team
Feb 3, 2016 · Big Data

How Baidu Maps Powers Its Open Platform with Big Data Architecture

This article explains how Baidu Maps’ open platform handles massive daily location data through real‑time and offline pipelines, Hadoop‑based offline computing, stream processing, and query engines built on MySQL, Redis, and Apache Kylin, while outlining future big‑data enhancements.

Apache KylinBaidu MapsData Pipeline
0 likes · 7 min read
How Baidu Maps Powers Its Open Platform with Big Data Architecture
21CTO
21CTO
Nov 21, 2015 · Big Data

Why Build a Kafka System? Core Use Cases and Design Principles

This article explains why Kafka is essential for activity and operational data pipelines, outlines key use cases such as news feeds, relevance ranking, security, monitoring, and reporting, and details its deployment topology, design decisions, and message persistence strategies.

Data PipelineDistributed MessagingKafka
0 likes · 14 min read
Why Build a Kafka System? Core Use Cases and Design Principles