This article details Alibaba's real‑time computing platform, outlining its operational challenges, the unified automation platform Aquila, proactive fault‑elimination strategies, and ongoing moves toward intelligent, data‑driven management to support massive workloads during events like Double‑11.
At ACM SIGIR 2018, Didi presented AI‑driven intelligent‑transportation research—including a ride‑sharing preference prediction paper, keynote insights on smart dispatch, maps and traffic, collaborations with over twenty cities and numerous universities, open data initiatives, and plans for new thematic research programs.
The article describes the evolution of 360's Titan big‑data processing platform through three architectural stages, details its functional modules, explains the DITTO component framework, context and rule‑engine abstractions, and shares practical case studies and personal insights on building a flexible, self‑service data platform.
This article shares Sogou's journey in constructing a large‑scale big data platform, covering business overview, the evolution of its operations infrastructure, productization practices, security measures, and practical tips for medium‑size teams seeking to add value from data.
This article chronicles the three‑generation evolution of Zhihu’s anti‑cheat platform Wukong, detailing its business context, spam taxonomy, multi‑layered control methods, architectural redesigns, strategy language improvements, graph‑based risk analysis, and the continuous integration of big‑data and machine‑learning techniques to combat content and behavior spam.
This article details Ctrip's experience with the Presto distributed SQL engine, outlining the initial performance and stability issues, the comprehensive enhancements made in security, resource control, compatibility, and monitoring, and the multi‑stage upgrade plan that guides its future evolution.
The article reviews the ten‑year evolution of Ant Financial's message queue, detailing its core reliability, consistency, availability and performance requirements, the architectural mechanisms built to meet them, the shift to pull‑mode and API‑mode designs, and the recent integration of compute capabilities to create a smart data transmission platform.
This article explains how JD.com evolved its recommendation system from simple product suggestions to a sophisticated, multi‑screen, multi‑type personalized engine using big‑data collection, real‑time behavior tracking, machine‑learning models, and a modular architecture that boosts conversion and user experience.
This article outlines Baidu's comprehensive product evaluation framework, describing its multi‑layer assessment system, the combination of subjective and objective metrics, and a suite of common evaluation methods such as indicator analysis, AB testing, user feedback, behavior analysis, big‑data profiling, and competitor comparison.
The article describes the design, data integration, ID mapping, tag generation, and application scenarios of the 58 user profiling platform, which aggregates billions of user IDs across multiple business lines to provide online and offline persona data for personalization, analytics, and AI modeling.
This article summarizes OPPO's rapid growth of its big‑data platform, detailing the three‑layer architecture, the evolution from Flume‑Kafka to NiFi for data ingestion, the upgrade of the OFlow task scheduler, comprehensive monitoring of data, resources and task SLA, and the development of a self‑service analytics tool called InnerEye to ensure stability, efficiency, and security.
JD.com's 618 shopping festival achieved 159.2 billion yuan in sales, leveraging AI and big data to enhance consumer experience and empower retail partners through intelligent supply chain solutions.
This article explains the internal workflow of Spark Streaming task submission, detailing how StreamingContext, DStream, receivers, and output operators are transformed into RDD jobs, and includes annotated Scala code examples that illustrate each step of the process.
This article examines the challenges of handling massive log data in high‑traffic e‑commerce platforms and presents a comprehensive architecture, optimization strategies, and practical implementations—including Rsyslog, Kafka, Fluentd, and the ELK stack—to improve scalability, performance, and reliability of log management systems.
At WGDC 2018, Didi’s mapping division revealed how its AI‑driven platform leverages massive real‑time travel data, machine‑learning and deep‑learning models—including a new ETA estimator, demand‑supply forecasting, and reinforcement‑learning order allocation—to deliver ultra‑accurate pick‑up points, route planning, and destination predictions, while opening de‑identified data and research topics to academia.
This article explains how enterprises can build a scalable data analytics platform on Hadoop by outlining the multi‑layer architecture, storage options, data synchronization methods, and ETL/offline computation techniques, while highlighting practical component choices such as Hive, HBase, Spark, and Oozie.
Tencent Cloud’s government‑cloud strategy, showcased by Guangdong’s “粤省事” platform, leverages WeChat as a single access point and a partner‑driven backend of AI, big‑data and IoT services to digitize certificates, streamline workflows for citizens, businesses and officials, and address low public‑service satisfaction by redesigning processes rather than merely automating them.
The article summarizes the GIAC Shenzhen conference, highlighting front‑end, mobile, and backend optimization techniques, large‑scale architecture designs, cloud‑native solutions, big‑data testing strategies, and quality assurance practices shared by top Chinese internet firms.
This article explains how Tencent’s ZhiYun multi‑dimensional monitoring system evolves from the Mobile Monitor platform, outlines its design principles, data‑factory capabilities, storage choices, and intelligent features, and demonstrates how it enables real‑time, multi‑dimensional analysis and alerting for large‑scale business operations.
Despite Gartner's 2017 claim that Hadoop is nearing the end of its production maturity, a series of interviews with Chinese big‑data experts reveal that Hadoop's ecosystem remains robust, with core components like HDFS, YARN, Spark, and HBase continuing to dominate the market.
An in‑depth comparison of Hadoop and Spark examines their architectures, performance, cost, security, and machine‑learning capabilities, helping readers decide which open‑source distributed processing platform best matches their batch, streaming, and analytical workloads.
This comprehensive guide explains Spark's ecosystem, execution principles, key features, deployment architectures, core concepts like RDD, Transformations, Actions, Jobs, Stages, Shuffle and Cache, as well as Spark Streaming mechanics and practical resource‑tuning tips for optimal big‑data processing.
The talk details Tencent Xinge’s architecture and cloud‑native practices that enable hundred‑billion‑level mobile push, combining terminal integration, real‑time backend filtering, distributed bitmap selection, precise‑push AI models, and DevOps pipelines to deliver fast, scalable, data‑driven notifications with effect tracking.
This article explains Spark's role in the DataMagic platform, outlines four practical steps to quickly master Spark, details key configuration and parallelism settings, shows how to modify Spark code, and provides operational tips for cluster management and job troubleshooting.
This article explains Spark's executor memory architecture, covering on‑heap and off‑heap allocation, static versus unified memory managers, storage and execution memory handling, RDD persistence levels, eviction policies, and shuffle memory usage, providing practical formulas and configuration tips for optimal performance.
This article details the evolution of Lianjia's massive‑data platform from its initial 0 version through 1.0 and 2.0, describing architectural challenges, a three‑layer redesign, data‑engine selection (ROLAP, MOLAP, Kylin), transparent compression techniques, and practical lessons for large‑scale data systems.
This guide details how to construct a real-time data processing platform on CentOS 7 using the Hadoop ecosystem—installing and configuring Zookeeper, Maven, Hadoop, Kafka, HBase, Spark, and Flume—followed by a Spark Streaming job that consumes Kafka messages and writes them into HBase.
This article walks through installing Elasticsearch 5.6.9, configuring system limits, creating indices, inserting and deleting documents, executing complex aggregation queries, and integrating Elasticsearch with Java using the TransportClient, providing a practical guide for building analytics on large‑scale data.
This article uses a human‑computer analogy and a playing‑card counting example to explain the fundamentals of distributed computing, why single machines cannot handle massive data, and how the MapReduce model’s four steps—split, transform, shuffle, and merge—solve big‑data problems.
In an interview, Zhihu senior vice‑president Li Dahai explains how AI, big data and Tencent Cloud enable Zhihu’s knowledge‑matching algorithms, boost answer volume by 300%, and shape the platform’s vision of connecting expert users while emphasizing principles over pure algorithmic tricks.
The article outlines how to build a credit‑based content management platform by describing the evolution of security practices, defining user‑generated, professional‑generated, and occupational content models, proposing a credit‑audit workflow with risk assessment, and presenting AI‑driven text classification and anti‑cheat methods to balance traffic, quality, and trust.
A Chinese company, StarRing Technology, became the first ever to pass the notoriously difficult TPC‑DS benchmark, revealing the test’s history, methodology, and why this achievement marks a major milestone for database performance in the big‑data era.
This article explains Hadoop's distributed storage and processing framework, details the MapReduce programming model, describes the classic JobTracker/TaskTracker architecture, outlines the shuffle and combine phases, and introduces YARN as a scalable replacement with its ResourceManager, ApplicationMaster, and NodeManager components.
In a recent Cloud Xi conference, Hua Xiansheng, deputy director of Alibaba’s DAMO Academy Machine Intelligence Lab, presented the City Brain initiative, unveiling three AI-powered products—Tianyao, Tianying, and Tianji—that leverage massive video and sensor data to achieve real‑time perception, decision‑making, prediction, and intervention for smarter urban governance.
This article explains the evolution of Kafka message formats from version 0.7.x through 0.8.x (including 0.9.x) to 0.10.x, detailing each field, compression handling, and the design motivations behind the changes.
Suning’s executive vice president outlines how digital empowerment—through AI, cloud computing, and big‑data analytics—creates a new smart‑retail ecosystem, emphasizing user‑centric services, data‑driven operations, and technology platforms that blend efficiency with a human touch.
This article details Chain Home's OLAP platform architecture and Kylin usage, covering the evolution from early ROLAP to MOLAP multi-dimensional engine, Kylin's basic principles, platform structure, application scenarios, usage specifications, capability extensions, and middleware development.
Baidu’s TSDB processes over 20 million data points per second per node and tens of thousands of queries per second cluster‑wide by employing a stateless read/write‑separated elastic architecture, multi‑layer storage across Redis, HBase and Hadoop, minute‑level geo‑redundant self‑healing, and a modified Gorilla compression that cuts storage by 80% with minimal CPU overhead.
The 2018 Gdevops Global Agile Operations Summit in Chengdu gathered industry experts who shared practical insights on AIOps implementation, sharding database ecosystems, DevOps adoption in traditional enterprises, large‑scale data management, ElasticSearch clustering, AWS blue‑green deployments, cloud database operations, Alibaba's double‑11 ops platform, 58 delivery mini‑program architecture, and scalable game service design.
This guide details the required system environment, download sources, and step‑by‑step offline file preparation for installing Cloudera's CDH 5.14 on a CentOS 6.7 server, including JDK, parcels, and MySQL connector setup.
This article outlines Suning's evolution from a basic network operations unit to a sophisticated, multi‑layered security architecture that integrates organizational structure, protection platforms, risk management, big‑data threat perception, and continuous improvement to safeguard e‑commerce operations.
This article explains how cloud computing, big data, and artificial intelligence are interrelated, describing the evolution from physical resource management to virtualized, elastic services, the roles of IaaS, PaaS, and SaaS, and how each technology benefits the others in modern applications.
This article explains how Elasticsearch and the ELK stack address challenges of storing, securing, retrieving, and analyzing massive data volumes by providing distributed real‑time search, log collection, visualization, and even serving as a NoSQL alternative for large‑scale applications.
The article explains the concept of distributed architecture, its key characteristics such as cohesion and transparency, showcases how MySQL and middleware like Mycat are used in e‑commerce platforms, and outlines the evolution, practical implementations, and challenges of building scalable distributed database systems.
Huawei’s HCDA‑BigData V1.0 certification, launched in April 2018, offers engineer‑level credentials, comprehensive training materials, a four‑day hands‑on program, and a Prometric exam to equip developers, partners, students, and ICT professionals with advanced big‑data development skills on FusionInsight.
The article presents Meituan's Traffic Compass—a data‑warehouse‑driven traffic analysis platform for the hotel‑travel business—detailing its background, challenges, architectural layers, dimensional modeling, Kylin‑based query engine, configuration mechanisms, performance metrics, and future optimization plans.
This article examines Spark's unified memory manager in version 2.2.1, detailing on‑heap and off‑heap memory regions, the four on‑heap memory pools, dynamic execution‑storage memory sharing, task memory accounting, and provides concrete calculation examples to explain UI discrepancies and runtime memory limits.
This article details Meituan's DataMan platform, describing the background of data quality challenges, the eight-step PDCA-driven solution, architectural design, technical stack, monitoring standards, and the resulting improvements in data governance and operational efficiency across their massive data warehouse ecosystem.
Milano, a distributed log collection and analysis platform built on the ELK stack, leverages Filebeat, Kafka, Logstash, Elasticsearch, and Kibana to provide high‑throughput, low‑latency, secure, and visual log management for massive clusters, addressing the challenges of traditional manual log inspection.
This article explains Hadoop’s core components, the MapReduce programming model, the detailed shuffle and merge processes, and how YARN replaces the classic JobTracker/TaskTracker design to improve scalability and resource utilization in large‑scale data processing clusters.
The article details Alipay’s comprehensive architecture for the Double 11 shopping festival, covering its three‑layer IAAS/PAAS/SAAS model, logical data‑center design, multi‑active disaster‑recovery, blue‑green deployment, distributed data sharding, transaction processing, and the Ant Credit Pay service’s performance and risk‑control mechanisms.
This guide introduces the most commonly used HBase operational tools—including Canary, hbck, HFile viewer, CopyTable, Export/Import, ImportTsv, CompleteBulkload, RowCounter, CellCounter, and clean utilities—explaining their purposes, typical use‑cases, and exact command syntax for effective cluster administration.
This article explains the origins and goals of cloud computing, how virtualization adds flexibility in time and space, the evolution from physical servers to public and private clouds, the role of IaaS, PaaS, and SaaS, and how big data and artificial intelligence intertwine with cloud services to enable modern intelligent applications.
The paper describes a high‑performance user‑behavior analysis system that processes hundreds of billions of daily logs for Meituan‑Dianping, using an inverted‑index structure with bitmap UUID sets and timestamp sequences, combined with Spark, Spring and Alluxio optimizations to cut query times from hours to under five seconds.
This article walks through a gray‑scale, controlled upgrade of Hive from version 0.13 to 2.1, covering metadata schema analysis, syntax compatibility, new Hive‑2.1 features, UDF adjustments, performance tweaks, and a step‑by‑step procedure to ensure stability and zero service interruption.
The March 2018 DBAplus Newsletter compiles the latest releases across RDBMS, NoSQL, NewSQL, time‑series and big‑data ecosystems, highlighting new features, performance improvements, compatibility updates and key technical links for Oracle, MySQL, MariaDB, SQL Server, DB2, PostgreSQL, TiDB, CockroachDB, InfluxDB, Hadoop and several Chinese‑made databases.
This article chronicles a game company's operations team evolution over seven years, detailing how it grew from a tiny manual crew to a large, automated, cloud‑native organization that built its own CDN, monitoring, and platform solutions while tackling scaling, reliability, and service‑orientation challenges.
This article presents a comprehensive attributes matrix and data‑flow model overview for major Apache streaming platforms, comparing versions, sponsors, event handling, fault tolerance, processing order, latency, resource management, APIs, and supported connectors to aid practical technology selection.
This article details Lianjia's journey of designing and implementing a low‑latency, stable real‑time computing platform using Spark Streaming on YARN, covering technical selection, architecture components, version compatibility challenges, exactly‑once semantics, graceful shutdown, Kafka tuning, and future enhancements.
The article presents a comprehensive solution for reducing Hadoop cluster storage consumption by applying ZFS‑based transparent compression and data‑governance techniques, detailing the technical background, design choices, implementation steps, performance optimizations, and observed storage savings.
Suning’s real‑time log analysis system integrates Flume, Kafka, Storm and Spark Streaming to collect, cleanse, and compute metrics like NDCG, ensuring low latency, high throughput, exact‑once processing, and robust data safety while supporting multi‑dimensional analytics on massive online‑offline traffic.
Apache Spark 2.3 introduces major upgrades such as millisecond‑latency continuous streaming, stream‑to‑stream joins, a native Kubernetes scheduler backend, accelerated Pandas UDFs, and several MLlib improvements, all aimed at making big‑data processing faster, easier, and smarter.
The article describes the evolution of Ctrip's wireless APM platform from the early UBT-based monitoring to a globally‑oriented, metric‑rich system that processes over 100 billion data points daily using Storm and Elasticsearch, detailing its design, key performance dimensions, data‑volume trade‑offs, and implementation choices.
The article outlines a systematic approach for large‑scale Hadoop clusters to monitor daily data growth, identify abnormal paths, manage rapid expansion, clean unused cold data, and implement capacity forecasts, while providing concrete daily and quarterly actions, Hive‑specific strategies, and practical examples to keep storage under control.
The article examines China’s push for big data sharing, highlighting Representative Zhang Jindong’s calls for a national open data platform, integration of AI and blockchain, and urgent legislation to address data fragmentation, security risks, and the need to empower SMEs against data monopolies.
The article explores how the integration of cloud computing, big data, and artificial intelligence is reshaping enterprise data platforms, outlining a multi‑stage evolution from data unification to ecosystem building and forecasting the strategic importance of data in future business transformation.
The article explains how Ctrip's big‑data platform introduced Alluxio to isolate real‑time Spark Streaming jobs from HDFS NameNode maintenance, reduce NameNode pressure, improve Spark SQL performance, and provide a unified storage layer across multiple HDFS clusters.
This article introduces Hulu's in‑house OLAP engine Nesto, detailing its near‑real‑time data ingestion, nested data model, TB‑level storage using HBase and Parquet, MPP query execution, custom predicate library, and the overall architecture that enables sub‑second ad‑hoc queries for user analytics.
CallGraph is JD.com's internally developed distributed tracing and service governance platform that addresses the challenges of monitoring complex microservice architectures by providing low‑intrusion, low‑latency tracing, real‑time analytics, configurable sampling, and integration with JMQ, Storm, Spark, HBase, and JimDB for both operational insight and performance optimization.
This article explains how real-time streaming has evolved from batch Hadoop systems through Lambda architecture to modern Kappa-style pipelines, highlighting its growing importance for developers, enterprises, and the integration of streaming with microservices, AI, and cloud-native technologies.
This article reviews recent progress in Bayesian machine learning, covering foundational theory, non‑parametric approaches such as Dirichlet and Indian buffet processes, regularized Bayesian inference, and scalable techniques for big‑data environments including stochastic variational methods, distributed algorithms, and hardware acceleration.
This article details JD's evolution from rule‑based recommendations to a multi‑screen, AI‑driven personalization platform, describing its system architecture, data pipelines, feature services, and key technologies that enable real‑time, user‑centric product suggestions across the e‑commerce ecosystem.
This article details the design, challenges, and implementation of Meituan’s Flow Compass—a data‑driven product that combines user, scene, and traffic source dimensions using a Kylin‑based warehouse to enable rapid, flexible traffic‑source analysis for hotel‑travel growth.
This article explains the core of search engine technology by describing what an inverted index is, how it is built using single‑pass memory and multi‑way merge methods, how indexes can be partitioned and incrementally updated, and how Hadoop can be used for large‑scale indexing.
iQIYI’s big‑data collection system has progressed from simple HTTP log uploads to a Flume‑Kafka pipeline and finally to a custom Venus‑Agent architecture with centralized configuration, persistent offsets, dual‑Kafka streams and Flink processing, now handling tens of millions of queries per second and over three hundred billion records daily to power its AI‑driven services.
Meituan designed a Storm‑based real‑time data processing platform that guarantees at‑least‑once delivery and high availability, employs a custom spout, regression‑driven traffic smoothing, and a low‑latency KV store with atomic operations, persisting results in Kafka, MySQL and Cellar to power merchant dashboards and heat‑tag analytics, while planning broader real‑time analytics expansion.
A 2017 analysis of China’s high‑end internet talent market reveals that cloud computing, big data and gaming lead salary rankings, while company financing stage, job role demand, city location and candidate age all shape hiring trends and compensation levels.
This article introduces Apache Solr and Elasticsearch, explains their shared Lucene foundation, compares their scalability, deployment ease, and key features, and highlights why Elasticsearch is often considered more efficient for large‑scale, multi‑tenant search applications.
The case study details Donghua Software’s smart flood‑control and drainage solution, which integrates IoT sensors, NB‑IoT/eLTE networks, Huawei’s FusionSphere cloud platform, big‑data analytics, and GIS to provide real‑time monitoring, predictive warnings, automated gate control, and efficient emergency dispatch for urban water management.
This article shares a comprehensive overview of game operation security at Tencent, covering personal background, real‑world incident cases, the inherent challenges of large‑scale game services, past monitoring efforts, and a new data‑driven alerting framework that dramatically reduces false alarms while protecting game economies.
This article walks through constructing a Spark-powered text sentiment analysis pipeline—from crawling movie reviews, preprocessing and feature extraction with jieba and TF‑IDF, to training Naive Bayes and SVM classifiers—while discussing Spark's advantages and ways to improve model accuracy.
This article surveys the technical architectures of major web platforms—including Wikipedia, Facebook, Yahoo! Mail, Twitter, Google App Engine, Amazon, and Youku—highlighting their design patterns, scaling techniques, storage solutions, and caching strategies to reveal how massive online services are built and operated.
The article presents Zhang Qi's insights on end‑to‑end quality assurance for JD's POP platform and 7FRESH stores, covering pre‑, during‑, and post‑release testing measures, big‑data‑driven automation, tester roles, team growth strategies, and knowledge‑sharing initiatives.
Analyzing 2017’s big data boom, the article explores how the 4V characteristics—volume, variety, velocity, and value—spurred innovations like distributed storage, NoSQL, real‑time stream processing, and AI integration, and predicts future hotspots such as SQL resurgence, cloud‑based platforms, and AI‑driven analytics.
Attending VLDB 2017 in Munich, the report summarizes the conference’s broad coverage of database research—from new hardware‑accelerated prototypes and Spark‑based big‑data processing to Oracle and SAP HANA case studies, keynotes, notable papers, and reflections on industry trends and Chinese contributions.
This article summarizes Wu Xiaoguang's talk at Gdevops 2017, detailing how Vipshop integrates data processing, analysis, and mining technologies—such as Flume, Kafka, Spark, and custom scheduling—to improve operational decision‑making, performance monitoring, root‑cause analysis, and predictive modeling across its e‑commerce platform.
This article introduces the concept, characteristics, and use cases of time series databases, explains the data model and challenges of traditional solutions, and provides a detailed overview of Tencent's Cloud Time Series Database (CTSDB) along with performance comparisons against InfluxDB.
In this lecture, Prof. Pei Dan of Tsinghua University outlines the evolution of intelligent operations from rule‑based automation to machine‑learning‑driven AIOps, discusses data, feedback loops, and practical challenges, and calls for stronger collaboration between industry and academia to accelerate research and deployment.
Meituan built a scalable scenario‑query platform that unifies traffic, activity and investment data by layering RPC services, a Storm‑driven pre‑computation tree stored in Redis/Tair, and a middle‑platform API with circuit‑breaker logic, cutting response times from seconds to under one second while dramatically reducing code coupling and simplifying future feature development.
This article explores how distributed systems determine node liveness, manage failover and recovery, and implement at‑most‑once, at‑least‑once, and exactly‑once processing guarantees—including opaque transactions and two‑phase commit—using examples from Kafka, Zookeeper, and big‑data pipelines.
This article explains how the open‑source DBus platform, combined with the Wormhole streaming engine, captures raw application logs, lets users configure visual rule operators, and transforms the unstructured message part into schema‑driven, Kafka‑ready data for downstream analytics.
This article presents a comprehensive case study of how WeChat Pay rebuilt its transaction record storage system to handle massive data volumes, improve performance, ensure data completeness, support flexible queries, and strengthen security through distributed key‑value storage, data partitioning, and operational safeguards.
This article shares a data engineer's three‑year experience using Apache Kylin for real‑time OLAP on petabyte‑scale data, describing the business background, challenges of pre‑computation, cube modeling, dimension reduction, and various optimization techniques such as hierarchy, mandatory, and joint dimensions, as well as precise count‑distinct handling.
This article explains Meitu's development of a distributed bitmap system that leverages the speed and storage efficiency of bitmap structures to handle massive user data, detailing its evolution, architectural choices, implementation practices, and lessons learned to inspire similar big‑data solutions.
This article introduces Meitu’s self‑developed log collection system Arachnia, explaining why a custom solution was needed for massive server‑side user‑behavior logs, the key requirements such as reliability and real‑time throughput, and the core architectural mechanisms that address those challenges.
At Meitu’s seventh Internet Technology Salon in Xiamen, over a hundred experts discussed recommendation algorithms and big‑data solutions, with talks on the Arachnia log‑collection system, the Naix distributed bitmap service, Meitu’s personalized recommendation pipeline challenges, and novel data‑missing‑theory models for improved performance.
The Suning Cloud Commerce Data‑Driven Forum gathered top experts from Intel, Mobike, Google, and others to share deep insights on big data, AI, and data‑driven retail, offering practical case studies, emerging technologies, and strategic perspectives for the industry’s digital transformation.
This report presents a systematic performance evaluation of Apache Flink and Apache Storm across multiple real‑time processing scenarios, measuring throughput, latency, message‑delivery semantics, and state‑backend effects, and provides recommendations for selecting the most suitable engine based on the observed results.
The article details WMDA, a no‑code and manual‑code data collection platform for PC, mobile and app that supports real‑time and offline user behavior analysis, describing its functional model, behavior taxonomy, five‑layer architecture, tracking techniques, circle‑selection, data services, streaming and batch processing pipelines, and related technologies such as Storm, Spark, Druid and Roaring Bitmap.
The article describes Alibaba's end‑to‑end automated fault recovery system for its massive network, covering extensive data collection, Spark‑based event processing, flexible alerting with Siddhi, alert convergence using PageRank, and scripted recovery actions to achieve high availability during the Double Eleven traffic surge.
