High-Performance User Behavior Analysis Solution for Massive Data

This paper presents a high-performance user behavior analysis solution designed to handle massive datasets, specifically addressing the challenge of analyzing hundreds of billions of user behavior logs collected daily by Meituan-Dianping. The solution significantly reduces analysis time from hours to seconds, greatly improving analysis experience and work efficiency.

The paper focuses on the ordered funnel analysis scenario, detailing the problem analysis, design approach, and engineering implementation and optimization process. It is based on a presentation from ArchSummit Beijing in December 2017.

The core problem involves multi-dimensional analysis combined with sequence matching for deduplication counting. The solution introduces an inverted index-based data structure that enables efficient dimension filtering and sequence matching. The index consists of three parts: attribute names and values, UUID sets stored in bitmap format for fast filtering, and timestamp sequences for sequence matching.

The algorithm design includes a matching algorithm that handles sequence relationships, time window constraints, and property/dimension requirements. The engineering implementation uses Spring for interface services, Spark for distributed computing, and Alluxio for storage. Key optimizations include local scheduling, memory mapping, and unsafe calls to achieve significant performance improvements.

The solution has been successfully deployed in production, handling hundreds of billions of daily data records, supporting hundreds of millions of active users, and achieving sub-5-second query response times for interactive analysis.