0 views collected around this technical thread.
This tutorial provides a detailed overview of Pandas, covering its core data structures, data import/export, selection, cleaning, aggregation, merging, and a practical sales analysis example, with complete code snippets for each operation.
This tutorial teaches how to use the Pandas library's groupby method to group data by one or multiple columns and apply aggregation functions such as sum, mean, max, and min, with clear code examples and a practical exercise.
This article describes the background, requirements, and architectural design of a custom counting system for a B2B second‑hand trading platform, covering dimension definitions, internal vs external implementations, data flow, persistence strategies, code interfaces, and read‑query optimizations.
The article describes how Bilibili tackled exploding fan‑out reads on its video detail page by introducing a Redis‑backed business‑association index and a generic Data Aggregation Gateway, which together cut downstream service traffic and load by over 90% while simplifying aggregation across multiple terminals and scenarios.
The Xianyu fan‑tag system solves the challenge of displaying full‑history purchase counts with real‑time updates and low‑latency, high‑throughput queries by daily exporting multi‑system data to a LevelDB‑based KV store, converting schemas, and applying real‑time compensation from transaction and follow‑change messages, merging offline and live data to produce sorted fan lists at ~10 k QPS.
The article explains how to define, collect, and calculate effective reading duration and screen count for mobile app H5 content using event timestamps, formulas, fragment reporting, and aggregation methods to improve content quality analysis and visualization.
Baidu’s advertising platform computes high‑precision response‑time quantiles at massive scale by intercepting each API call, locally summarizing data with mergeable T‑Digest histograms, periodically uploading compressed, Base64‑encoded summaries to a warehouse where they are merged on demand, enabling low‑latency, cost‑effective percentile analysis with sub‑0.1% error.
This article explains how a user‑behavior data dashboard originally built on a daily T+1 batch process was redesigned to achieve real‑time T+0 updates by introducing message‑queue notifications, a dedicated aggregation service, and MongoDB storage, improving data freshness and user experience.
The article presents a real‑time quantile solution using the TDigest data structure, which clusters data into centroids and stores digests in Redis or Doris, pre‑computes quantiles for all dimension combinations, and provides a reusable API that delivers fast, accurate, low‑memory quantile statistics for diverse business scenarios.
The article presents a general solution that streamlines daily business issue diagnosis by aggregating multi‑service data via GraphQL, converting it into human‑readable semantics, and automatically detecting anomalies with a rule engine, enabling one‑click retrieval, reduced hand‑offs, and extensible monitoring for platforms such as Xianyu.
The article examines the key challenges of real‑time data aggregation—data freshness, timely processing, and result visibility—and surveys common solutions such as timestamp‑based sync, CDC, full and incremental computation, storage formats, and trigger mechanisms.
Hawkeye, iQiyi’s self‑developed front‑end monitoring platform, unifies exception, performance and behavior tracking, automatically aggregates events with unique error IDs, isolates business‑type alerts via configurable codes, links front‑end and back‑end traces through Trace IDs, and leverages a JSSDK, backend services, big‑data stream processing and storage in Redis, Elasticsearch and HBbase, with future plans for small‑program SDKs, wider framework support and open‑source release.
The article explains how MySQL multi‑source replication can be employed to back up, horizontally or vertically partition, and merge data from several servers into a single target database, illustrating three typical scenarios and providing example SQL for data consolidation.
The article details Youku’s new media‑asset platform, which replaces a fragmented MySQL‑based system with a domain‑driven entity model stored in Ali‑HBase, leverages Elasticsearch for flexible front‑and back‑end search, and adds an aggregation layer that unifies diverse data sources and reusable computation tasks, delivering high‑availability, low‑latency service for billions of daily API calls.
Xianyu introduced a lightweight data‑aggregation service and adopted Alibaba’s TQL GraphQL gateway, allowing front‑ends to fetch multiple domain data in a single request, which cut development costs by over 50%, reduced latency, and accelerated page composition across more than 30 pages.
This article explains how the data aggregation platform adopts Apache Flink for high‑throughput, low‑latency stream processing, covering the complete workflow from data source integration, transformation operations, windowing and time concepts, to a concrete order‑count example with custom aggregation logic.
This article details the design and operation of a high‑concurrency hotel quote search platform, covering business background, core functionalities, system layers, data fetching, aggregation, scheduling, and price‑update mechanisms to ensure comprehensive, real‑time hotel pricing for users.
This article walks through installing Elasticsearch 5.6.9, configuring system limits, creating indices, inserting and deleting documents, executing complex aggregation queries, and integrating Elasticsearch with Java using the TransportClient, providing a practical guide for building analytics on large‑scale data.
The article shares practical experiences from large‑scale mobile client development, covering user crash scenarios, non‑intrusive data collection, AOP‑based interaction and network monitoring, performance instrumentation, and data aggregation techniques to improve debugging efficiency and operational reliability.
This article explains how to count paid and unpaid orders per user‑product pair in PostgreSQL by aggregating order statuses into arrays, using a sub‑query with GROUP BY and a CASE expression to produce accurate payment success and total order counts.