Impala Deployment and Optimization in Sensors Data's Multi-Dimensional Analytics Platform

1. Sensors Data Product Architecture The platform consists of three layers: a data foundation (collection, transport, governance, storage, query, and intelligence) built on a private cloud, an analytics cloud (behavior analysis, alerts, user profiles, ad‑placement and business analysis), and a marketing cloud (campaigns, WeChat operations, canvas workflows), all supported by data‑driven consulting services.

2. Technical Architecture Data is ingested via various SDKs and tools, passed through Nginx to a log extractor, transformed into the Sensors Data protocol, and streamed into Kafka. A custom Data Loader subscribes to Kafka, writes data to Kudu in real time, and periodically converts it to Parquet for column‑store efficiency. Yarn schedules Kafka consumption and preprocessing tasks, while the Impala‑based query engine translates client requests into SQL, caches results, and monitors performance.

3. Real‑Time Analysis Engine Based on Impala User‑behavior requirements demand flexible, low‑latency queries over massive, high‑cardinality dimensions. Impala’s MPP architecture runs compute and storage on the same nodes, sharing memory, CPU, and disk. It comprises StateStore, CatalogD, and ImpalaD processes; ImpalaD nodes act as Coordinators or Executors. Despite high memory usage, Impala offers excellent query speed, prompting further improvements for fault tolerance.

4. Query Performance Optimizations

• Old Storage Mode : Data partitioned by day and event with limited ordering caused slow full‑sort operations for large funnels.

• New Storage Mode : Pre‑sorted by day, user ID, and timestamp; archival of cold data to S3; hot data kept in Kudu for fast updates.

• User‑Behavior Sequence Optimization : Leveraged ordered scans and shuffle exchanges to eliminate costly sort stages, achieving 6‑40× speedups and reducing memory usage to one‑fifth.

• Join Elimination : Converted full joins to inner joins with runtime filters, dramatically shrinking data flow.

• Expression Push‑Down : Moved complex CASE/regex logic to the multi‑threaded Scan layer, cutting data transfer by over 80% and improving execution time.

5. Query Resource Estimation Two main issues—insufficient memory estimates and large queries blocking small ones—are addressed by (a) historical‑signature‑based memory prediction, (b) refined formula‑based estimates for Agg/Join/Sort operators, and (c) a retry mechanism for failed queries. A dual‑queue scheduler separates small and large queries, and an improved FIFO‑plus‑time‑aware algorithm ensures timely execution of short queries.

The estimation workflow generates a query signature, looks up historical resource usage, falls back to formula estimates if needed, schedules execution, and updates the signature store with actual consumption. Tests on a 10‑node cluster show estimation accuracy close to real memory usage and error‑rate reductions exceeding 80% for many customers.

6. Future Plans Ongoing work includes open‑sourcing more optimizations (join elimination, expression push‑down, formula‑based memory prediction, scheduler enhancements), implementing elastic compute for dynamic scaling, and building query observability tools that let users monitor and manage resource‑intensive queries themselves.

7. Q&A Highlights The session concluded with a discussion on ordered funnel analysis, the custom materialize_expr hint, upcoming community contributions, and the data‑import pipeline that writes to Kudu before converting to Parquet.