Spark optimization — 3 Technical Articles

Jul 4, 2021 · Big Data

Reconstructing Cloud Music Membership Automation Data Warehouse: Practices, Standards, and Performance Optimizations

This article details the background, challenges, and step‑by‑step reconstruction of Cloud Music's membership automation data warehouse, covering modeling standards, task decoupling, Spark performance tuning, and data quality checks to achieve higher efficiency and reliability.

Data ModelingData WarehouseData quality

0 likes · 9 min read

Reconstructing Cloud Music Membership Automation Data Warehouse: Practices, Standards, and Performance Optimizations

Tencent Cloud Developer

Apr 12, 2018 · Big Data

Spark Usage in DataMagic Platform: A Practical Guide

This guide explains how DataMagic leverages Spark on YARN for fast, scalable offline analytics—covering Spark’s core role, four steps to master its terminology, configurations, parallelism, and code modification, plus practical deployment scripts, dynamic resource tuning, MongoDB export, job troubleshooting, and cluster upkeep for trillion‑record workloads.

DataMagicSparkSpark optimization

0 likes · 11 min read

Spark Usage in DataMagic Platform: A Practical Guide

WeChat Backend Team

Jan 18, 2018 · Information Security

How WeChat Detects Anomalous Users at Billion‑Scale: Inside Its Fast, Scalable Framework

This article explains how WeChat’s security team builds a scalable anomaly‑detection framework that partitions billions of user accounts, weights suspicious attributes, computes similarity graphs, and leverages Spark optimizations and graph‑partitioning techniques to efficiently identify malicious user clusters.

Anomaly DetectionLarge-Scale GraphSpark optimization

0 likes · 18 min read

How WeChat Detects Anomalous Users at Billion‑Scale: Inside Its Fast, Scalable Framework