Design Architecture and Technical Strategies for Big Data Products

Many people have read various books and articles about big data, but the information is often scattered and lacks a systematic approach; this article provides a comprehensive overview of big‑data product design architecture and technical strategies.

Big‑data product design is divided into five steps.

Step 1: Embed data collection points across different front‑end channels; without comprehensive data, big‑data analysis is impossible.

Step 2: Use ETL to structure and load the multidimensional data collected.

Step 3: Build a data storage management subsystem, aggregate the standardized data into a data warehouse, and then decompose the warehouse into basic data marts.

Step 4: Based on the various data marts, employ R packages for data modeling and algorithm design; this stage involves heavy participation from product and operations teams and forms the foundation of many user‑profile systems.

Step 5: Combine the established data models and algorithms with front‑end channel and business characteristics to automatically match backend models and deliver personalized products and services.

Establish a systematic data collection indicator system

Creating a data collection and analysis indicator system is the basis for forming marketing data marts and a prerequisite for covering the breadth and depth of user behavior data.

The data collection analysis system must include full‑activity touchpoint data, structured and unstructured user data, enabling the classification and aggregation of attributes and values that form the foundation for discovering new marketing events.

Building a marketing data indicator analysis model, upgrading data collection, and leveraging user behavior touchpoints to establish consumption features and individual attributes across three dimensions (user behavior analysis, business operation analysis, marketing analysis) creates a user behavior feature analysis model; user‑dimension indicators result from cross‑referencing analysis elements with lifecycle touchpoints.

Current big‑data platforms often face three key issues: data is aggregated by channel/date/region without per‑user granularity; statistics are scale‑level and unsuitable for deep mining; and the data cannot support acquisition, retention, or marketing push functions.

To enable personalized front‑end analysis, each statistical data point must be tagged with a user identifier, allowing clicks to display the corresponding user’s behavior data and link to related pages.

By centering on the user, data collection dimensions can be defined as follows: user identity information, social life information, asset information, behavior preference information, shopping preference, user value, feedback, and loyalty.

User identity dimension: gender, age, zodiac, city of residence, active region, ID information, education, income, health, etc.

User social life dimension: industry, occupation, presence of children, children’s ages, vehicle ownership, housing type, communication status, data usage, etc.

User behavior preference dimension: online shopping behavior, risk sensitivity, price sensitivity, brand sensitivity, yield sensitivity, product preference, channel preference, etc.

User shopping preference dimension: category preference, product preference, purchase frequency, browsing preference, ad preference, shopping time preference, maximum single‑purchase amount, etc.

User feedback dimension: participated activities, discussions, favorited products, purchased items, recommended products, commented products, etc.

Based on the collected multidimensional data, ETL performs the following operations:

Data imputation: fill missing or empty values, flag unresolvable records.

Data replacement: substitute invalid data.

Format normalization: convert source formats into target formats suitable for the warehouse.

Primary‑foreign key constraints: enforce data integrity, redirect illegal data to error files.

Data merging: use table joins with indexed fields for efficient queries.

Data splitting: rearrange rows/columns, sort or renumber, remove duplicates.

Processing layer: a Hadoop cluster reads business data from sources, performs parallel computation, filters and merges data to produce target datasets.

Data Modeling, User Portraits, and Feature Algorithms

Extract marketing‑related customer, product, and service data, apply clustering and association analysis to build data models, configure user rule attributes and tags, and create a user rule set. The rule engine enables real‑time marketing pushes and condition‑triggered actions, synchronizing with front‑end channels and feeding execution feedback back to the big‑data system.

Automatically match rules based on personalized front‑end behavior and trigger push content

Based on the full‑process activity trajectory, the system analyzes all user touchpoints across online and offline channels, tags users to form behavior portraits, and derives segmentation rules. Each user attribute can have multiple values, configurable per activity, supporting black‑ and white‑list management.

Pre‑configured activity rules and models can be triggered by front‑end interactions; the system selects the highest‑matching rule to push marketing content, collects real‑time feedback, and continuously optimizes rule parameters and content.

FineBI data visualization

The big‑data system, combined with the marketing system, currently supports user portrait tagging, rule configuration, and push mechanisms. Future plans include integrating deep‑learning functions that automatically collect and analyze real‑time user data, compute function parameters, and generate highly matched marketing rules and content.

Machine self‑learning models are the core of future deep learning; extensive sampling, training, validation, and parameter tuning are required to determine precise function factors, enabling automatic calculation of marketing rules and recommendation models based on real‑time user behavior.

Beyond deep self‑learning, the system will gradually open up to third‑party platforms, expanding customer data coverage across the entire online‑offline lifecycle, enlarging data marts and event libraries, and deeply mining comprehensive customer needs to enhance product sales capability and overall user experience.

Data analysis is not a trivial matter https://www.jianshu.com/p/df813555e583

Recommended Reading:

Why does Alibaba's development manual state that Arrays.asList() cannot use its modification methods?

Dada O2O Backend Architecture Evolution: From 0 to 4000 High‑Concurrency Requests

iQIYI's Database Selection Strategy: Practical and Uncomplicated!

OPPO's Million‑Level High‑Concurrency MongoDB Cluster Optimization

Alibaba Cloud Redis Development Guidelines – Must Save!

I Spent 10 Hours Writing an Alibaba‑Style Data Middle‑Platform Analysis for Beginners

Nginx Core Architecture: Why It Supports High Concurrency

Volatile Keyword Interview Essentials

5‑Minute Cache Consistency Optimization Guide

ArchDigest – Architecture Knowledge | Large‑Scale Websites | Big Data | Machine Learning