Expert Insights on User Profiling and Stream Processing in Big Data

Key Questions

Q1: What are good solutions for user behavior analysis and how to build user profiles?

A simple idea is to consider the vast diversity of individuals; if your dimension design cannot cover billions of possible combinations, you quickly hit a ceiling. Credit‑card companies, for example, use thousands of attributes (cultural, social, physiological, personality, consumption) and keep expanding them.

Typical attributes such as age can be split into many granular buckets (e.g., <12;13‑21;22‑30;…>80) or combined with derived fields like birthday broken down into weekday, day‑of‑month, season, holiday, lunar calendar, leap month, moon phase, etc. Income level, education, graduation status, and other life‑stage dimensions can also be layered.

By grouping behaviors (calls to customer service, lottery participation, form filling, coupon usage, card activation, transactions, repayments, credit limit changes, adding secondary cards, etc.) along these dimensions, data scientists can apply matrix operations or other analytical methods to determine which dimensions most influence specific decisions, enabling precise marketing, risk control, personalized recommendation, and cross‑selling.

For instance, credit‑card marketing can shift from blind mass mailing to targeting a few thousand high‑value customers, reducing cost and negative user sentiment.

Q2: What are the mainstream stream‑processing and real‑time solutions, and what factors matter when choosing?

Data streams are often overlooked compared to databases, yet many real‑world scenarios are inherently “flow” rather than “store”. A database is like a reservoir; a data stream is like a pipe delivering water continuously.

Key evaluation criteria include:

Latency – how quickly the system can compute results from incoming data.

Throughput – the total volume of data the system can handle per unit time.

Parallelism – ability to scale across multiple nodes.

Fault tolerance and scheduling – handling node failures without data loss.

Historically, major players such as StreamBase, Aleri, Coral8, and Apama (now part of SAP HANA Smart Data Streaming) were adopted by Wall Street for low‑latency trading. In China, Sybase ESP (now SAP) and StreamBase remain commercial options, while open‑source alternatives like Apache Storm and Esper are also viable, depending on performance requirements.