Continuously Understanding Every User: Deep Dive into User Profile Architecture and Lifecycle

Why User Profiles Matter

User profiles (User Profile) are the most important asset produced by the feature pipeline, aggregating user‑side features to continuously describe each user. Without them, a recommender can only distribute content by overall popularity, ignoring personal preferences.

Profiles enable fine‑grained matching and negative‑content suppression, improving click‑through, watch‑through and dwell time.

Composition of a User Profile

Industrial‑grade profiles consist of four major parts:

Static basic attributes (age, gender, device, registration channel, membership level, region, etc.).

Long‑term positive and negative interests derived from aggregated behavior over 30 days or more.

Real‑time short‑term interests captured by NRT pipelines (minutes‑level updates).

Various user embeddings (category, semantic, global user tower) used for vector retrieval or as high‑order features.

Static features provide stable context for both offline and online stages, while dynamic interests capture evolving tastes.

User Profile Lifecycle

Cold start : New users have only static attributes. Strategies include registration‑based layering, look‑alike matching, explore‑and‑exploit, and rapid NRT‑based short‑term profiling, followed by Batch‑pipeline long‑term profiling after a day.

Interest growth & stability : As behavior logs accumulate, interest tags gain confidence, converge to a few core categories, and deepen in weight. Confidence improves with more interactions; low‑confidence tags are updated aggressively, high‑confidence tags more smoothly.

Interest drift : Sudden events (e.g., a sports tournament) cause temporary spikes in new interest tags. The system captures these via short‑term pipelines and, if sustained, integrates them into long‑term interests.

Interest decay : To reflect timeliness, weight decay (often exponential, e.g., Newton’s cooling law) reduces the influence of older actions. The decay coefficient k is tuned per business (e.g., 0.1‑0.3 for fast‑changing short‑video platforms, 0.02‑0.05 for knowledge‑sharing platforms). Engineering safeguards include weight clipping, higher initial weights for strong actions (complete view, share), and reset mechanisms for revived interests.

Lifecycle governance : Profiles are stored offline (Hive/Iceberg/Delta Lake) for training and versioning, and online in a Feature Server backed by Redis (hot users) and HBase (cold users). Governance includes coverage monitoring, PSI stability checks, latency tracking, and business‑metric correlation.

Profile System Design

Early systems used a dedicated User Profile Pipeline (UPP). Modern architectures embed profile aggregation as a sub‑module of both Batch and NRT pipelines, producing a unified User Profile asset stored offline and served online.

Computation Example

Given five articles and four user actions with weights (click = 1, like = 2, collect = 3), the pipeline associates each action with its content tags, applies the decay formula, and sums weighted scores. The resulting tag scores (e.g., "新能源汽车" = 5.71) are assembled into a JSON‑like profile containing static attributes, long‑term interests, short‑term interests, embeddings and behavior statistics.

{
  "基础属性": {"城市": "北京", "年龄": 28, "性别": "男"},
  "用户ID": "小A",
  "长期兴趣标签": {"新能源汽车": 5.71, "小米SU7": 0.90, ...},
  "短期兴趣": {...},
  "Embedding": [...],
  "行为统计": {...}
}

Online Storage

Offline profiles are kept in columnar stores (Parquet/ORC) for analysis and training. Online profiles reside in a Feature Server: hot and moderately active users in Redis for millisecond latency, cold users in HBase with lazy promotion upon re‑activation. Only the top‑K interest tags are retained online to save space.

Fusion of Long‑ and Short‑Term Interests

Simple weighted fusion creates a single interest vector but loses granularity. Production systems usually keep them separate: long‑term vectors are used for ANN retrieval, short‑term sequences for session‑based recall, and both are fed as independent features to ranking models, optionally gated by a small network.

Engineering Safeguards

Profile quality monitoring covers coverage, PSI stability, and timeliness. Bad‑case tracing links negative user feedback to the snapshot of the profile that generated the recommendation. A/B testing validates any logic change, and resource‑cost governance balances storage (binary serialization, pruning) and compute (Spark/Flink parallelism, QPS throttling).

User Sequence Modeling

Statistical tags lose order and decision logic. Sequence models (RNN/GRU, DIN, DIEN, SASRec, BERT4Rec) capture temporal dependencies and intent evolution. Industrial deployments truncate or sample sequences, pre‑compute embeddings offline, cache online, and combine multiple behavior streams.

LLM‑Era Profiles

Large language models enable semantic profile generation, intent inference, and interactive querying. Typical deployment patterns are offline batch generation, two‑stage routing (LLM only for uncertain users), or knowledge‑distillation into lightweight models.

Conclusion & Next Steps

User profiling is a continuous learning loop: behavior → feature → aggregation → profile → recommendation → new behavior. The next article will explore other feature‑pipeline outputs such as sample assets and model assets.