How User Profiling Powers Modern Recommendation Systems

In the previous two articles we introduced the overall architecture of recommendation systems and detailed content features; this article continues with another important module—user profiling.

What is User Profiling

User profiling is a model built from user interests, behaviors, and attributes. By researching users and analyzing their actions, and aligning with business needs, users are grouped and typical features are abstracted into structured information.

For example, a typical user profile looks like:

{
    "Age": "25",
    "Occupation": "White-collar",
    "Education": "211 university graduate",
    "Industry": "Internet",
    "Location": "Beijing",
    "RelationshipStatus": "Single",
    "Hobby": "Rock music",
    "FavoriteFood": "Japanese cuisine",
    "Income": "Medium"
}

Importance of User Profiling

Many internet applications require recommendation or prediction. Beyond collaborative filtering, recommendations must consider user attributes.

User profiling aggregates discrete features into a finite range, providing a universal pattern to explain user behavior and attributes, forming the foundation of recommendation systems.

At Baixing.com, we collect browsing records and user attributes, abstracting them into tags, e.g.:

User A: {
    City: "Shanghai",
    InterestedCategories: ["Pet cats", "Used engineering trucks", "Used motorcycles"]
}

After deploying a profiling‑based recommendation strategy, click‑through rate increased by 50%.

Dimensions of User Profiling

Based on Baixing.com’s business, we consider three dimensions:

User Attributes

Includes age, gender, education, income, family situation, etc., closely related to content preferences. Acquisition can be costly.

User Status

Encompasses city, device model, network condition, time of behavior, emotions, and context. Most status information can be captured technically.

User Interests

The “category” of classified information reflects interests. Knowing which categories a user searches, browses, or posts reveals preferences. NLP techniques (tokenization, blacklist filtering, keyword extraction) can generate tags using algorithms such as TF‑IDF or TextRank.

Algorithms Needed

Classification Algorithms

When direct attribute data is unavailable, classification methods (Naïve Bayes, decision trees, logistic regression, SVM) can infer attributes. Example: using browsing patterns to predict age and gender.

Autoregressive Algorithms

To compute tag weights, we combine current behavior weight with decayed historical weights, e.g., current = 0.5×current + 0.25×yesterday + 0.125×day‑before, requiring careful parameter selection.

How to Compute User Profiles

Offline Computation

Store user actions (browse, post, contact) in HDFS, assign weights (higher for contact), and apply autoregressive algorithms to derive tag scores. Hadoop Streaming normalizes scores for categories, cities, and tags.

Real-time Computation

Real-time pipelines (e.g., Kafka) update profile tables with recent behavior, using higher decay to emphasize short‑term interests while following the same overall methodology.

Evaluating Profile Effectiveness

Manual Sampling

Spot‑check or random sample tags to verify accuracy; spot checks are limited in scale.

Model Metrics

Use accuracy, recall, AUC on test sets.

Business Feedback

Track recommendation clicks and tag‑level click‑through rates; conduct A/B tests.

Cross-validation

Compare profile data with external user information when available.

Conclusion

User profiling is a vital component of recommendation systems; by representing user attributes and preferences, profiling‑based recommendations achieve significantly better results.

Related articles: “Recommendation System” series Recommendation System Overview Content Features User Profiling (this article)