How to Build a Recommendation System from Scratch: Key Concepts and Strategies

Understanding Recommendation Algorithms

Recommendation systems aim to create more efficient connections between users and content, saving time and cost. They consist of three core components: data, algorithms, and architecture.

Data provides information about users and items, including attributes and behavior signals such as clicks, purchases, or gameplay.

Algorithms process massive data to generate personalized recommendations, replacing manual strategies.

Architecture ensures real‑time, automated operation, handling request reception, data processing, storage, model computation, and result delivery.

Overall Framework

The recommendation pipeline typically includes the following modules:

Protocol scheduling – sending user requests (e.g., ID, location) and returning recommendation results.

Recommendation algorithm – applying logical rules to produce final recommendations.

Message queue – collecting and processing user behavior data.

Storage units – persisting different data types (e.g., MySQL for content tags, Redis for real‑time data, TDW for analytics).

User Profiling

3.1 User Tags abstract multidimensional characteristics into representative labels, forming a comprehensive user portrait.

3.2 Types of User Portraits

1. Raw Data includes four aspects:

User data – gender, age, channel, registration time, device model, etc.

Content data – category, keywords, tags extracted from articles or games.

User‑content interaction – behaviors indicating preferences for specific categories or tags.

External data – additional signals from other platforms to enrich the portrait.

2. Fact Tags are divided into static (stable personal attributes) and dynamic (behavioral signals). Dynamic tags further split into explicit actions (likes, shares, ratings) and implicit actions (clicks, dwell time).

3. Model Tags are derived from fact tags via clustering or weighted calculations, enhancing the information used for recommendation.

Content Profiling

Content portraits involve extracting keywords, tags, and visual features using NLP and image processing, while environmental variables (time, location, surrounding content) also influence recommendation decisions.

Algorithm Construction

5.1 Recommendation Algorithm Flow – The basic logic transforms user and item information into recommendation results. Simple popularity ranking is insufficient; personalized interest requires complex rule‑based computation.

The algorithm pipeline consists of:

Recall – narrowing millions of items to a manageable candidate set.

Filter – removing already‑consumed or unsuitable items.

Ranking – ordering the candidates.

Mixing – adjusting the order to avoid over‑concentration.

Strong rules – applying business‑specific overrides (e.g., promotion top‑ranking).

Recall Strategies

Hot recall – selecting recently popular items.

Collaborative recall – leveraging similarity between users.

Tag recall – using user‑generated tags.

Time recall – prioritizing the newest content.

Ranking Strategies

5.3 Model‑Based Ranking (Logistic Regression Example)

Logistic regression converts linear outputs into probabilities via a sigmoid function, suitable for binary outcomes such as click prediction. The model is trained on labeled samples (positive: clicked, negative: not clicked) and uses engineered features from user and content portraits.

Feature engineering explores four dimensions:

Basic data

Trend data

Temporal data

Cross features

Evaluation Metrics

6.1 Hard and Soft Indicators

Hard metrics – e.g., click‑through rate, conversion.

Soft metrics – user satisfaction, content diversity, long‑tail discovery.

6.2 Measuring Recommendation Effectiveness

Offline experiments – repeated testing on historical data.

User feedback – small‑scale testing to gather qualitative impressions.

Online A/B testing – real‑time comparison of algorithm variants.

Beyond the Algorithm

Recommendation systems can amplify information inequality and echo chambers, but they also enable long‑term value by exposing users to diverse, high‑quality content. Strategies to mitigate bias include promoting exploratory content, expanding the resource pool for niche interests, and integrating algorithmic decisions with product design and hidden user‑experience metrics.