Exploring Large Language Models for Recommendation Systems: Experiments and Insights

The recent surge of large language models (LLMs) has prompted researchers to explore their potential in recommendation systems. Two main usage strategies are discussed: using LLMs as a backbone model (e.g., BERT4Rec, UniSRec, P5) and using them as a supplement to generate richer user/item embeddings or textual explanations.

Three ranking paradigms are introduced for top‑K item selection: point‑wise (scoring each item individually), pair‑wise (comparing item pairs), and list‑wise (asking the model to order a set of items directly). The overall evaluation pipeline involves constructing prompts that contain a task description, demonstration examples, and a new input query.

Experiments were conducted on four datasets—MovieLens (movies), Amazon (books and music), and MIND (news)—using baselines such as Random, Pop, Matrix Factorization (MF), and Neural Collaborative Filtering (NCF). Metrics include NDCG and MRR.

Key findings include:

LLMs significantly outperform random and popularity baselines across domains.

ChatGPT achieves the best performance among LLMs, especially in list‑wise ranking where a single query yields strong results.

Traditional models still surpass LLMs when ample interaction data is available, but LLMs excel at cold‑start scenarios due to their world knowledge.

Prompt design matters: zero‑shot prompts beat random/pop, while few‑shot prompts further improve performance; however, adding more examples or history items can introduce noise and degrade results.

Cost analysis shows list‑wise ranking offers the best improvement per unit cost, whereas pair‑wise provides higher performance at higher session cost.

Case studies illustrate successful ranking with explanations, failures where the model refuses to answer or provides incorrect rankings, and the need for post‑processing to handle such issues.

The discussion highlights open questions about combining LLMs with traditional embeddings, the necessity of natural‑language interfaces, and the importance of fine‑tuning LLMs for domain‑specific recommendation tasks.

Overall, the work demonstrates that LLMs can enhance recommendation systems, particularly for cold‑start problems and explainable ranking, while acknowledging the trade‑offs in efficiency and the complementary role of conventional models.