Semantic Embedding with Large Language Models: A Comprehensive Survey

This article provides a comprehensive survey of semantic embedding techniques enhanced by Large Language Models (LLMs). Semantic embedding, which maps text into semantic space, is a core technology in natural language processing, information retrieval, and recommendation systems for capturing deep semantic information in text.

The article begins by reviewing the evolution of semantic embedding methods from early approaches like Word2vec and GloVe to context-sensitive models like BERT, RoBERTa, and Sentence-BERT, and more recent contrastive learning frameworks like SimCSE. It then compares LLM-based semantic embeddings with traditional approaches, highlighting differences in model structure, training methods, embedding quality, and application scenarios.

The core content focuses on two main research directions for improving semantic embeddings using LLMs: (1) Synthetic Data Generation - leveraging LLMs to generate high-quality training data, including models like E5-mistral-7b-instruct, SFR-Embedding-Mistral, and Gecko; (2) Model Backbone - using LLMs as the core architecture for embedding, including NV-Embed-v2, BGE-EN-ICL, Echo-mistral, LLM2Vec, GRIT, GTE-Qwen1.5-7B-instruct, and stella_en_1.5B_v5.

Key techniques discussed include: contrastive learning with prompts, in-context learning (ICL), bidirectional attention mechanisms, two-stage instruction tuning, knowledge distillation from LLMs, and multi-task training. The article also addresses challenges including computational resource requirements, data privacy concerns, prompt quality dependency, and the need for better interpretability.