Fine‑Tuning Text Embeddings for Domain‑Specific Search: A Complete Walkthrough

Introduction

Embedding models convert text into semantic vectors, enabling retrieval and classification. Generic embeddings may underperform on domain‑specific tasks, so fine‑tuning is presented as a solution. The article uses a "job‑search matching" scenario to illustrate the process.

Why Simple Similarity Search Falls Short

Semantic search retrieves items with high vector similarity, but similarity does not guarantee relevance. For example, a query about updating a payment method may retrieve a billing‑section paragraph that is semantically close yet does not answer the user’s question.

Fine‑Tuning Embedding Models

Fine‑tuning adjusts a pre‑trained embedding model with additional training on domain data. In the job‑matching case, the model must understand domain‑specific terms such as "scaling" and "instances" in cloud‑computing contexts.

Data Preparation

Collect positive‑negative pairs from the datastax/linkedin_job_listings HuggingFace dataset.

Generate synthetic, human‑like queries for each job description using OpenAI's Batch API (GPT‑4o‑mini), reducing cost by 50% (total $0.12).

Clean job descriptions to stay within the 512‑token limit of most embedding models.

For each positive pair, select a negative job description with the lowest similarity using SentenceTransformer("all‑mpnet‑base‑v2") embeddings.

Split the dataset into train (80%), validation (10%), and test (10%) sets and push to the HuggingFace Hub.

Choosing a Pre‑Trained Model

Several base and semantic‑search models are compared via a triplet evaluator. The model sentence‑transformers/all‑distilroberta‑v1 achieved the highest cosine accuracy on the validation set (≈0.88) and was selected for fine‑tuning.

Selecting a Loss Function

Based on the (anchor, positive, negative) triplet format, MultipleNegativesRankingLoss from Sentence‑Transformers is used.

Training Configuration

from sentence_transformers import SentenceTransformerTrainingArguments
num_epochs = 1
batch_size = 16
lr = 2e-5
finetuned_model_name = "distilroberta-ai-job-embeddings"
train_args = SentenceTransformerTrainingArguments(
    output_dir=f"models/{finetuned_model_name}",
    num_train_epochs=num_epochs,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    learning_rate=lr,
    warmup_ratio=0.1,
    batch_sampler=BatchSamplers.NO_DUPLICATES,
    eval_strategy="steps",
    eval_steps=100,
    logging_steps=100,
)

Model Fine‑Tuning

from sentence_transformers import SentenceTransformerTrainer
trainer = SentenceTransformerTrainer(
    model=model,
    args=train_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["validation"],
    loss=loss,
    evaluator=evaluator_valid,
)
trainer.train()

Evaluation

After fine‑tuning, the model reaches 99% accuracy on the validation set and 100% on the test set, demonstrating the effectiveness of domain‑specific fine‑tuning.

Optional Model Publishing

# push model to Hugging Face Hub
model.push_to_hub(f"shawhin/{finetuned_model_name}")

Conclusion

Fine‑tuning transforms a generic embedding model into a domain‑adapted one, dramatically improving the relevance of semantic search results for specialized tasks such as matching job seekers with appropriate job descriptions.