InstEmb: Future‑Aware Instruction‑Following Embeddings for Product Knowledge Representation

Introduction

Embedding models are fundamental for retrieval, clustering, semantic matching and RAG systems. Traditional embeddings focus on input‑intrinsic semantics, but instruction‑following scenarios require the model to anticipate how it will answer a query. InstEmb, presented at ICML 2026, addresses this gap by letting embeddings capture "future output" semantics without additional decoding.

Problem Statement

In product‑search examples, a query like "What features does this tent have for outdoor use?" is semantically close to candidate items, yet the user cares about specific dimensions (durable / waterproof vs compact / lightweight). Standard LLM embeddings, which usually take the hidden state of the last input token, miss the latent semantics that would appear in the model's eventual answer.

Method

Core Idea : Append a set of learnable look‑ahead tokens to the input sequence. During training, a frozen teacher model receives the true output (truncated gold answer) while the student receives the look‑ahead tokens. The student’s look‑ahead token states are trained to align with the teacher’s output‑conditioned hidden states, thereby learning output‑aware semantics.

Student input: [instruction + input + <eos> + look-ahead tokens]
Teacher input: [instruction + input + <eos> + truncated gold output]

Two self‑distillation objectives are explored:

MSE loss : Directly regress the student’s hidden states to the teacher’s.

KL divergence loss : Align the output probability distributions via the language‑model head.

To preserve stable input semantics, InstEmb applies multi‑view contrastive learning on the final input token. Four views are constructed for each sample: two student encodings with different dropout masks, the frozen teacher encoding, and the student encoding of the gold output. An InfoNCE loss encourages the final token to remain input‑intrinsic while integrating teacher and output information.

Finally, Dual‑Anchor Alignment Pooling (DAAP) combines two anchors: the hidden state of the last input token (input‑intrinsic) and the average hidden state of the look‑ahead tokens (output‑aware). The final embedding is the average of these two anchors, explicitly reflecting both semantic sources.

Experiments

InstEmb is built on LLaMA‑3‑8B‑Instruct and trained for one epoch on ~200 k abstractive QA examples from 11 datasets (learning rate 5e‑6, Adam). During inference, eight look‑ahead tokens are used.

Evaluation covers three groups of tasks:

Instruction‑following retrieval: FollowIR, InfoSearch.

Instruction‑following embedding: Inst.STSb, IntentEmotion, NYTCluster.

General sentence‑embedding benchmarks: AskUbuntu, 20News, SciDocsRR, StackOverflowDup.

Key results:

InstEmb‑MSE achieves an average score of 28.5 and +15.6 p‑MRR on FollowIR, surpassing strong baselines such as FollowIR‑7B and PromptRetriever.

InstEmb‑MSE‑DAAP reaches 67.08 average on instruction‑following embedding tasks, beating the Inbedder reimplementation (59.90).

InstEmb‑KL‑DAAP attains 63.39 on general embedding benchmarks, showing that the method retains broad semantic capability.

Ablation Studies

Removing the contrastive view or the output‑aware anchor degrades performance markedly, confirming their importance. The MSE objective excels on fine‑grained instruction tasks, while KL is more robust on generic embedding tasks. Varying the look‑ahead token length shows a steep gain from 0 to 1 token, with diminishing returns for longer sequences on short‑output tasks.

Interpretability Analysis

Attention visualizations reveal that the original LLaMA‑3 model concentrates attention at the sequence start (attention sink). After InstEmb training, attention shifts toward the system‑prompt and instruction boundaries, indicating more task‑relevant focus.

Cosine similarity between look‑ahead tokens and gold output tokens is higher than between the last input token and later positions, confirming that look‑ahead tokens indeed capture output‑aware semantics.

Conclusion

InstEmb demonstrates that instruction‑following embeddings can be enriched with future output semantics through learnable look‑ahead tokens, self‑distillation, multi‑view contrastive learning, and dual‑anchor pooling. The approach yields large gains on retrieval and embedding benchmarks while requiring only a single prefilling pass, eliminating the latency of decode‑then‑encode pipelines. Future embedding designs can thus benefit from anticipating model outputs within the continuous representation space.