Engineering Data for R&D Large Language Models: From Pre‑training to Prompt Design

Core Strategies for Knowledge‑Intensive Domains

Domain‑Adaptive Pre‑Training (DAPT) : Continue pre‑training a foundation model on a domain‑specific corpus, optionally with a domain‑adapted tokenizer. See Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks .

Supervised Fine‑Tuning (SFT) : Align the model using a mixture of general‑purpose and domain‑specific instruction data.

Retrieval‑Augmented Generation (RAG) : Couple the LLM with a domain‑adapted retriever so that generated answers are grounded in up‑to‑date factual documents.

R&D Data Engineering Scope

Narrow R&D data engineering focuses on collecting task‑relevant raw data to build pre‑training corpora or fine‑tuning datasets. Broad R&D data engineering treats all research assets (text, code, specifications, knowledge graphs, etc.) as inputs for model training, fine‑tuning, or retrieval‑based systems.

Pre‑training Phase

Typical pre‑training corpora mix general text (CommonCrawl, dialogue datasets, Books3) with domain‑specific text (multilingual data, scientific papers, source code). After collection, a cleaning pipeline is applied:

Quality filtering using heuristic rules.

Deduplication at document, paragraph, and sentence levels.

Privacy removal to strip personal or sensitive information.

Tokenization with sub‑word vocabularies.

Key references: “A Survey of Large Language Models” and “A Survey of Knowledge‑Enhanced Pre‑trained Models”.

Continual Pre‑training (Domain‑specific Corpus)

When the distribution gap between pre‑training and downstream data is large, perform a second stage of domain‑adaptive or task‑adaptive pre‑training (DAPT/TAPT). Example: legal NER benefits from continued pre‑training on Chinese court documents, statutes, and legal commentary.

Tokenizer adaptation (ChipNeMo four‑step method) :

Train a tokenizer from scratch on the domain corpus.

Identify high‑frequency tokens missing from the general tokenizer.

Extend the general tokenizer’s vocabulary with these new tokens.

Initialize embeddings for the new tokens (e.g., by copying from similar tokens or random init).

Case studies: Lawyer LLaMA (legal domain) and ChipNeMo (chip design) source corpora include court websites, internal NVIDIA documents, and CLUE/CLUE‑C corpora.

Fine‑tuning and Instruction Formatting

Two main adaptation methods after pre‑training:

Instruction Tuning : Build instruction‑formatted examples (task description, input‑output pair, optional demonstrations) and fine‑tune the model in a supervised manner.

Alignment Tuning : Adjust model behavior to align with human values or preferences.

Public instruction datasets (P3, FLAN, OpenAssistant, Self‑Instruct) provide a base. For domain‑specific tasks, experts create curated instruction sets (e.g., legal exam QA, chip‑design script generation). A mixed dataset of ~128 000 samples combines open‑source instruction data (OASST, FLAN, P3) with a small proprietary set.

Typical instruction instance structure:

Task description: <description>
Demonstrations (optional): <example>
Input: <input>
Output: <output>

Quality of instruction data (scale, clarity, inclusion of chain‑of‑thought samples) strongly influences downstream performance.

Retrieval‑Augmented Generation (RAG)

General LLMs often hallucinate or lack up‑to‑date knowledge. RAG mitigates this by retrieving relevant domain passages at inference time and feeding them as context.

Legal RAG example: a RoBERTa‑based retriever trained on legal articles achieves 0.543 Macro Recall@1 and 0.807 Macro Recall@3 on a held‑out set.

Chip‑design RAG uses an E5‑based retriever fine‑tuned on internal documentation, improving answer accuracy for engineering‑assistant queries.

Prompt Engineering for Inference

Effective prompts should include:

Clear task description.

Explicit input specification.

Relevant context (retrieved documents, KG snippets, tables, etc.).

Model‑specific style (e.g., “Let’s think step‑by‑step”).

Few‑shot demonstrations when feasible.

Well‑crafted prompts, especially those that convey code logic or API usage, dramatically raise downstream metrics such as code‑generation success rates.

Coordinated Strategy: Prompt + RAG + SFT

Prompt engineering, SFT, and RAG are complementary:

SFT adapts the base model to the target task.

RAG provides factual grounding from an external knowledge base.

Prompt design steers generation toward the desired format and reasoning style.

Combining all three yields the most reliable and high‑performing system for specialized R&D applications.

Key Takeaways

Data quality and domain relevance are as critical as model size.

Domain‑adaptive tokenizers and continual pre‑training bridge the gap between generic and specialized corpora.

Instruction‑tuned datasets should balance breadth (open‑source data) with depth (expert‑crafted domain examples).

RAG mitigates hallucinations by grounding outputs in retrieved documents.

Prompt design must clearly state the task, input format, context, style, and include concise demonstrations.