Building LLM Pre-training Data: Markdown Structured Chunking & Rule+LLM Quality Filtering

Why Pre-training Chunking Differs from RAG

Pre-training requires feeding the model fixed-length text units. The author identifies three reasons whole documents cannot be used directly: (1) extreme length variance — books, CVE descriptions, and short news differ by hundreds of times, causing severe padding waste in batch processing; (2) model context limit ( cutoff_len) forces splitting of long documents; (3) knowledge units need clear boundaries so the model learns complete contextual dependencies.

Unlike RAG where chunks are 300–500 characters for retrieval, pre-training chunks should be 1000–2000 characters to capture full argumentation. Overlap is discouraged because repeated content wastes tokens and risks local overfitting.

Parameter Choices for This Cybersecurity Corpus

chunk_size : 1500 characters — target chunk length; too small fragments semantics, too large creates length imbalance

min_size : 100 characters — chunks shorter than this are merged into the previous chunk to avoid fragments

max_slices_per_section : 20 — alert threshold; exceeding suggests whole-page tables or code blocks needing manual review

Markdown Structured Chunking Pipeline

2.1 Regular Expressions for Headings and Code Fences

import os, re, glob, json, hashlib
INPUT_DIR = "./cleaned_md"
OUTPUT_DIR = "./llamafactory_data"
CHUNK_PATH = os.path.join(OUTPUT_DIR, "chunks.jsonl")
CHUNK_SIZE, MIN_SIZE = 1500, 100
MAX_SLICES = 20
HEADING_RE = re.compile(r"^(#{1,6})\s+.+$", re.MULTILINE)
FENCE_RE = re.compile(r"```.*?```", re.DOTALL)

2.2 Section Splitting by Headings

Documents are split at heading lines (# to ######), preserving the heading at the start of each section. Any preamble before the first heading becomes its own section.

def split_sections(md_text):
    """Split by headings, keep heading line; preamble before first heading becomes separate section"""
    matches = list(HEADING_RE.finditer(md_text))
    if not matches:
        return [("", md_text.strip())]
    sections = []
    if matches[0].start() > 0:
        head = md_text[:matches[0].start()].strip()
        if head:
            sections.append(("", head))
    for i, m in enumerate(matches):
        end = matches[i+1].start() if i+1 < len(matches) else len(md_text)
        sections.append((m.group(0).strip(), md_text[m.start():end].strip()))
    return sections

2.3 Atomic Paragraph Extraction

Code fences are kept intact as single units; remaining text is split on blank lines. This prevents splitting inside code blocks that may contain '#' comments or blank lines.

def atomic_paragraphs(block):
    """Extract paragraphs: code fences stay whole, rest split on blank lines"""
    units, cursor = [], 0
    for m in FENCE_RE.finditer(block):
        for p in re.split(r"
\s*
", block[cursor:m.start()]):
            if p.strip():
                units.append(p.strip())
        units.append(m.group(0).strip())  # fence as atomic unit
        cursor = m.end()
    for p in re.split(r"
\s*
", block[cursor:]):
        if p.strip():
            units.append(p.strip())
    return units

2.4 Handling Oversized Units

Units exceeding CHUNK_SIZE are first split by sentence (Chinese/English punctuation). Sentences longer than the limit are hard-windowed. Code fences are never split.

def split_oversized(unit, size, overlap):
    """Split oversized unit: sentence-level first, then hard window; code fences untouched"""
    if unit.lstrip().startswith("```"):
        return [unit]
    sents = re.findall(r".+?[。！？；
.!?;]+", unit, flags=re.DOTALL) or [unit]
    pieces, buf = [], ""
    for s in sents:
        if len(s) > size:
            if buf:
                pieces.append(buf)
                buf = ""
            for i in range(0, len(s), size - overlap):
                pieces.append(s[i:i+size])
        elif len(buf) + len(s) <= size:
            buf += s
        else:
            pieces.append(buf)
            buf = (buf[-overlap:] if overlap else "") + s
    if buf:
        pieces.append(buf)
    return pieces

2.5 Greedy Packing of Atomic Units

Units are packed greedily into chunks up to CHUNK_SIZE. Oversized units are split first via split_oversized.

def pack_units(units, size):
    """Greedy pack atomic units into chunks; split oversized units before packing"""
    windows, cur = [], []
    def flush():
        if cur:
            windows.append("

".join(cur))
            cur[:] = []
    for unit in units:
        if len(unit) > size:
            flush()
            windows.extend(split_oversized(unit, size))
            continue
        if not cur or len("

".join(cur)) + 2 + len(unit) <= size:
            cur.append(unit)
        else:
            flush()
            cur.append(unit)
    flush()
    return windows

2.6 Document-Level Orchestration

chunk_markdown

processes each section: short sections accumulate in a buffer ( buf) until they reach CHUNK_SIZE, then are packed together. Long sections are split individually. If a section yields >20 chunks, an alert prints for manual inspection (likely tables/code). Post-processing merges chunks shorter than MIN_SIZE into the previous chunk (or next chunk for the first chunk). Each chunk receives a 12-char MD5 ID derived from source, index, and text prefix.

def chunk_markdown(md_text, source):
    chunks, buf = [], []
    for title, block in split_sections(md_text):
        if len(block) >= CHUNK_SIZE:
            if buf:
                chunks.extend(pack_units(buf, CHUNK_SIZE))
                buf = []
            slices = pack_units(atomic_paragraphs(block), CHUNK_SIZE)
            if len(slices) > MAX_SLICES:
                print(f"[ALERT] {source} section '{title[:20]}' produced {len(slices)} chunks, manual check needed")
            chunks.extend(slices)
        else:
            buf.append(block)
            if len("

".join(buf)) >= CHUNK_SIZE:
                chunks.append("

".join(buf))
                buf = []
    if buf:
        chunks.extend(pack_units(buf, CHUNK_SIZE))
    # merge too-short chunks
    merged = []
    for c in chunks:
        if merged and len(c) < MIN_SIZE:
            merged[-1] += "

" + c
        else:
            merged.append(c)
    if len(merged) > 1 and len(merged[0]) < MIN_SIZE:
        merged[1] = merged[0] + "

" + merged[1]
        merged = merged[1:]
    records = []
    for i, text in enumerate(merged):
        cid = hashlib.md5(f"{source}-{i}-{text[:64]}".encode()).hexdigest()[:12]
        records.append({"id": cid, "source": source, "index": i, "text": text})
    return records

2.7 Batch Processing Entry Point

def run_chunk():
    files = glob.glob(os.path.join(INPUT_DIR, "**", "*.md"), recursive=True)
    total = 0
    with open(CHUNK_PATH, "w", encoding="utf-8") as f:
        for path in files:
            with open(path, encoding="utf-8") as fp:
                records = chunk_markdown(fp.read(), os.path.basename(path))
            for r in records:
                f.write(json.dumps(r, ensure_ascii=False) + "
")
            total += len(records)
    print(f"Processed {len(files)} documents, produced {total} chunks -> {CHUNK_PATH}")

Rule-Based Pre-filtering Before LLM Judgment

To save API costs, deterministic rules quickly discard obvious noise. Only ambiguous chunks go to the LLM.

DROP_TITLES = ("参考文献", "References", "致谢", "Acknowledgement",
               "目录", "作者简介", "版权声明", "版权所有")

def rule_filter(rec):
    """Return (keep, reason)"""
    text = rec["text"]
    plain = re.sub(r"[\s#>*\-|\[\]()]", "", text)
    if len(plain) < MIN_SIZE:
        return False, "effective char count too low"
    first_line = next((ln for ln in text.splitlines() if ln.strip()), "")
    if any(k in first_line for k in DROP_TITLES):
        return False, f"no-value section: {first_line[:20]}"
    lines = [ln for ln in text.splitlines() if ln.strip()]
    cite_hits = sum(bool(re.search(r"https?://|\[\d+\]|arxiv|doi", ln)) for ln in lines)
    if lines and cite_hits / len(lines) > 0.5:
        return False, "suspected reference/link list"
    symbol_ratio = len(re.findall(r"[^\w\s\u4e00-\u9fa5]", text)) / max(len(text), 1)
    if symbol_ratio > 0.35:
        return False, "high special-symbol ratio, suspected garbled text"
    return True, ""

The four rules target: (1) ultra-short fragments, (2) known no-value section headers, (3) citation/link-dense blocks (likely bibliographies), (4) high symbol-to-text ratio (OCR garbage).

LLM-Based Quality Judgment

Chunks passing rules are evaluated by an LLM (qwen-plus via Aliyun Bailian) on three dimensions:

Completeness — not a heading fragment, half-table, truncated formula/code.

Relevance — contains substantive cybersecurity/computer-science knowledge, not ads, navigation, acknowledgements, or bibliographies.

Cleanliness — no OCR garble, meaningless symbol piles, or promotional content.

Prompt (truncated for brevity):

JUDGE_PROMPT = """You are a cybersecurity corpus quality auditor. Judge if the text slice is suitable for LLM pre-training.
Dimensions:
1. Completeness: not heading fragment, half-table, truncated formula/code
2. Relevance: contains cybersecurity/CS knowledge, not ads/navigation/acknowledgements/bibliographies
3. Cleanliness: no OCR garble, meaningless symbols, promotional content
Output JSON only: {"keep": 1 or 0, "score": 0-10 integer, "reason": "brief reason"}"""

Judgment runs with 8 parallel workers, retry logic (3 attempts, exponential backoff), and checkpointing every 100 chunks. Failures default to keep=True to avoid false drops.

Results & LLamaFactory Format

On the test corpus (31 chunks), the pipeline dropped 15 chunks. The final chunks_judged.jsonl contains only chunks with keep=true. LLamaFactory expects a JSONL file where each line has a single text field; extracting that field from the judged records yields the ready-to-use pre-training dataset.

Summary

This article delivers an end-to-end, reproducible pipeline: (1) heading-aware Markdown splitting, (2) atomic paragraph extraction preserving code fences, (3) sentence-aware oversized splitting with greedy packing, (4) short-chunk merging, (5) four high-precision rule filters, (6) LLM-as-judge on completeness/relevance/cleanliness, producing a clean chunks_judged.jsonl directly consumable by LLamaFactory for pre-training.