Fun with Large Models

Sep 23, 2026 · Artificial Intelligence

Building LLM Pre-training Data: Markdown Structured Chunking & Rule+LLM Quality Filtering

This guide details a complete pipeline for constructing LLM pre-training datasets: splitting Markdown documents into semantic chunks (1500 chars, no overlap) via heading-aware parsing, greedy packing, and oversized-unit handling, then filtering noise through rule-based checks (length, citations, symbols) and LLM-based quality judgment (completeness, relevance, cleanliness) to produce LLamaFactory-ready JSONL corpora.