Building LLM Pre-training Data: Markdown Structured Chunking & Rule+LLM Quality Filtering
This guide details a complete pipeline for constructing LLM pre-training datasets: splitting Markdown documents into semantic chunks (1500 chars, no overlap) via heading-aware parsing, greedy packing, and oversized-unit handling, then filtering noise through rule-based checks (length, citations, symbols) and LLM-based quality judgment (completeness, relevance, cleanliness) to produce LLamaFactory-ready JSONL corpora.
