How Easy Dataset Automates High‑Quality LLM Fine‑Tuning Data from Unstructured Docs

Hello, welcome to Code Secret Garden, I am ConardLi.

Many researchers have been using Easy Dataset in their projects and need proper citations; together with Dr. Yaowei Zheng and members of the Beihang ACT lab we have written a paper titled Easy Dataset to facilitate citation.

Paper URL: https://arxiv.org/abs/2507.04009

The paper is posted on arXiv and submitted to EMNLP 2025.

1. Research Background and Problem Statement

Large language models (LLMs) excel on general tasks but adapting them to specific domains such as finance or medicine remains challenging due to the scarcity of high‑quality domain data.

Difficulty parsing heterogeneous unstructured documents (text, tables, images in PDFs) leads to incomplete or inaccurate information extraction.

Generated QA pairs often lack diversity and fidelity, causing over‑fitting or domain bias.

Lack of an end‑to‑end user‑friendly interface makes it hard for non‑technical users, and human‑in‑the‑loop mechanisms are missing, jeopardizing data quality.

To address these issues, the paper proposes Easy Dataset , a GUI‑based framework that automates the entire pipeline from unstructured documents to high‑quality fine‑tuning data, and has already earned over 9,200 stars on GitHub.

GitHub: https://github.com/ConardLi/easy-dataset

2. Core Contributions

Unified End‑to‑End Framework : Integrates adaptive document processing with persona‑driven data synthesis to generate fine‑tuning data with minimal manual intervention.

User‑Friendly Visual Interface : Provides an intuitive GUI for non‑technical users and incorporates human‑machine interaction to continuously refine intermediate results.

Empirical Effectiveness : In a financial QA task, models fine‑tuned on data generated by the framework achieve significantly higher domain performance while retaining general knowledge.

3. Framework Design: Core Modules of Easy Dataset

The system consists of two main components— Adaptive Document Processing and Persona‑Driven Data Synthesis —supplemented by model configuration and dataset export features, all supporting human‑in‑the‑loop interaction.

3.1 Adaptive Document Processing

The goal is to convert raw unstructured documents into coherent text blocks through two steps: document parsing and hybrid chunking.

Document Parsing : Precise extraction for multiple formats (plain text, Markdown, DOCX, PDF).

Plain text/Markdown: retain semantics with lightweight processing.

DOCX: convert to Markdown via Mammoth, removing noisy formatting.

Complex PDFs: layout analysis to separate text and visual regions; visual text is processed with vision‑language models and tools like MinerU.

Hybrid Chunking : Split parsed text into semantically coherent chunks that fit LLM context windows, using length‑driven, structure‑driven, and manual adjustment strategies, with a visual interface for fine‑tuning.

3.2 Persona‑Driven Data Synthesis

Generates high‑quality, diverse QA pairs by leveraging role‑guided prompts.

Basic QA Generation : Combine text blocks with customizable system prompts to control style, audience, and tone; introduce random punctuation deletion to reduce over‑reliance on punctuation.

Persona‑Driven QA Generation (Core Innovation) : Define a Genre‑Audience (GA) pair where “Genre” specifies QA style (e.g., financial news summary) and “Audience” specifies the asker’s background (e.g., executive, tax expert). Two‑stage generation first creates diverse GA pairs, then guides the LLM to produce QA pairs from multiple perspectives.

Human‑in‑the‑Loop Optimization : Visual interface allows users to review and edit QA pairs, while the LLM can automatically refine answers and reasoning steps.

3.3 Model Configuration and Dataset Export

Model Configuration : GUI supports multiple LLMs (API‑based or local such as Ollama) with adjustable generation parameters (temperature, top‑p, etc.) for different domains.

Dataset Export : Export to JSON, JSONL, CSV, compatible with Alpaca, ShareGPT schemas, and automatically generate LlamaFactory training configs for seamless fine‑tuning.

4. Experimental Validation

The framework’s effectiveness is evaluated on a financial QA task.

Setup :

Data: 5 latest financial reports (beyond LLM knowledge cutoff).

Fine‑tuned model: Qwen2.5‑7B‑Instruct, compared against the base model, naive synthetic data, and persona‑driven synthetic data.

Metrics: Domain performance assessed by DeepSeek‑V3 as a judge; general ability measured by MMLU, CMMLU, HellaSwag, etc.

Results :

Domain score: persona‑driven data yields 59.6, surpassing naive synthetic (57.0) and the base model (3.2).

General ability: scores remain comparable to the base model (e.g., 75.5 vs. 76.3), showing that domain knowledge is acquired without sacrificing general competence.

5. Comparison with Existing Tools

Table‑1 (not shown) contrasts Easy Dataset with Distilabel, Kiln, Curator, highlighting advantages:

Adaptive Parsing : Precise extraction across multiple formats, including complex PDFs.

Human‑in‑the‑Loop : Full‑process visual interface for real‑time review and adjustment.

Persona‑Driven : GA pairs generate diverse QA pairs for better domain adaptation.

End‑to‑End GUI : Integrated parsing, chunking, generation, and export, enabling non‑technical users to operate the pipeline directly.

6. Conclusion

Easy Dataset

combines adaptive document processing, persona‑driven synthesis, and an interactive GUI to provide an efficient, user‑friendly solution for LLM domain fine‑tuning. It lowers the technical barrier for building domain data, improves data quality and diversity through role‑driven generation, and has already demonstrated strong results in finance, offering a new practical path for domain adaptation of LLMs.

Final Links

Paper: https://arxiv.org/abs/2507.04009

GitHub: https://github.com/ConardLi/easy-dataset

@misc{miao2025easydataset,
  title={Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documents},
  author={Ziyang Miao and Qiyu Sun and Jingyuan Wang and Yuchen Gong and Yaowei Zheng and Shiqi Li and Richong Zhang},
  year={2025},
  eprint={2507.04009},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2507.04009}
}