How Modern Programming Languages Build Documentation Systems with Markdown DSLs

Last year we discussed IDE support for programming languages; this article shifts focus to how different languages provide documentation support, especially the documentation found in standard libraries. The goal is to lay a theoretical foundation for the Datum Lang documentation system.

Next‑Generation Language Documentation Systems

Programmers often avoid writing documentation, yet it remains essential for standard libraries, APIs, and SDKs. Each language adopts distinct practices.

We previously analyzed Rust's rustdoc automation. Extending that analysis, we surveyed mainstream languages since 2012, quickly extracting patterns from their codebases.

From the survey:

Rust, Julia, Dart, Kotlin, and Swift all use Markdown as a base and apply custom extensions. Swift adds special markers such as Complexity for algorithmic complexity notes.

Julia's DocumenterTools heavily customizes Markdown, using index.md for special processing and adding many custom syntaxes.

Both Rust and Julia validate code snippets in documentation; Rustdoc compiles and runs examples using assertions like assert_eq!, while Julia's DocumenterTools supports setup blocks and additional assertions.

R Markdown demonstrates an executable‑document approach that blends code and narrative, a topic for future articles.

Rust Documentation Test Example

A simple Rust code block example:

# Examples
```rust
assert_eq!(2 + 2, 4);
```

Rustdoc processes the markdown, extracts Rust code blocks, generates compilable test code, compiles it, and runs the test. The steps are:

Parse Markdown and locate Rust code blocks (default to Rust if language not specified).

Transform the block into compilable code.

Compile the generated test code (failure indicates a test failure).

Execute the compiled test.

The resulting test harness looks like:

#![allow(unused)]
fn main() {
    #[allow(non_snake_case)]
    fn _doctest_main__some_unique_name() {
        assert_eq!(2 + 2, 4);
    }
    _doctest_main__some_unique_name();
}

If compilation and execution succeed, the documentation example is correct. See Rustdoc source in the librustdoc repository for details.

Building a Documentation System: Markdown as a DSL

The core idea is to treat Markdown as an extensible domain‑specific language (DSL). By extending Markdown, we can add custom functionality and visualizations as needed.

0. Markdown Foundations

Markdown has become the de‑facto standard for developer documentation.

1. Designing Extensible Documentation DSL

Key techniques:

Executable code blocks : Allow code blocks to carry extra meaning. Julia's Documenter.jl defines eval, repl, and jldoctest to run code, display REPL output, or perform doctests.

Custom syntax : Introduce symbols for advanced features. Kotlin's Dokka uses [] to link elements, blending with Markdown.

Document orchestration : Custom directives such as @ref, @docs, @meta, and @content enable referencing, metadata, and content management within Markdown.

2. Designing for Accuracy: Documentation Tests

To keep documentation and code in sync, documentation tests are employed. For Rust and Julia, typical checks include:

Code compiles without errors.

Code runs without runtime errors, supporting language‑specific assertions.

Program output matches the documented output.

The implementation follows the Rust example described earlier.

3. Building an Open Collaboration Platform

A documentation system should be open‑source, allowing community contributions to attract more developers.

Other Thoughts

Life is short—let's program with Markdown.