How Programming Languages Really Work: Inside the Compiler Journey

How Programming Languages Really Work

Understanding the inner workings of compilers helps you use them more efficiently. This guide follows the compilation process step‑by‑step, offering many links, code samples, and diagrams to aid comprehension.

Author’s Note

This is a rewritten version of a popular Medium article, now using Rust as the primary language because of its clarity and performance for compiler development.

Simple Introduction

What Is a Compiler?

A compiler is software that reads a text file (source code) and, after extensive processing, produces a binary file. It translates human‑readable code into machine‑readable instructions.

<code>// An example compiler that turns 0s into 1s, and 1s into 0s. fn main() { let input = "1 0 1 A 1 0 1 3"; let output: String = input.chars().map(|c| { if c == '1' { '0' } else if c == '0' { '1' } else { c } }).collect(); println!("{}", output); // 0 1 0 A 0 1 0 3 } </code>

What Does a Compiler Do?

In short, a compiler reads source code and produces a binary file. The process includes:

Reading individual tokens from the source.

Classifying tokens (identifiers, numbers, symbols, operators).

Using pattern matching to build an expression tree (AST).

Traversing the AST to generate binary data (often via assembly).

Although the final output is binary, compilers typically generate assembly code first, which is then assembled into machine code.

What Is an Interpreter?

An interpreter also reads source code but skips code generation, executing the abstract syntax tree (AST) directly. This provides faster feedback during debugging, though it requires the interpreter to be present on the target machine.

1. Lexical Analysis

Lexical analysis (tokenization) splits the input into meaningful units such as numbers, identifiers, and symbols. For example, the string 12+3 is broken into tokens: 12, +, 3.

Rust code can be used to implement a simple lexer that converts characters into 32‑bit integers and recognizes the plus sign.

<code>int main() { int a; int b; a = b = 4; return a - b; } </code>

2. Parsing

The parser consumes tokens from the lexer and checks whether they fit a defined grammar, building an abstract syntax tree (AST). A common approach is recursive‑descent parsing.

Example EBNF for simple arithmetic:

<code>expr = additive_expr ; additive_expr = term (('+' | '-') term) ; term = number ; </code>

Parsing 12+3 yields an AST representing the addition operation.

3. Code Generation

The code generator walks the AST and emits target code or assembly. Tools like Godbolt Compiler Explorer let you see the generated assembly for a given source.

Example on Godbolt

Back‑ends can produce code for multiple languages; for instance, the Haxe compiler can target C++, Java, Python, and more.

After assembly is generated, it is assembled into object files (.o or .s), then linked to produce an executable, shared library, or static library.

Summary

Understanding compilers empowers you to use programming languages more effectively and may inspire you to design your own language someday.

Further Reading

Crafting Interpreters – guide to building C and Java interpreters.

Write a Compiler – a highly useful tutorial.