How to Build a Crash‑Safe KV Store Engine in Go

Project Overview

EasyDB is a lightweight key‑value store written in Go. It keeps in‑memory records in a SkipList and guarantees durability and crash‑safety through a Write‑Ahead Log (WAL). The source code (~1 000 lines) is hosted at https://github.com/gofish2020/easydb.

Code Logic Structure

Opening a database with easydb.Open creates a *DB instance. During initialization the engine reads existing segment files, parses each LogRecord , and loads the records into a skiplist . This restores the memory tables ( *memtable ) and separates them into an active and an immutable table, following the LSM‑Tree architecture. Each *memtable contains a skiplist for in‑memory data and a *wal object that records changes to disk. The WAL splits its log files into segments of configurable size, analogous to Kafka’s log‑segmenting.

The db.Put method starts a write transaction by creating a batch and acquiring an exclusive lock on the DB. User data is placed into batch.pendingWrites . When batch.Commit is invoked, all pending writes are flushed atomically to both the in‑memory skiplist and the WAL, after which the lock is released.

The db.Get method starts a read transaction by creating a batch and acquiring a shared lock. It scans the memory tables in reverse order (starting from the most recent *memtable ) to locate the requested key, then releases the lock.

The db.Delete operation follows the same transaction pattern as Put , recording a delete marker in the WAL and removing the key from the in‑memory structures.

Example Usage

package main

import (
    "fmt"
    "github.com/gofish2020/easydb"
    "github.com/gofish2020/easydb/utils"
)

func main() {
    options := easydb.DefaultOptions
    options.DirPath = utils.ExecDir() + "/data"

    db, err := easydb.Open(options)
    if err != nil {
        panic(err)
    }
    defer func() { _ = db.Close() }()

    // put a key
    err = db.Put([]byte("name"), []byte("easydb"), nil)
    if err != nil {
        panic(err)
    }

    // get the key
    val, err := db.Get([]byte("name"))
    if err != nil {
        panic(err)
    }
    println(string(val))

    // delete the key
    err = db.Delete([]byte("name"), nil)
    if err != nil {
        panic(err)
    }

    // attempt to get the deleted key
    val, err = db.Get([]byte("name"))
    if err != nil {
        if err == easydb.ErrKeyNotFound {
            fmt.Println("key not exist")
            return
        }
        panic(err)
    }
    println(string(val))
}

WAL Log Format

The WAL file is divided into segment files according to the SegmentSize configuration.

Each segment is split into 32 KB blocks; a block stores multiple chunk records.

Each chunk consists of a 7‑byte header (4‑byte checksum, 2‑byte length, 1‑byte type) followed by the payload. The checksum validates length + type + payload to ensure data integrity.

A logical record may span several chunks.

If a block cannot accommodate a new chunk, the remaining space is padded with invalid bytes.

Implementation details are largely derived from the LotusDB project.