Introducing Karta: A Lightweight Go Library for Asynchronous and Batch Function Task Processing

In early 2024, the author, a veteran Go developer, presents Karta, a lightweight Go library designed to simplify and accelerate asynchronous and batch function task processing.

Typical batch tasks using simple for loops become inefficient at large scale, and while channels and goroutines can address concurrency, they often require extensive boilerplate for queue management, error handling, and timeout control.

Karta offers two operating modes: Pipeline for unknown‑size task streams, where tasks are queued and processed by a configurable pool of workers; and Group for known‑size batches, which splits tasks into equally sized groups and executes them concurrently.

The library’s API is intentionally minimal, exposing a Config struct with fluent WithWorkerNumber , WithHandleFunc , WithResult , and WithCallback options, and providing Group and Pipeline objects with methods such as Map , Submit , SubmitWithFunc , and lifecycle controls like Stop .

Example code demonstrates how to configure a two‑worker group to process a slice of integers:

package main

import (
    "time"
    k "github.com/shengyanli1982/karta"
)

func handleFunc(msg any) (any, error) {
    time.Sleep(time.Duration(msg.(int)) * time.Millisecond * 100)
    return msg, nil
}

func main() {
    c := k.NewConfig()
    c.WithHandleFunc(handleFunc).WithWorkerNumber(2).WithResult()

    g := k.NewGroup(c)
    defer g.Stop()

    r0 := g.Map([]any{3, 5, 2})
    println(r0[0].(int))
}

Another snippet shows a pipeline with a custom delayed queue and mixed task submissions:

package main

import (
    "fmt"
    "time"
    k "github.com/shengyanli1982/karta"
    "github.com/shengyanli1982/workqueue"
)

func handleFunc(msg any) (any, error) {
    fmt.Println("default:", msg)
    return msg, nil
}

func main() {
    c := k.NewConfig()
    c.WithHandleFunc(handleFunc).WithWorkerNumber(2)
    queue := k.NewFakeDelayingQueue(workqueue.NewSimpleQueue(nil))
    pl := k.NewPipeline(queue, c)
    defer func() { pl.Stop(); queue.Stop() }()

    _ = pl.Submit("foo")
    _ = pl.SubmitWithFunc(func(msg any) (any, error) {
        fmt.Println("SpecFunc:", msg)
        return msg, nil
    }, "bar")

    time.Sleep(time.Second)
}

By abstracting common concurrency patterns, Karta reduces repetitive code, improves maintainability, and offers a fast, lightweight solution for most batch‑processing scenarios in Go projects.