How Arkit’s Go Plugin Architecture Boosts Data Collection and Monitoring

1. Overview

Arkit is the unified monitoring agent of the Yunzhou Observation product, serving as the foundation that collects system metrics and process information from Linux and Windows, reads various data sources such as files, message queues, and network streams, and forwards them to multiple storage targets with flexible parsing configurations.

2. Data Collection and Parsing

Arkit supports dozens of data sources, including files, and gathers hundreds of monitoring metrics. It performs edge‑side parsing to transform data before transmission, reducing redundancy, network bandwidth, and storage consumption while simplifying downstream processing. Collected data can be sent to various destinations.

After raw data is read, it can be sent directly or passed through the parsing module for operations such as CSV, JSON, Grok, Nginx log parsing, or custom plugin parsing, which can be hot‑updated without restarting Arkit.

Data format parsing (CSV, JSON, Grok, Nginx logs)

Custom plugin parsing (user‑written Go plugins, hot‑updated)

Users can develop plugins with the provided web editor, compile them on the backend, test with sample data, and select the custom plugin when configuring a collection task.

3. Plugin Functionality

Arkit is written in Go, so plugins must be built as Go dynamic plugins. While embedding script engines (Lua, JS, Python) offers flexibility, their execution speed is far lower than native Go code, which is unacceptable for high‑volume log processing.

The chosen solution is Go’s native plugin mechanism. The architecture of Arkit’s plugin module is illustrated below.

3.1 Origin of Go Plugins

Go plugins were introduced in Go 1.8 to enable dynamic code loading without recompiling the whole program, addressing needs such as micro‑service extensibility, plugin‑based architecture, hot updates, and third‑party module integration. They provide zero‑copy memory mapping, efficient symbol resolution, and near‑native call performance.

3.2 First Demo

<code>// plugin-impl.go
package main

import "fmt"

func Hello() string {
    return "Hello from golang plugin"
}

// plugin must have a main package and main function
func main() {}
</code>

<code># Build the plugin
go build -buildmode=plugin -o plugin.so plugin-impl.go
</code>

<code>// main.go
package main

import (
    "fmt"
    "plugin"
)

func main() {
    // Load plugin
    p, err := plugin.Open("plugin.so")
    if err != nil {
        panic(err)
    }
    // Lookup symbol
    sym, err := p.Lookup("Hello")
    if err != nil {
        panic(err)
    }
    // Type assertion
    hello := sym.(func() string)
    fmt.Println(hello())
}
</code>

3.3 Internal Implementation

Key components of Go plugins include:

Symbol table management – exported symbols are generated at compile time and looked up at runtime using ELF/Mach‑O formats.

Memory management – plugins are loaded into isolated memory spaces via memory‑mapping, supporting garbage collection.

Type system – plugins and the host must be built with the same Go version; type information is matched at runtime to ensure safety.

3.4 Core Data Structures

<code>type Plugin struct {
    pluginpath string               // plugin file path
    syms       map[string]interface{} // symbol table
    loaded     bool                 // whether loaded
}

type _dl_info struct {
    dli_fname *byte // file name
    dli_fbase *byte // base address
    dli_sname *byte // symbol name
    dli_saddr *byte // symbol address
}
</code>

3.5 Performance Advantages

Go plugins achieve high performance through zero‑copy memory mapping, O(1) hash‑based symbol lookup, delayed loading, symbol caching, and optimized type‑checking caches.

<code>// Simplified memory‑mapping implementation
func mmap(file *os.File, size int) ([]byte, error) {
    return syscall.Mmap(int(file.Fd()), 0, size,
        syscall.PROT_READ|syscall.PROT_EXEC,
        syscall.MAP_SHARED)
}
</code>

Benchmark results show only a ~26 % overhead compared with direct function calls:

<code>BenchmarkPluginCall-8    10000000    158 ns/op
BenchmarkDirectCall-8    10000000    125 ns/op
</code>

4. Custom Plugin Practice in Arkit

If built‑in plugins are insufficient, users can write Go plugins to perform flexible parsing and transformation.

Interface Definition

<code>type PluginParser interface {
    Parse(string, map[string]string) map[string]interface{}
}

type pluginImp struct{}
var Plugin pluginImp
</code>

Example 1 – Append Hostname to Log Line

<code>package main // do not modify

import "os"

var hostname string

func getHostname() string {
    h, err := os.Hostname()
    if err != nil {
        h = "unknown"
    }
    return h
}

// parse2String must keep this signature
func parse2String(line string, params map[string]string) string {
    if hostname == "" {
        hostname = getHostname()
    }
    return hostname + " " + line
}
</code>

Example 2 – Parse Nginx Log to JSON

<code>package main // do not modify

import (
    "regexp"
    "strings"
)

var regex = regexp.MustCompile(`^"([^\"]*)"\s+(\d+)\s+(\d+)\s+"([^\"]*)"\s+"([^\"]*)"\s+"([^\"]*)"\s+([\d]*)\s+([\d.]*)\s+([0-9\-\.:]*)\s+([\d]+)\s+([\d.]*)$`)

// parse2JSON must keep this signature
func parse2JSON(line string, params map[string]string) map[string]interface{} {
    results := make(map[string]interface{}, 1)
    // parsing logic omitted for brevity
    return results
}
</code>

5. Limitations

Platform: currently only Linux is supported; Windows is not.

Version dependency: plugin and host must use the same Go version and compatible third‑party libraries.

Debugging complexity: plugin debugging and error tracing are more difficult.

About Yunzhou Observation

Yunzhou Observation, launched by 360 Zhihui Cloud, is an all‑in‑one data collection and monitoring solution that covers infrastructure, application performance, and cloud‑native business metrics, providing full‑stack observability to help users detect and resolve system issues quickly.