Using Go’s unique Package for Efficient String Interning

What Is String Interning

String interning stores only one copy of identical string data in memory, allowing multiple variables to reference the same data and thus reducing memory allocations. Java developers are familiar with this technique, and Python’s sys.intern function provides similar functionality, automatically interning short strings without special characters.

Interning creates a "string cache" that can be accessed via variables or constants. Because strings are immutable in Java, Python, and Go, shared references cannot be accidentally modified.

Benefits and Risks

Interning reduces the number of memory allocations and overall memory usage, which is especially valuable when processing large amounts of repeated textual data (e.g., deserializing logs). However, if the cache hit rate is low, the overhead of maintaining the cache can outweigh any gains and increase GC pressure.

Manual Interning in Go

The basic implementation uses a map from string to string and clones the input to avoid retaining references to large substrings:

type StringIntern struct {
    m map[string]string
}

func (s *StringIntern) Intern(str string) string {
    ret, ok := s.m[str]
    if !ok {
        ret = strings.Clone(str)
        s.m[str] = ret
    }
    return ret
}

var si StringIntern
s1 := "hello"
s2 := si.Intern(s1)

Because the map holds the interned strings, the implementation must clone the input to prevent memory leaks when the input is a substring of a larger buffer.

For deserialization scenarios, the unsafe conversion unsafe.String(bytes, length) can be used to avoid extra allocations.

go4.org/intern Library

The go4.org/intern package provides a ready‑made interning solution with a similar API. It uses a map protected by a mutex, making it safe for concurrent use.

import "go4.org/intern"

s1 := "hello"
s2 := intern.GetByString(s1)

While convenient, the library incurs a lock on each lookup.

unique Package and Interning

The unique package, introduced for low‑cardinality data such as IPv6 address components, offers a generic unique.Handle[T] type and a unique.Make constructor. It stores interned values in a lock‑free hash trie, allowing fast 8‑byte comparisons instead of full string comparisons.

import "unique"

s1 := unique.Make("hello")
s2 := unique.Make("hello")
fmt.Println(s1 == s2) // true

Because the handle is only a pointer‑size value, it reduces memory footprint (e.g., a struct with four string fields drops from 64 bytes to 32 bytes when using unique.Handle[string]).

Performance Comparison

Three parsing functions were benchmarked: a baseline version allocating strings, a version using go4.org/intern, and a version using unique.Make. The benchmark data (shown in the chart) indicate that interning eliminates allocations, and the unique implementation is faster than intern but about 25 % slower than the original allocation‑free version due to hash‑trie lookups.

In production, the memory savings from interning often outweigh the modest speed penalty, especially for long‑running services where GC pressure matters.

Conclusion

String interning is an effective optimization for low‑cardinality data such as repeated field values in deserialization. The unique package provides a lock‑free, generic solution that reduces memory usage and enables fast equality checks, making it the preferred choice in modern Go code when interning is needed.