How Go’s New unique Package Enables Efficient Interning and Memory Savings

The Go 1.23 standard library adds a new unique package aimed at canonicalizing comparable values so that identical values share a single in‑memory copy, saving memory and making equality checks cheap pointer comparisons.

Simple Interning with a Map

A basic interning implementation can be built with a global map[string]string. The Intern function looks up the string in the map; if absent it clones the string, stores it, and returns the pooled copy.

var internPool map[string]string

// Intern returns a string equal to s, possibly sharing storage with a previous call.
func Intern(s string) string {
    pooled, ok := internPool[s]
    if !ok {
        // Clone to avoid retaining a substring of a larger string.
        pooled = strings.Clone(s)
        internPool[pooled] = pooled
    }
    return pooled
}

This approach works for many duplicate strings (e.g., when parsing text) but has three major drawbacks:

Strings are never removed from the pool, leading to unbounded memory growth.

The map is not safe for concurrent goroutine access.

It only handles strings, while the interning idea applies to any comparable type.

Moreover, the implementation misses an optimization: once two strings are interned, their equality can be decided by a single pointer comparison instead of a full content scan.

Design of the unique Package

The package introduces a generic Make function (similar to Intern) that works with any comparable type T. Internally it stores values in a generic concurrent map and returns a Handle[T] wrapper. Handle[T] has two crucial properties:

Two handles are equal only if the underlying values used to create them are equal.

Handle comparison is extremely cheap because it reduces to a pointer comparison, which is far faster than comparing large strings or structs.

Beyond providing cheap equality, a Handle[T] also keeps the canonical value alive: as long as any map holds a handle, the internal map retains the canonical copy. When no handles reference a value, the entry is marked deletable and the garbage collector can reclaim it, giving a clear strategy for removing unused interned items.

Real‑World Example: net/netip Interning

The standard library’s net/netip package uses unique to intern the addrDetail struct inside an Addr. The simplified code looks like this:

type Addr struct {
    // Canonicalized address details.
    z unique.Handle[addrDetail]
}

type addrDetail struct {
    isV6   bool   // false for IPv4, true for IPv6
    zoneV6 string // non‑empty only for IPv6
}

var z6noz = unique.Make(addrDetail{isV6: true})

func (ip Addr) WithZone(zone string) Addr {
    if !ip.Is6() {
        return ip
    }
    if zone == "" {
        ip.z = z6noz
        return ip
    }
    ip.z = unique.Make(addrDetail{isV6: true, zoneV6: zone})
    return ip
}

Because many IP addresses share the same zone string, interning the addrDetail reduces the average memory footprint of each netip.Addr. After interning, zone comparison becomes a simple pointer equality, dramatically speeding up address comparisons.

String Interning Footnote

While unique can intern strings, it differs from traditional string interning because the Handle[T] must be retained to prevent premature removal from the internal map. The article notes that a fully transparent string interning (without an explicit handle) is still a future possibility, akin to the classic Lisp symbol interning.

History, Weak Pointers, and Future Directions

The net/netip package previously used the go4.org/intern package, which relied on unsafe weak pointers. Implementing unique required adding proper weak‑pointer support to the Go runtime, turning weak pointers into a publicly proposed feature.

This work also inspired a replacement for finalizers, leading to a more efficient and easier‑to‑use finalizer alternative. With comparable‑value hash functions on the horizon, the article foresees a future where Go can build high‑performance caches using the interned values.

References

unique package introduction: https://go.dev/blog/unique

Generic concurrent map used internally: https://pkg.go.dev/internal/[email protected]

go4.org/intern (original intern package): https://pkg.go.dev/go4.org/intern

Public proposal for weak pointers: https://go.dev/issue/67552

Finalizer replacement discussion: https://go.dev/issue/67535

Efficient cache outlook: https://go.dev/issue/67552#issuecomment-2200755798