Go encoding/json/v2: 14-Year Evolution from Legacy Flaws to Modern Design

Introduction

Go's encoding/json has been a core standard library since Go 1.0 (2012), but it carried well-known design flaws: case-insensitive field matching, silent tolerance of duplicate keys and invalid UTF-8, and time.Duration serialized as raw nanoseconds. These issues persisted for 14 years due to Go's strict backward compatibility promise (Go 1 Compatibility Promise), which guarantees that programs written to the Go 1 spec continue to compile and run correctly without modification.

Unfixable Problems in v1

The Original Issue

In 2016, an issue (#14750) questioned case-insensitive matching. Russ Cox replied that the behavior was documented since at least Go 1.2 and therefore not a bug. The compatibility promise only allows changes for security vulnerabilities, undefined behavior, spec errors, or obvious bugs — none of which applied.

What the Compatibility Promise Says

Its goal is to ensure programs written to the Go 1 specification continue to compile and run correctly throughout the lifetime of that specification.

Case-insensitive matching was documented, so it was not undefined behavior or a bug. Security concerns were classified as "dangerous defaults" rather than vulnerabilities. Thus the behavior could not be changed without breaking existing programs.

Deferred to Go 2

Similar issues like time.Duration serialization (#4712) were closed with "everything will be reconsidered for Go 2." By 2017, such problems were categorized as unsolvable in current Go.

A Failed Fix Attempt

In 2020, maintainer Daniel Martí (mvdan) wrote a patch (CL 224079) that fixed case-insensitive matching with a 1% decode slowdown. Despite the patch working, the core team rejected it because it would break dependent programs and impose a performance regression on all users. The patch was abandoned in 2024.

v1 vs v2 Behavior Differences

In Go 1.27, both encoding/json (v1) and encoding/json/v2 coexist. Example:

package main
import (
  jsonv1 "encoding/json"
  jsonv2 "encoding/json/v2"
  "fmt"
)

type User struct {
  Name string `json:"name"`
}

func main() {
  in := []byte(`{"NAME":"gopher"}`)
  var a User
  err1 := jsonv1.Unmarshal(in, &a)
  fmt.Printf("v1: %+v  err=%v
", a, err1)
  var b User
  err2 := jsonv2.Unmarshal(in, &b)
  fmt.Printf("v2: %+v  err=%v
", b, err2)
}

v1: {Name:gopher}  err=<nil>
v2: {Name:}  err=<nil>

v2 does not match NAME to name; it ignores unknown members unless RejectUnknownMembers option is set.

Duplicate Keys and Invalid UTF-8

v1 silently accepts duplicate keys (last-wins) and replaces invalid UTF-8 with replacement characters. v2 rejects both at the syntactic layer via encoding/json/jsontext:

dup := []byte(`{"name":"alice","role":"user","role":"admin"}`)
bad := []byte("{\"name\":\"go\xffpher\"}")
// v1 dup: {Name:alice Role:admin}  err=<nil>
// v2 dup: err=jsontext: duplicate object member name "role"
// v1 utf8: err=<nil> -> "gopher"
// v2 utf8: err=jsontext: invalid UTF-8 within "/name" after offset 11

Duplicate keys can cause security mismatches between proxy and backend (discussed in #63397).

Four Categories of v1 Problems

Discussion #63397 classified issues into:

Missing functionality : no way to specify time.Time format, ignore specific values, serialize nil slice/map as [] / {}, no inline tag.

API deficiencies : Decoder.Decode ignores trailing garbage, no config options for Marshal / Unmarshal, Compact / Indent / HTMLEscape only write to *bytes.Buffer.

Performance bottlenecks : MarshalJSON returns []byte causing allocation and re-parse; UnmarshalJSON causes double parsing, especially costly with nested types (Kubernetes OpenAPI case). Streaming APIs still buffer entire values.

Behavioral flaws : tolerate invalid UTF-8 and duplicate keys, case-insensitive matching, MarshalJSON called only for addressable values (slice elements yes, map values no).

The addressability flaw example:

type Tag struct { Name string }
func (t *Tag) MarshalJSON() ([]byte, error) {
  return []byte(`"` + strings.ToUpper(t.Name) + `"`), nil
}

slice := []Tag{{Name: "go"}}        // addressable
m := map[string]Tag{"lang": {Name: "go"}} // not addressable
// v1 slice: ["GO"]
// v1 Map : {"lang":{"Name":"go"}}
// v2 slice: ["GO"]
// v2 Map : {"lang":"GO"}

Even though this is a bug, it couldn't be fixed because too many systems implicitly depended on the inconsistent behavior.

Third-Party Workarounds

High-Performance Libraries

Community libraries emerged: mailru/easyjson (code generation), json-iterator/go (reflection with caching), goccy/go-json (opcode compilation, bitmap for ≤16 fields), bytedance/sonic (JIT machine code, SIMD). All aimed to be drop-in replacements (change one import).

Cost of Drop-In Compatibility

To be seamless replacements, they had to replicate v1's flaws exactly: case-insensitive matching, silent duplicate key handling, nil slice → null. This led to bugs: goccy/go-json#568 shows case-insensitive matching breaks for structs with ≥17 fields because the bitmap optimization path omitted the v1 compatibility logic. The bug is subtle: adding one field flips parsing behavior.

Dangerous Runtime Dependencies

goccy/go-json

used linkname to access internal runtime types. When Go team cleaned internals (#67401), it broke goccy/go-json, and the team feared changing internals because Kubernetes and others depended on it. This inverted the dependency: third-party libraries constrained stdlib evolution.

Why Stdlib Didn't Adopt Them

Discussion #63397 states: external implementations heavily use unsafe, making them unsuitable for stdlib. The 2021 Go Developer Survey showed users prioritized reliability and safety over CPU/memory performance. Joe Tsai's benchmark repo go-json-experiment/jsonbench notes goccy/go-json has data races and memory corruption, unsafe for production.

Origins of v2

2020 Design Draft

After abandoning the patch, mvdan drafted v2 design (Oct 2020) with acknowledgments from Philip Pearl, Matt Layher, Dave Cheney, Chris Hines, Roger Peppe, Joe Tsai. Joe Tsai (Google, Protobuf Go API) contributed first prototype code weeks later.

Core contradictions identified:

Unavoidable memory buffering Marshaler / Unmarshaler cannot pass options MarshalJSON always allocates Decoder.Decode encourages misuse

Design principles: correctness over performance, no unsafe, no code generation. Borrowed ideas from json-iterator/go (allocation reduction), Phil Pearl's Marshaler analysis, Dave Cheney's fast tokenizer.

Key early goal: "If v1 must be kept forever, reimplementing v1 on top of v2 would be hugely beneficial."

Syntax-Layer First Implementation

Repo github.com/go-json-experiment/json started Oct 23, 2020. First commits built syntactic layer ( jsontext predecessor) before semantic layer (Go type mapping). This separation came from Joe Tsai's protojson experience: Protobuf required strict JSON spec compliance and true streaming, which encoding/json couldn't provide, forcing a custom internal JSON implementation. That internal layer became the inspiration for jsontext.

Six Goals in README

95-99% backward compatibility, not 100%, to allow correcting recognized mistakes.

Reimplement v1 on v2 foundation.

README also listed five possible outcomes, first being project abandonment.

Production Validation at Tailscale

Joe Tsai joined Tailscale (July 2021), introduced experimental module into production (Oct 2022). Used in core backend services (not open source). Validated correctness and performance. Warning: experimental module should not be used in public libraries to avoid diamond dependency conflicts.

From Discussion to Formal Proposal

math/rand/v2 Precedent

Discussion #63397 opened Oct 5, 2023, two days after math/rand/v2 proposal (#61716) was approved. Russ Cox's earlier discussion #60751 established stdlib's first v2 pattern.

Core Controversies

Four main debates:

Map output order : v1 stable sorted; v2 non-deterministic. Decision: default no sort, optional Deterministic option.

nil slice/map : v1 → null; v2 → [] / {}. Opponents argued breaks round-tripping and loses nil semantics.

omitempty criteria : v1 uses Go zero value; v2 uses JSON empty value. Shift from Go type system to JSON type system.

null into non-nullable Go type : both v1 and v2 accept without error. Proposal to reject was declined: shouldn't reject valid JSON to accommodate Go's static types.

Permanent v1 Support

Official docs: "All new 'json' use cases should use v2, but v1 will receive official support forever." Joe Tsai's GopherCon 2023 talk "The Future of JSON in Go" same week.

Formal Proposal and Feature Freeze

Proposal #71497 (Jan 31, 2025) covered both encoding/json/v2 and encoding/json/jsontext. Naming changes: MarshalWriter / MarshalNext → MarshalWrite / MarshalEncode; MarshalerV2 / UnmarshalerV2 → MarshalerTo / UnmarshalerFrom. Damien Neil (Apr 10, 2026) emphasized avoiding feature creep; willing to cut features to ship.

GOEXPERIMENT=jsonv2 Public Preview

Go 1.25 (Aug 2025) included v2 behind GOEXPERIMENT=jsonv2. Go blog post Sep 9, 2025 highlighted community-driven development. Working group formed Nov 20, 2025 with public meeting notes.

Resolving time.Duration

Sub-proposal #71631. v1 serializes to nanosecond integer (no unit, loses JS float64 precision after 104 days). Options: keep nanosecond, switch to ISO 8601, or require explicit format. Decision: require explicit format. Marshal on raw time.Duration errors unless format specified. Neither bad default nor silent change.

Proposal Approved

Apr 16, 2026: Active. Apr 29: one-week deadline. Apr 30: Joe Tsai updated proposal targeting Go 1.27 and withdrew format tag (see below). May 6: final review meeting. May 13: approved. May 22: milestone locked to Go 1.27. Jun 9: implemented and closed. 10 years 2 months from first issue.

How v1 Is Built on v2

Reimplementing v1 Atop v2

Go 1.27 release notes: "encoding/json package now fully backed by v2 core implementation. Original marshal/unmarshal behavior preserved, but error message text may differ slightly." The 2020 README vision realized.

13 Legacy Behavior Flags

v1 compatibility achieved via DefaultOptionsV1 injecting 13 implicit flags (from internal/jsonflags/flags.go):

const (
  _Bools = (maxArshalV2Flag >> 1) << iota
  CallMethodsWithLegacySemantics        // marshal or unmarshal
  FormatByteArrayAsArray                // marshal or unmarshal
  FormatBytesWithLegacySemantics        // marshal or unmarshal
  FormatDurationAsNano                  // marshal or unmarshal
  MatchCaseSensitiveDelimiter           // marshal or unmarshal
  MergeWithLegacySemantics              // unmarshal
  OmitEmptyWithLegacySemantics          // marshal
  ParseBytesWithLooseRFC4648            // unmarshal
  ParseTimeWithLooseRFC3339             // unmarshal
  ReportErrorsWithLegacySemantics       // marshal or unmarshal
  StringifyWithLegacySemantics          // marshal or unmarshal
  UnmarshalAnyWithRawNumber             // unmarshal; for jsonv1.Decoder.UseNumber
  UnmarshalArrayFromAnyLength           // unmarshal
  maxArshalV1Flag
)

Six flags carry WithLegacySemantics suffix, explicitly marking them as historical baggage. FormatDurationAsNano is the 2017 Russ Cox "can't change" core detail. ParseTimeWithLooseRFC3339 and ParseBytesWithLooseRFC4648 reflect historically lax RFC adherence. UnmarshalArrayFromAnyLength allows 3-element JSON array into 5-element Go array.

Fine-Grained Control with DefaultOptionsV1

Example demonstrates mixing v1 baseline with selective v2 overrides:

jsonv2.Marshal(v, jsonv1.DefaultOptionsV1())                    // exact v1 behavior
jsonv2.Marshal(v, jsonv1.DefaultOptionsV1(), jsontext.AllowDuplicateNames(false)) // v1 but reject duplicate keys
jsonv2.Marshal(v, jsonv1.CallMethodsWithLegacySemantics(true)) // v2 but keep v1 addressability quirk
jsonv2.Marshal(v)                                               // pure v2

This turns the binary choice (all v1 flaws or breaking change) into a per-call knob.

Practical Verification

Two Implementations Coexisting in Source

Go 1.27 source tree shows encode.go (old, build tag !goexperiment.jsonv2) and v2_encode.go (new, build tag goexperiment.jsonv2). GOEXPERIMENT=jsonv2 default in 1.27; opt-out via GOEXPERIMENT=nojsonv2 falls back to 2010 code (to be removed later). Copyright headers show lineage: 2010 (v1), 2020 (v2 semantic), 2023 ( jsontext).

Deep Comparison with nojsonv2

Testing edge cases (empty string value, truncated input, trailing garbage, array into struct, unsupported chan type) shows byte-for-byte identical error messages. v2_inject.go bridges v2 to v1's custom error types like *MarshalerError.

Intentional Error Message Randomization

v2 randomly alternates between "cannot" and "unable to" in error strings (once per process) to prevent developers from relying on exact error text (Hyrum's Law). Implementation uses Go's random map iteration order:

var errorModalVerb = sync.OnceValue(func() string {
  for phrase := range map[string]struct{}{"cannot": {}, "unable to": {}} {
    return phrase
  }
  return ""
})

This mirrors Go's earlier map iteration randomization to break ordering dependencies.

Measured Performance

Official: Marshal similar, Unmarshal significantly faster. Blog claimed up to 10x Unmarshal speedup. Author's benchmarks (Go 1.27, darwin/arm64, Apple M5 Pro):

BenchmarkUnmarshalV1-18   2295  523371 ns/op  121.66 MB/s  242924 B/op  4012 allocs/op
BenchmarkUnmarshalV2-18   2716  446989 ns/op  142.44 MB/s  242924 B/op  4012 allocs/op
BenchmarkMarshalV1-18     5594  213990 ns/op            66148 B/op  3 allocs/op
BenchmarkMarshalV2-18     5516  217250 ns/op            66467 B/op  3 allocs/op

Unmarshal only 1.18x faster; Marshal ~same. The 10x claim comes from unmarshaling into any:

BenchmarkAnyV1-18    921  1306860 ns/op  48.72 MB/s  739932 B/op  23014 allocs/op
BenchmarkAnyV2-18   2091   581713 ns/op 109.45 MB/s  626968 B/op  17012 allocs/op

2.25x faster, fewer allocations. But v1's Unmarshal into any is slower than v2's native API because v1 forces AllowDuplicateNames flag (v1 default), which disables v2's optimized fast path for any (see arshal_default.go line 1904). Enabling AllowDuplicateNames on v2 drops performance to v1 levels:

BenchmarkAnyV2AllowDup-18  994  1219219 ns/op  52.22 MB/s  739925 B/op  23014 allocs/op

Thus, to get full speed, must call v2 API directly.

16 Major Behavioral Differences

Official migration guide lists 16 divergences (v1 → v2):

Field matching: case-insensitive → case-sensitive omitempty: Go zero value → JSON empty value string tag: works on string/bool/number, no recursion → only number, supports recursion

nil slice/map: null → [] / {} Go array length: any length → must match exactly [N]byte: number array → Base64 string

Pointer receiver methods: only if addressable → always

Map key custom methods: never → always

Map key order: sorted deterministic → non-deterministic

HTML/JS escaping: forced → minimal required

Invalid UTF-8: silent replace → error

Duplicate keys: silent last-wins → error

null into non-zero: sometimes zero, sometimes ignore → always zero

Merge logic: fragmented/inconsistent → deep merge objects, else replace time.Duration: nanosecond integer → no default, must specify format

Structurally invalid Go types: may pass compile, no runtime error → immediate runtime error

Many v1 behaviors described as "inconsistent" or "sometimes... sometimes not."

Post-v2 Ecosystem

Withdrawn format Tag

format

tag (e.g., format:RFC1123, format:iso8601, format:units for time.Duration) was removed days before release because Go 1.28 is expected to introduce "Typed struct tags" (#74472) which would express formatting more naturally (e.g., time.Time {json.Format(time.RFC3339)}). Workgroup prioritized shipping over feature creep (Damien Neil's warning).

time.Duration Deadlock

Without format tag, time.Duration cannot be serialized natively in v2:

type Config struct { Timeout time.Duration `json:"timeout"` }
type ConfigTagged struct { Timeout time.Duration `json:"timeout,format:units"` }
// v1: {"timeout":90000000000} err=<nil>
// v2: err=json: cannot marshal from Go time.Duration ... no default representation
// v2 format:units: err=json: cannot marshal ... unsupported `format` tag option

Workaround: implement custom Marshaler or use external github.com/go-json-experiment/json which still exposes the internal ExperimentalSupportFormatTag switch. The implementation code remains in stdlib but locked behind internal package.

Third-Party Library Reactions

bytedance/sonic

: massive 2500+ line PR, compatibility matrix doc ( docs/sonic-go127-compatibility.md), CI runs both default and nojsonv2 modes. Differences noted: overflow handling ( +Inf vs 0), map[float64]string support, AppendText priority. goccy/go-json: no visible response, issues/roadmap stagnant. json-iterator/go: archived (May 2024). Its "100% compatible" claim coexisted with open issue "Not 100% compatible" for years.

protojson and jsontext

protojson

(which inspired jsontext) hasn't adopted jsontext yet due to Go Protobuf's policy of supporting multiple old Go versions. Issue golang/protobuf#1673 still open. Joe Tsai's envisioned API ( MarshalWrite, MarshalEncode, etc.) would allow seamless delegation. Michael Stapelberg (protobuf maintainer) acknowledged v2 release as validation of their wait strategy, but migration remains pending.

Why the v2 Path?

Three Design Principles (Russ Cox)

Semantic Import Versioning : math/rand and math/rand/v2 are independent packages, can coexist. Foundation for stdlib v2 packages.

Absolute Respect for Users : Changes must have undeniable justification, worth migration cost. Old package is the reference point; evolution fixes fatal flaws, not aesthetic preferences.

Help Users Stuck in v1 : Ideally v2 wraps v1 (or vice versa) so legacy users get bug fixes and performance without code changes. Russ Cox admitted this is rarely fully achievable. encoding/json/v2 not only met the third principle but inverted it: v1 became a thin wrapper on v2. All existing v1 callers automatically run on modern engine; future fixes land in v2 and benefit both. Stdlib maintains single parsing implementation.

Russ Cox warned: no flood of v2 packages; each handled with extreme caution. math/rand/v2 (Go 1.22) and encoding/json/v2 (Go 1.27) spaced five major versions apart.

Conclusion

From a 2016 issue questioning case-insensitive matching to v2 approval in 2026: 10 years 2 months. From Go 1.0: 14 years. The process: write patch, measure cost, abandon; draft design, build prototype, dogfood in production; public discussion, decompose into sub-proposals; experimental flag, year of public testing; flip default, keep escape hatch. Result: zero forced breaking changes for existing code.

Compatibility transformed from zero-sum game to a per-call knob via DefaultOptionsV1() plus selective overrides. Developers now choose their contract per invocation.

Imperfections remain: format tag removed awaiting typed struct tags, leaving time.Duration unserializable natively; GOEXPERIMENT=nojsonv2 escape hatch will eventually disappear, removing 2010 encode.go. The 13 legacy flags will remain as permanent record of 14 years of compromises and decisions. encoding/json/v2 has launched; the next chapter of its ecosystem awaits.