Go GOPROXY Fallback Pitfall: Comma Separators Don't Handle 500 Errors—Use Pipe Instead

Background: CI Lint Failure Due to Corrupted Module

While building a Docker image locally with make image, the author encountered a zip: not a valid zip file error for github.com/klauspost/compress/@v/v1.18.5.zip from the configured GOPROXY mirror. The local make build and make lint had passed, so the issue appeared only in CI. The CI lint step failed with type-check errors: e.Interval undefined and m.Interval undefined in internal/sensor/builtin/echo_sensor.go and internal/sensor/metric_engine.go, caused by the corrupted module download.

Common Fallback Configuration and Its Limitation

The standard practice is to configure multiple GOPROXY mirrors separated by commas, e.g., GOPROXY=https://goproxy.proxy2,https://goproxy.proxy1,direct. The author applied this, expecting that if proxy2 failed, proxy1 would be tried, then direct. However, CI still failed with multiple 500 Internal Server Error responses from proxy2. The fallback did not trigger because the comma separator only triggers fallback on HTTP 404 or 410 errors.

Root Cause: Comma Separator Only Falls Back on 404/410

Claude Code analysis revealed the key distinction: with comma separators, go mod download falls back only on 404/410; with pipe separators ( |), it falls back on any error. Changing the configuration to GOPROXY=https://goproxy.proxy2|https://goproxy.proxy1|direct resolved the issue permanently.

Source Code Analysis: proxyList and TryProxies

The author examined Go 1.26.0 source code in cmd/go/internal/modfetch/proxy.go.

proxyList function

func proxyList() ([]proxySpec, error) {
    proxyOnce.Do(func() {
        ...
        goproxy := cfg.GOPROXY
        for goproxy != "" {
            var url string
            fallBackOnError := false // default false
            if i := strings.IndexAny(goproxy, ",|"); i >= 0 {
                url = goproxy[:i]
                fallBackOnError = goproxy[i] == '|' // true only for pipe
                goproxy = goproxy[i+1:]
            } else {
                url = goproxy
                goproxy = ""
            }
            ...
            proxyOnce.list = append(proxyOnce.list, proxySpec{
                url:            url,
                fallBackOnError: fallBackOnError,
            })
        }
        ...
    })
    return proxyOnce.list, proxyOnce.err
}

The fallBackOnError field in proxySpec is set to true only when the separator is |.

proxySpec struct

type proxySpec struct {
    url            string
    fallBackOnError bool // if true, any error triggers fallback; if false, only os.ErrNotFound (404/410)
}

TryProxies function

func TryProxies(f func(proxy string) error) error {
    proxies, err := proxyList()
    ...
    for _, proxy := range proxies {
        err := f(proxy.url) // execute HTTP request
        if err == nil {
            return nil // success
        }
        isNotExistErr := errors.Is(err, fs.ErrNotExist)
        ...
        // decide whether to fall back
        if !proxy.fallBackOnError && !isNotExistErr {
            break // stop trying further proxies
        }
    }
    return bestErr
}

The loop breaks unless fallBackOnError is true or the error is fs.ErrNotExist (404/410).

HTTPError.Is Method Confirms 404/410 Only

In cmd/go/internal/web/api.go, the HTTPError.Is method returns true only for status codes 404 or 410:

func (e *HTTPError) Is(target error) bool {
    return target == fs.ErrNotExist && (e.StatusCode == 404 || e.StatusCode == 410)
}

Thus, a 500 error is not considered fs.ErrNotExist, so with comma separators ( fallBackOnError=false) the loop breaks and no fallback occurs.

Solution: Use Pipe Separator for Fallback on Any Error

Setting GOPROXY with pipe separators (e.g., https://goproxy.proxy2|https://goproxy.proxy1|direct) sets fallBackOnError=true for each proxy except the last, causing go mod download to retry the next proxy on any error. The author verified this fixes the CI failures consistently.

Verification Demo

A reproducible demo is provided at

https://github.com/jianghushinian/blog-go-example/tree/main/multi-goproxy

. Readers can clone and run it to observe the different fallback behaviors for comma vs. pipe separators.

Go Module Proxy Protocol Endpoints

A Go module proxy implements the following core endpoints: /<module>/@v/list – lists all known versions (plain text, one per line) /<module>/@v/<version>.info – version metadata (JSON: Version, Time) /<module>/@v/<version>.mod – go.mod file content (text) /<module>/@v/<version>.zip – module source zip (binary) /<module>/@latest – latest version info (JSON: Version, Time)

Example:

https://goproxy.cn/github.com/klauspost/compress/@v/v1.18.5.info

returns JSON metadata for that version.

Conclusion

The comma-separated GOPROXY fallback chain only handles 404/410 errors. For unstable proxies that return 5xx or other errors, the pipe separator must be used to enable fallback on any error. This behavior is documented in the Go module reference ( https://go.dev/ref/mod#goproxy-protocol) but rarely mentioned in community articles.

References

Go Modules Reference: https://go.dev/ref/mod#goproxy-protocol Goproxy.cn: https://goproxy.cn/ Goproxy.io: https://goproxy.io/ Go Proxy source code:

https://github.com/golang/go/tree/go1.26.0/src/cmd/go/internal/modfetch

Demo repository:

https://github.com/jianghushinian/blog-go-example/tree/main/multi-goproxy