Operational Pitfalls of Using S3 Website Hosting and CDN Query‑String Caching

Preface

In the fast‑moving world of online services, even seasoned operators encounter unexpected failures; this article reviews several real‑world incidents and the underlying causes.

1. Hidden Limitations of S3 Website Hosting

S3’s static‑website feature integrates nicely with CloudFront, but it was designed primarily as a highly‑available object store, leading to three major constraints:

1. No Server‑Side Include (SSI) support

Modern static sites often rely on SSI, which S3 cannot provide.

2. UTF‑8‑only filenames

Files with multibyte encodings must be UTF‑8; otherwise uploads fail.

3. No Multi‑Range support

Concurrent segmented downloads that use multiple Range headers are either ignored or only the first range is honored, breaking many download accelerators.

Because of these issues, the team migrated the overseas game website away from S3 back to traditional web servers, retaining S3 only for download‑centric resources where strict controls can mitigate the risks.

2. QueryString (QS) – The Hidden Firestarter

QS parameters are used for two main static‑site scenarios:

Version control at the CDN cache layer.

Page customization at the application layer.

CDN caching policies for QS typically fall into two categories:

Completely ignore the query string (the simplest and most common default).

Ignore only a subset of parameters.

When the chosen caching policy does not match the intended QS usage, cache‑penetration occurs, overloading the origin and, in extreme cases, causing near‑total cache failure.

Guidelines:

If the query string carries meaningless random tokens, configure the CDN to ignore it.

If the query string conveys meaningful data (e.g., version numbers, user IDs), the CDN should cache with the QS preserved, unless business requirements dictate otherwise.

Even when the CDN strips QS for caching, the original request still carries the parameters to the origin, so downstream services can handle them.

Note: Some CDN vendors also drop the QS when forwarding to the origin, which can break redirects and other logic.

3. Protocol Compatibility Chaos

1. Divergent HTTP implementations

Tests comparing Nginx and Apache on static‑file requests revealed differing status‑code behaviours, especially for HEAD + Range requests. AWS returns 200 for HEAD + Range, interpreting HEAD as a metadata request, while the RFC expects a 206 when a valid Range is present.

2. Real‑world impact of protocol differences

During a gray‑release, the game’s media service switched its origin from Apache to Nginx. Nginx began returning 412 responses for malformed If‑Match headers sent by some clients, whereas Apache silently ignored them. The fix was to drop the offending If‑Match header.

These discrepancies illustrate why a continuously updated checklist and rigorous acceptance criteria are essential for stable operations.

Conclusion

CDN‑related pitfalls extend far beyond the examples above, including 3XX redirects, Range anomalies, cache‑stampede, and client‑side hijacking. The team now employs a detect‑plus‑fallback mechanism to mitigate most issues, but full‑coverage remains elusive, requiring ongoing collaboration with CDN providers.