DNS Resolution Failure for goofy.app in Singapore Office Caused by DNSSEC Misconfiguration

Cause

Last month a colleague reported that the internal service domain goofy.app and its subdomains were inaccessible from the Singapore office, while they worked fine from the VPN and other China offices.

Investigation

Testing https://foo.goofy.app/ from Singapore resulted in a DNS resolution error:

$> curl https://foo.goofy.app/
curl: (6) Could not resolve host: foo.goofy.app

Accessing the service by IP with a Host header succeeded, confirming that the network path was fine but DNS resolution failed.

✅ Accessible from China office network

✅ Accessible via Singapore office VPN

❌ Not accessible from Singapore office network

✅ Network layer reachable, DNS lookup fails

❌ Public DNS in Singapore fails

The problem clearly pointed to DNS.

Hypothesis 0: Incorrect internal DNS configuration

The domain is actually a public domain that points to internal IPs, so it should resolve publicly. An online DNS test (ping.sx) showed that the A record resolves globally but only from China.

Hypothesis 1: GeoDNS

GeoDNS could restrict resolution to China, but the DNS team confirmed no such setting. Using dig with +subnet also returned no result for a Chinese IP range.

Hypothesis 2: Cross‑border link issue

Trace and NS queries showed that the .app TLD’s name servers are outside China, so the issue was not a cross‑border network problem.

Hypothesis 3: .app TLD NS failure

Testing another .app domain ( get.app ) resolved correctly, indicating the TLD itself was not broken.

Hypothesis 4: DNSSEC misconfiguration

Using Google DNS’s web interface revealed a DNSSEC validation failure for goofy.app . Disabling DNSSEC validation allowed the domain to resolve.

Further inspection of the DNSSEC chain showed an erroneous DS record (a wrong SHA‑256 key) breaking the trust chain.

Resolution

The DNS operations team removed the stray SHA‑256 key from the DS record in the DNS management console. After the removal, DNSSEC validation succeeded and the domain resolved globally.

All systems are now green. 🎉

Post‑mortem Analysis

What happened?

A misconfigured DNSSEC DS record caused most global DNS resolvers to discard the answer, resulting in resolution failure.

Why did it work in China but not abroad?

Most Chinese DNS resolvers have DNSSEC validation disabled, so they ignored the bad DS record.

Overseas resolvers typically enable DNSSEC, so they rejected the response.

Where was the error?

The DS record contained an incorrect hashed KSK, likely carried over from a previous DNS provider migration. Deleting the DS record broke the DNSSEC chain, effectively disabling DNSSEC for the domain.

Could the investigation have been faster?

Running a test on https://dns.google at the start would have shown the DNSSEC failure immediately.

The initial dig output already contained a SERVFAIL , which indicates DNSSEC validation failure.

Alternative mitigation ideas

Editing the local hosts file – not scalable for many subdomains.

Deploying a pure internal DNS for .app – costly due to the new TLD.

Properly configuring DNSSEC – feasible but would still depend on cross‑border DNS reliability.

Further Reading: DNSSEC Mechanics

How DNSSEC Works

DNSSEC Principles and Analysis

Protecting Website Access – Alibaba Cloud DNS Supports DNSSEC