Why My Spring Cloud Gateway Returns 404 After Nacos Config Changes – A Year-Long Debugging Tale

Problem Description

Using SpringCloud Gateway 3.0.1 with JDK 8 and Nacos for dynamic routing, the API returns a 404 error each time the Nacos route configuration is modified, but the error disappears after restarting the gateway.

Environment Preparation

Three backend services are started on ports 8103, 12040, and 12041. In Nacos the following routes are defined and placed in the same weight group xiaofu-group to enable weight‑based load balancing:

- id: xiaofu-8103
  uri: http://127.0.0.1:8103/
  predicates:
    - Weight=xiaofu-group, 2
    - Path=/test/version1/**
  filters:
    - RewritePath=/test/version1/(?<segment>.*),/${segment}
- id: xiaofu-12040
  uri: http://127.0.0.1:12040/
  predicates:
    - Weight=xiaofu-group, 1
    - Path=/test/version1/**
  filters:
    - RewritePath=/test/version1/(?<segment>.*),/${segment}
- id: xiaofu-12041
  uri: http://127.0.0.1:12041/
  predicates:
    - Weight=xiaofu-group, 2
    - Path=/test/version1/**
  filters:
    - RewritePath=/test/version1/(?<segment>.*),/${segment}

Issue Investigation

Continuous requests are generated with JMeter. Logs show that all three instances receive traffic, confirming that load balancing works initially. After manually removing the instance xiaofu-12041, JMeter starts returning 404 errors and the gateway console still shows the deleted route being chosen, indicating that stale routing data is used.

The root cause is identified as the gateway’s cache: although Nacos removes the route, the WeightCalculatorWebFilter continues to use the old weight data stored in its internal groupWeights map.

Source Code Analysis

The filter maintains a groupWeights variable. When a configuration change event occurs, the method addWeightConfig(WeightConfig weightConfig) is invoked:

@Override
public void onApplicationEvent(ApplicationEvent event) {
    if (event instanceof PredicateArgsEvent) {
        handle((PredicateArgsEvent) event);
    } else if (event instanceof WeightDefinedEvent) {
        addWeightConfig(((WeightDefinedEvent) event).getWeightConfig());
    } else if (event instanceof RefreshRoutesEvent && routeLocator != null) {
        if (routeLocatorInitialized.compareAndSet(false, true)) {
            routeLocator.ifAvailable(locator -> locator.getRoutes().blockLast());
        } else {
            routeLocator.ifAvailable(locator -> locator.getRoutes().subscribe());
        }
    }
}

The comment of addWeightConfig states that it only creates a new GroupWeightConfig and does not modify existing entries, meaning removed routes are never cleared from the cache.

void addWeightConfig(WeightConfig weightConfig) {
    String group = weightConfig.getGroup();
    GroupWeightConfig config;
    // only create new GroupWeightConfig rather than modify
    if (groupWeights.containsKey(group)) {
        config = new GroupWeightConfig(groupWeights.get(group));
    } else {
        config = new GroupWeightConfig(group);
    }
    // ... calculations ...
    groupWeights.put(group, config);
}

Solution

Since upgrading to SpringCloud Gateway 4.1.0 does not resolve the issue, a custom listener is added to synchronize the cache with the latest Nacos configuration.

@Slf4j
@Configuration
public class WeightCacheRefresher {
    @Autowired
    private WeightCalculatorWebFilter weightCalculatorWebFilter;
    @Autowired
    private RouteDefinitionLocator routeDefinitionLocator;
    @Autowired
    private ApplicationEventPublisher publisher;

    /** Listen to route refresh events and update weight cache */
    @EventListener(RefreshRoutesEvent.class)
    public void onRefreshRoutes() {
        log.info("Detected route refresh event, syncing weight cache");
        syncWeightCache();
    }

    /** Sync weight cache with current route definitions */
    public void syncWeightCache() {
        try {
            Field groupWeightsField = WeightCalculatorWebFilter.class.getDeclaredField("groupWeights");
            groupWeightsField.setAccessible(true);
            @SuppressWarnings("unchecked")
            Map<String, Object> groupWeights = (Map<String, Object>) groupWeightsField.get(weightCalculatorWebFilter);
            if (groupWeights == null) {
                log.warn("Weight cache not found");
                return;
            }
            log.info("Current weight cache: {}", groupWeights.keySet());

            Set<String> currentRouteIds = new HashSet<>();
            Map<String, Map<String, Integer>> currentGroupRouteWeights = new HashMap<>();
            routeDefinitionLocator.getRouteDefinitions()
                .collectList()
                .subscribe(definitions -> {
                    definitions.forEach(def -> {
                        currentRouteIds.add(def.getId());
                        def.getPredicates().stream()
                            .filter(p -> p.getName().equals("Weight"))
                            .forEach(p -> {
                                Map<String, String> args = p.getArgs();
                                String group = args.getOrDefault("_genkey_0", "unknown");
                                int weight = Integer.parseInt(args.getOrDefault("_genkey_1", "0"));
                                currentGroupRouteWeights.computeIfAbsent(group, k -> new HashMap<>())
                                    .put(def.getId(), weight);
                            });
                    });
                    log.info("Current route IDs: {}", currentRouteIds);
                    log.info("Current weight groups: {}", currentGroupRouteWeights);

                    Set<String> groupsToRemove = new HashSet<>();
                    Set<String> groupsToUpdate = new HashSet<>();
                    for (String group : groupWeights.keySet()) {
                        if (!currentGroupRouteWeights.containsKey(group)) {
                            groupsToRemove.add(group);
                            log.info("Weight group [{}] no longer exists, will be removed", group);
                            continue;
                        }
                        Map<String, Integer> configuredRouteWeights = currentGroupRouteWeights.get(group);
                        Object groupWeightConfig = groupWeights.get(group);
                        try {
                            Field weightsField = groupWeightConfig.getClass().getDeclaredField("weights");
                            weightsField.setAccessible(true);
                            @SuppressWarnings("unchecked")
                            LinkedHashMap<String, Integer> weights = (LinkedHashMap<String, Integer>) weightsField.get(groupWeightConfig);
                            Set<String> routesToRemove = weights.keySet().stream()
                                .filter(id -> !configuredRouteWeights.containsKey(id))
                                .collect(Collectors.toSet());
                            Set<String> routesWithWeightChange = new HashSet<>();
                            for (Map.Entry<String, Integer> entry : weights.entrySet()) {
                                String routeId = entry.getKey();
                                Integer cachedWeight = entry.getValue();
                                if (configuredRouteWeights.containsKey(routeId)) {
                                    Integer configuredWeight = configuredRouteWeights.get(routeId);
                                    if (!cachedWeight.equals(configuredWeight)) {
                                        routesWithWeightChange.add(routeId);
                                        log.info("Route [{}] weight changed from {} to {}", routeId, cachedWeight, configuredWeight);
                                    }
                                }
                            }
                            Set<String> newRoutes = configuredRouteWeights.keySet().stream()
                                .filter(id -> !weights.containsKey(id))
                                .collect(Collectors.toSet());
                            if (!routesToRemove.isEmpty() || !routesWithWeightChange.isEmpty() || !newRoutes.isEmpty()) {
                                log.info("Weight group [{}] changes: remove {}, weight change {}, add {}", group, routesToRemove, routesWithWeightChange, newRoutes);
                                groupsToUpdate.add(group);
                            }
                            // apply removals
                            for (String routeId : routesToRemove) {
                                weights.remove(routeId);
                            }
                            if (weights.isEmpty()) {
                                groupsToRemove.add(group);
                                log.info("Weight group [{}] empty after removals, will be removed", group);
                            }
                        } catch (Exception e) {
                            log.error("Error processing weight group [{}]", group, e);
                        }
                    }
                    // remove stale groups
                    for (String group : groupsToRemove) {
                        groupWeights.remove(group);
                        log.info("Removed weight group: {}", group);
                    }
                    // recompute updated groups
                    for (String group : groupsToUpdate) {
                        try {
                            Map<String, Integer> configuredRouteWeights = currentGroupRouteWeights.get(group);
                            groupWeights.remove(group);
                            log.info("Removed weight group [{}] for recomputation", group);
                            Method addWeightConfigMethod = WeightCalculatorWebFilter.class.getDeclaredMethod("addWeightConfig", WeightConfig.class);
                            addWeightConfigMethod.setAccessible(true);
                            for (Map.Entry<String, Integer> entry : configuredRouteWeights.entrySet()) {
                                String routeId = entry.getKey();
                                Integer weight = entry.getValue();
                                WeightConfig weightConfig = new WeightConfig(routeId);
                                weightConfig.setGroup(group);
                                weightConfig.setWeight(weight);
                                addWeightConfigMethod.invoke(weightCalculatorWebFilter, weightConfig);
                                log.info("Added weight config for route [{}] group [{}] weight {}", routeId, group, weight);
                            }
                        } catch (Exception e) {
                            log.error("Error recomputing weight group [{}]", group, e);
                        }
                    }
                    log.info("Weight cache sync completed, groups: {}", groupWeights.keySet());
                });
        } catch (Exception e) {
            log.error("Failed to sync weight cache", e);
        }
    }
}

After deploying this listener, each Nacos route change triggers a cache refresh, eliminating the 404 error. No official fix was found, suggesting the issue is a bug in the framework’s weight‑caching logic.