Why SpringBoot Admin Shows Services Offline and How to Fix Sentinel Health

SpringBoot Admin can be regarded as a UI for monitoring Spring Actuator endpoints, allowing easy inspection of service runtime status.

Some Services Appear Offline

After integrating certain components, the status shows offline even though the application continues to provide services.

The displayed online status is read from the Spring actuator health endpoint data.

curl http://127.0.1.1:5002/actuator/health

{"status":"DOWN"}

At this point SpringBoot Admin cannot retrieve more details; it is recommended to enable detailed endpoint information in the development environment.

management:
  endpoints:
    web:
      exposure:
        include: '*'
  endpoint:
    health:
      show-details: always  #dev open

Retrying the request shows the status of all components, revealing that the sentinel component is down, which affects the overall status.

curl http://127.0.1.1:5002/actuator/health

{"status":"DOWN","components":{"discoveryComposite":{"status":"UP","components":{"discoveryClient":{"status":"UP","details":{"services":["pigx-upms-biz","pigx-tx-manager","pigx-daemon-elastic-job","pigx-mp-platform","pigx-daemon-quartz","pigx-gateway","pigx-auth","pigx-pay-platform","pigx-codegen","pigx-oa-platform","pigx-monitor"]}}}},"diskSpace":{"status":"UP","details":{"total":42927636480,"free":21334122496,"threshold":10485760}},"nacosConfig":{"status":"UP"},"nacosDiscovery":{"status":"UP"},"ping":{"status":"UP"},"reactiveDiscoveryClients":{"status":"UP","components":{"Simple Reactive Discovery Client":{"status":"UP","details":{"services":[]}}}},"redis":{"status":"UP","details":{"version":"5.0.7"}},"refreshScope":{"status":"UP"},"sentinel":{"status":"DOWN","details":{"dataSource":{},"enabled":true,"dashboard":{"description":"pigx-sentinel:5020 can't be connected","status":"DOWN"}}}}

Sentinel Dashboard Can't Be Connected

Inspect SentinelHealthIndicator.doHealthCheck to see how Sentinel health is evaluated.

protected void doHealthCheck(Health.Builder builder) throws Exception {
    Map<String, Object> detailMap = new HashMap<>();

    // Get heartbeat sender
    HeartbeatSender heartbeatSender = HeartbeatSenderProvider.getHeartbeatSender();
    // Send heartbeat
    boolean result = heartbeatSender.sendHeartbeat();
    if (result) {
        // Success marked as UP
        detailMap.put("dashboard", Status.UP);
    } else {
        // Failure marked as DOWN
        dashboardUp = false;
        detailMap.put("dashboard", new Status(Status.DOWN.getCode(),
                consoleServer + " can't be connected"));
    }

    // If Dashboard and DataSource are both OK, the health status is UP
    if (dashboardUp && dataSourceUp) {
        builder.up().withDetails(detailMap);
    } else {
        builder.down().withDetails(detailMap);
    }
}

The SimpleHttpHeartbeatSender.sendHeartbeat handles the heartbeat transmission.

public boolean sendHeartbeat() throws Exception {
    // 1. Determine if client and server have interacted; only then dashboard shows data
    if (TransportConfig.getRuntimePort() <= 0) {
        RecordLog.info("[SimpleHttpHeartbeatSender] Runtime port not initialized, won't send heartbeat");
        return false;
    }
    InetSocketAddress addr = getAvailableAddress();
    if (addr == null) {
        return false;
    }

    // 2. Call dashboard API
    SimpleHttpRequest request = new SimpleHttpRequest(addr, TransportConfig.getHeartbeatApiPath());
    request.setParams(heartBeat.generateCurrentMessage());
    SimpleHttpResponse response = httpClient.post(request);
    if (response.getStatusCode() == OK_STATUS) {
        return true;
    }
    return false;
}

Which step causes the service to be marked down? The logs show the first step (runtime port not initialized) leads to a down status.

Check the Sentinel client logs:

cat ~/csp/sentinel-record.log

2020-03-23 13:54:04.634 INFO [SimpleHttpHeartbeatSender] Runtime port not initialized, won't send heartbeat

The initial lack of interaction causes the service to be considered down.

How Sentinel Initializes Interaction

Initialize the interaction port with commandCenter.start().

public class CommandCenterInitFunc implements InitFunc {

    @Override
    public void init() throws Exception {
        CommandCenter commandCenter = CommandCenterProvider.getCommandCenter();

        if (commandCenter == null) {
            RecordLog.warn("[CommandCenterInitFunc] Cannot resolve CommandCenter");
            return;
        }

        commandCenter.beforeStart();
        // Start initializing port interaction
        commandCenter.start();
        RecordLog.info("[CommandCenterInit] Starting command center: "
                + commandCenter.getClass().getCanonicalName());
    }
}

Configure eager initialization so the interaction is set up at startup rather than on first traffic.

public class SentinelAutoConfiguration {

    @Value("${project.name:${spring.application.name:}}")
    private String projectName;

    @Autowired
    private SentinelProperties properties;

    @PostConstruct
    private void init() {
        // earlier initialize
        if (properties.isEager()) {
            InitExecutor.doInit();
        }
    }
}

spring:
  application:
    name: @artifactId@
  cloud:
    sentinel:
      eager: true