OpenFeign Timeout Retries Cause Duplicate Orders: Layered Retry Analysis & Idempotency Fix

Problem: Duplicate Orders from Hidden Retries

A single channel order produced three database records with different order_no but identical biz_no, created at 14:08:45, 14:08:56, and 14:09:07. The caller logs show three read timeouts before the business throws an error.

2026-09-22 14:08:33.512 INFO  [order-gateway,3f9a1c] c.e.g.ChannelClient : 发起渠道下单 bizNo=20260922001
2026-09-22 14:08:43.604 WARN  [order-gateway,3f9a1c] c.e.g.ChannelClient : Read timed out executing POST http://channel-service/order/create
2026-09-22 14:08:54.712 WARN  [order-gateway,3f9a1c] c.e.g.ChannelClient : Read timed out executing POST http://channel-service/order/create
2026-09-22 14:09:05.821 WARN  [order-gateway,3f9a1c] c.e.g.ChannelClient : Read timed out executing POST http://channel-service/order/create
2026-09-22 14:09:05.830 ERROR [order-gateway,3f9a1c] c.e.g.OrderService  : 渠道下单失败，转人工处理 bizNo=20260922001

The three records are spaced 11 seconds apart — exactly the 10-second read timeout plus a 1-second backoff. The downstream service takes a fixed 12 seconds, so each retry fires before the previous attempt finishes, and the channel successfully processes all three.

Timeout Defaults and Read Timeout Semantics

Feign's Request.Options defaults to 10-second connect timeout and 60-second read timeout (since early Feign versions). For modern service-to-service calls, 60 seconds is too long; a stuck downstream can exhaust a 200-thread pool in seconds.

public Options() {
    this(10, TimeUnit.SECONDS, 60, TimeUnit.SECONDS, true);
}

Configuration prefix changed in Spring Cloud 4.x: from feign.client.config to spring.cloud.openfeign.client.config. Upgrading without updating config silently reverts to 60-second read timeout, with no log indication.

spring:
  cloud:
    openfeign:
      client:
        config:
          default:
            connect-timeout: 3000
            read-timeout: 10000
            logger-level: basic
          channel-service:
            read-timeout: 15000

Service-name override (from @FeignClient name) has higher priority than default. Misspelling the name falls back to defaults silently.

Read timeout maps to socket setSoTimeout — it measures the maximum interval between two data packets, not total response time. A 90-second endpoint that sends 1KB every 5 seconds will never hit a 10-second read timeout, occupying the caller thread for the full 90 seconds. To bound total latency, wrap the call with CompletableFuture.orTimeout or configure gateway route timeouts.

Connect timeout covers TCP handshake, TLS handshake, and connection-pool acquisition. When HttpClient 5's pool is exhausted, the request blocks in getConnection until connect timeout, logging a connect timeout even though the downstream never received a packet — misleading troubleshooting toward network hardware.

Four Retry Layers and Their Switches

A single call can pass through four retry layers, each with different defaults and triggers:

Feign Retryer : Default retry: No (Spring Cloud registers Retryer.NEVER_RETRY). Trigger: RetryableException. Switch: Register Retryer Bean.

LoadBalancer : Default retry: No. Trigger: Requires spring-retry on classpath. Switch: spring.cloud.loadbalancer.retry.enabled.

HttpClient 5 : Default retry: Yes. Trigger: Idempotent methods (GET, HEAD) on IO exception. Switch: RequestConfig.

Nginx / Gateway : Default retry: Depends on config. Trigger: error and timeout. Switch: proxy_next_upstream.

The third layer is most easily missed. HttpClient 5's DefaultHttpRequestRetryStrategy retries GET/HEAD on IO exceptions by default, assuming they are safe per HTTP spec. If a write operation is incorrectly mapped to @GetMapping (e.g., cancel order, issue coupon), the transport layer will retry on timeout, executing the side effect again. Such interfaces never error in tests; they only surface when downstream times out.

Nginx's proxy_next_upstream defaults to error timeout, retrying on another upstream on timeout. Combined with application-level retries, a single user click can generate 2 × 3 = 6 backend requests. The layers multiply: Feign 3 attempts × LoadBalancer 2 instances × HttpClient retry = up to a dozen requests, often unaccounted for until rate limiting crashes the caller.

Feign Retryer: Default Off, Cost of Turning On

Native Feign's Retryer.Default uses 100ms initial interval, 1s max interval, 5 attempts, 1.5x backoff. Spring Cloud's FeignClientsConfiguration overrides with Retryer.NEVER_RETRY — documented as intentional. Projects using plain Feign get 5 retries by default, silently resubmitting a timed-out POST four extra times.

To enable, provide a Retryer bean in a @FeignClient configuration:

@Configuration
public class ChannelFeignConfig {
    @Bean
    public Retryer channelRetryer() {
        // 100ms start, 1s cap, max 3 attempts — only for read-only APIs
        return new Retryer.Default(100, TimeUnit.SECONDS.toMillis(1), 3);
    }
}

The author never enables retry on write-operation clients. Feign's Retryer retries the entire execute on any RetryableException, ignoring idempotency — unlike HttpClient which at least checks the HTTP method. Read operations (inventory query, product detail, config fetch) benefit from retry; 2 retries with adequate backoff (not 100ms) can eliminate most transient timeouts. Write operations must handle retries at the business layer with idempotency keys.

Server-Side State After Client Read Timeout

Timeout is a client-side event; the server is unaware. The request arrives, parses, validates, opens a transaction, writes three tables, publishes a message — taking 12 seconds total. At 10 seconds the client closes the socket and throws timeout. The server, mid-transaction (e.g., writing second table), commits at 12 seconds and attempts to write the response on a closed connection, triggering Tomcat's ClientAbortException:

2026-09-22 14:08:45.512 WARN  [http-nio-8080-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] threw exception
java.io.IOException: 你的主机中的软件中止了一个已建立的连接。

This WARN appears after business logic returns; transaction is committed, message sent. Operations often dismiss it as harmless until reconciliation reveals duplicate records under the same business number. The root cause: client thinks failure, server thinks success. Retries amplify the divergence — each retry adds another duplicate.

Idempotency Key and Unique Index Placement

The only reliable guard against duplicates is a database unique constraint. Upper layers can be disabled or tuned, but network timeout always leaves the client uncertain whether the server executed.

Idempotency key is generated by the caller and sent with the request. For channel orders, the upstream callback's bizNo serves as a natural unique key. For user orders, the frontend must generate a UUID, place it in a header (e.g., Idempotency-Key), and reuse the same key for the same submission (refreshing the page breaks idempotency).

A Feign RequestInterceptor adds the header only for write operations (POST):

@Component
public class IdempotencyInterceptor implements RequestInterceptor {
    private static final String HEADER = "Idempotency-Key";
    @Override
    public void apply(RequestTemplate template) {
        if (!"POST".equals(template.method())) return;
        String key = IdempotencyContext.current();
        if (key != null && template.headers().get(HEADER) == null) {
            template.header(HEADER, key);
        }
    }
}

IdempotencyContext

is a ThreadLocal set at business entry and cleared in finally.

Server-side table must have a unique index; a check-then-insert race allows two concurrent retries to both insert. The schema:

CREATE TABLE `idempotent_record` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `idem_key` varchar(64) NOT NULL COMMENT '调用方传入的幂等键',
  `biz_type` varchar(32) NOT NULL COMMENT '业务类型，避免不同业务键冲突',
  `biz_no` varchar(64) DEFAULT NULL COMMENT '首次请求产生的业务单号',
  `status` tinyint NOT NULL DEFAULT 0 COMMENT '0处理中 1成功 2失败',
  `create_time` datetime(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
  `expire_time` datetime NOT NULL COMMENT '过期后可清理，避免表无限增长',
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_key_type` (`idem_key`, `biz_type`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

Processing flow: acquire lock via unique index insert; if duplicate key, load existing record. On success, update status=1 and store biz_no. On failure, delete the record so a retry with the same key re-executes (failure has no side effect, so this trade-off is acceptable).

@Service
public class OrderCreateService {
    private final JdbcTemplate jdbc;
    private final OrderMapper orderMapper;
    public CreateResult create(String idemKey, CreateCmd cmd) {
        try {
            jdbc.update("INSERT INTO idempotent_record(idem_key, biz_type, expire_time) VALUES (?, ?, ?)",
                    idemKey, cmd.getBizType(), LocalDateTime.now().plusHours(24));
        } catch (DuplicateKeyException e) {
            return loadExisting(idemKey);
        }
        try {
            String orderNo = orderMapper.insertOrder(cmd);
            jdbc.update("UPDATE idempotent_record SET status = 1, biz_no = ? WHERE idem_key = ? AND biz_type = ?",
                    orderNo, idemKey, cmd.getBizType());
            return CreateResult.success(orderNo);
        } catch (Exception e) {
            jdbc.update("DELETE FROM idempotent_record WHERE idem_key = ? AND biz_type = ?",
                    idemKey, cmd.getBizType());
            throw e;
        }
    }
}

Unaddressed case: if the first request is still running ( status=0), the second request must return "processing" and let the caller retry later — not success or failure. loadExisting must handle this branch.

Redis setIfAbsent can serve as a fast-path pre-check to reduce DB load, but cannot be the sole guard: Redis restart loses keys, TTL misconfiguration, or primary-secondary failover can lose writes. The database unique index remains the ultimate safety net.

Reproducible Duplicate Request Test

A minimal project: a slow endpoint (12s sleep) and a caller with short timeout and retry.

@RestController
public class ChannelController {
    private final OrderMapper orderMapper;
    @PostMapping("/order/create")
    public String create(@RequestBody CreateCmd cmd,
                         @RequestHeader(value = "Idempotency-Key", required = false) String idemKey)
            throws InterruptedException {
        Thread.sleep(12_000);
        String orderNo = orderMapper.insertOrder(cmd, idemKey);
        log.info("渠道下单完成 bizNo={} orderNo={}", cmd.getBizNo(), orderNo);
        return orderNo;
    }
}

Caller config: read timeout 5s, Retryer.Default(1000, 1000, 3) (fixed 1s interval, 3 total attempts).

spring:
  cloud:
    openfeign:
      client:
        config:
          channel-service:
            connect-timeout: 2000
            read-timeout: 5000

Run one call, wait for server to finish all three, query DB:

curl -s -X POST http://127.0.0.1:8080/gateway/order \
  -H "Content-Type: application/json" \
  -d '{"bizNo":"20260922001","amount":1980}'

mysql -e "SELECT id, order_no, biz_no, create_time FROM orders WHERE biz_no = '20260922001'"

+----+----------------------+-------------+-------------------------+
| id | order_no             | biz_no      | create_time             |
+----+----------------------+-------------+-------------------------+
|  1 | CH20260922000869     | 20260922001 | 2026-09-22 14:08:45.512 |
|  2 | CH20260922000870     | 20260922001 | 2026-09-22 14:08:51.604 |
|  3 | CH20260922000871     | 20260922001 | 2026-09-22 14:08:57.712 |
+----+----------------------+-------------+-------------------------+

Three records spaced 6 seconds apart (5s read timeout + 1s backoff). Server's fixed 12s processing means each retry lands before the previous completes; row count equals attempt count.

Add Idempotency-Key to caller, switch server to unique-index insert logic, rerun — only one row remains. Client still times out; business must still run a compensation job to query the true result — idempotency table alone doesn't solve that.

Configuration-to-Behavior Mapping

Common combinations yield vastly different request counts; worth verifying in order during troubleshooting. Spring Cloud default: one request ( Retryer.NEVER_RETRY). Adding Retryer.Default(100, 1000, 3) makes three — and it retries writes, contrary to many expectations. Enabling LoadBalancer retry multiplies by instance count (requires spring-retry on classpath; missing dependency silently disables it). Switching to HttpClient 5 with GET-mapped write operations adds another hidden layer — the most covert, invisible in business logs.

Timeouts must align with business. Default 60s read timeout is too long for service calls; 3-8s is typical. Shorter timeouts increase retry probability, which demands idempotency. It's a loop: the starting point is whether the business interface can tolerate duplicate execution.

Author's practice: no Retryer on write clients, 2 retries on read clients, all write endpoints carry idempotency keys. Shorter timeouts improve perceived latency. Cost: every write endpoint needs an idempotency table and a cleanup job for expired records — unavoidable operational work.