Designing a Robust Asynchronous Processing SDK with Spring, Kafka, and XXL‑Job

Introduction

When a system evolves, core code changes frequently, increasing the risk of defects. Extending functionality while preserving performance and quality often leads teams to use asynchronous thread pools, which add uncertainty. This SDK provides a generic, non‑intrusive way to execute asynchronous tasks reliably.

Purpose

The SDK guarantees that annotated methods are executed asynchronously without blocking the main workflow and supplies fallback mechanisms to avoid data loss, achieving eventual consistency.

Advantages

Non‑intrusive design: independent database, scheduler, message queue, and manual UI (single‑sign‑on).

Leverages Spring @TransactionalEventListener so async processing does not affect business logic even if the async strategy fails.

If a method runs inside a transaction, the async event is processed after the transaction commits or rolls back.

When the transaction commits but the async strategy fails, a fallback plan executes (unless the database, message queue, or method itself is broken).

Principle

During container startup, all beans are scanned and methods annotated with @AsyncExec are cached. At runtime an AOP aspect publishes an event for the method. A @TransactionalEventListener with fallbackExecution=true and phase=AFTER_COMPLETION processes the async execution strategy, ensuring handling regardless of transaction outcome.

Components

Kafka message queue

XXL‑Job scheduler

MySQL database

Spring AOP aspect

Vue front‑end UI

Design Patterns

Strategy pattern

Template method

Dynamic proxy

Database Scripts

CREATE TABLE `async_scene` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'Primary key',
  `application_name` varchar(100) NOT NULL DEFAULT '' COMMENT 'Application name',
  `method_sign` varchar(50) NOT NULL DEFAULT '' COMMENT 'Method signature',
  `scene_name` varchar(200) NOT NULL DEFAULT '' COMMENT 'Business scene description',
  `async_type` varchar(50) NOT NULL DEFAULT '' COMMENT 'Async strategy type',
  `queue_name` varchar(200) NOT NULL DEFAULT '' COMMENT 'Queue name',
  `theme_value` varchar(100) NOT NULL DEFAULT '' COMMENT 'Consumer topic',
  `exec_count` int NOT NULL DEFAULT '0' COMMENT 'Retry count',
  `exec_deleted` int NOT NULL DEFAULT '0' COMMENT 'Delete after execution',
  `async_version` varchar(50) NOT NULL DEFAULT '' COMMENT 'Component version',
  `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation time',
  `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Update time',
  `cdc_crt_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Record insert time',
  `cdc_upd_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'Record update time',
  PRIMARY KEY (`id`),
  UNIQUE KEY `uk_application_sign` (`application_name`,`method_sign`),
  KEY `idx_cdc_upd_time` (`cdc_upd_time`)
) ENGINE=InnoDB COMMENT='Async scene table';

CREATE TABLE `async_req` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'Primary key',
  `application_name` varchar(100) NOT NULL DEFAULT '' COMMENT 'Application name',
  `sign` varchar(50) NOT NULL DEFAULT '' COMMENT 'Method signature',
  `class_name` varchar(200) NOT NULL DEFAULT '' COMMENT 'Full class name',
  `method_name` varchar(100) NOT NULL DEFAULT '' COMMENT 'Method name',
  `async_type` varchar(50) NOT NULL DEFAULT '' COMMENT 'Async strategy type',
  `exec_status` tinyint NOT NULL DEFAULT '0' COMMENT '0: init, 1: fail, 2: success',
  `exec_count` int NOT NULL DEFAULT '0' COMMENT 'Execution count',
  `param_json` longtext COMMENT 'Request parameters',
  `remark` varchar(200) NOT NULL DEFAULT '' COMMENT 'Business description',
  `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation time',
  `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Update time',
  PRIMARY KEY (`id`),
  KEY `idx_applocation_name` (`application_name`),
  KEY `idx_exec_status` (`exec_status`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Async request table';

CREATE TABLE `async_log` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'Primary key',
  `async_id` bigint NOT NULL DEFAULT '0' COMMENT 'Async request ID',
  `error_data` longtext COMMENT 'Error information',
  `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation time',
  PRIMARY KEY (`id`),
  KEY `idx_async_id` (`async_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Async log table';

Asynchronous Strategies

The SDK defines three layers:

Strategy selection (choose Kafka, XXL‑Job, or direct DB insert)

Template method execution (common workflow: persist request, publish event, handle result)

Dynamic proxy handling (generated at runtime for annotated methods)

Configuration (Apollo)

# Switch (default off)
async.enabled=true

# Application name
spring.application.name=your-app

# Data source (Druid)
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://127.0.0.1:3306/fc_async?useUnicode=true&characterEncoding=utf-8&zeroDateTimeBehavior=convertToNull&useSSL=false&allowMultiQueries=true&rewriteBatchedStatements=true
spring.datasource.username=user
spring.datasource.password=******
spring.datasource.filters=config
spring.datasource.connectionProperties=config.decrypt=true;config.decrypt.key=your-key

# Static resources
spring.resources.add-mappings=true
spring.resources.static-locations=classpath:/static/

# Thread pool defaults (override as needed)
async.executor.thread.corePoolSize=10
async.executor.thread.maxPoolSize=50
async.executor.thread.queueCapacity=10000
async.executor.thread.keepAliveSeconds=600
async.exec.deleted=true
async.topic=${spring.application.name}
async.exec.count=5
async.retry.limit=100
async.comp.limit=100
async.login=false

Usage

# 1. Enable async processing
scm.async.enabled=true

# 2. Annotate methods that need async execution (must be Spring‑proxied)
@AsyncExec(type=AsyncExecEnum.SAVE_ASYNC, remark="Data dictionary")

# 3. Manual handling UI (optional)
http://localhost:8004/async/index.html

Notes

When configuring the SDK, set a unique spring.application.name. The default queue name is ${async.topic}_${spring.application.name}_async_queue. Ensure business logic is idempotent because a single queue is shared by all instances of the same application. The SDK provides two scheduled tasks:

Async retry task (default every 2 minutes, configurable retry count).

Async compensation task (default every hour, processes records older than one hour).

Effect Demonstration

Running the SDK generates logs and database entries that show successful asynchronous execution, automatic retries on failure, and eventual consistency across services.

Git Repository

Source code: https://github.com/xiongyanokok/fc-async.git