Designing a Robust Asynchronous Processing SDK with Spring, Kafka, and MySQL

1 Introduction

A well‑designed system must follow the open‑closed principle; as business logic evolves, core code changes frequently, increasing the risk of bugs. Most new features extend existing functionality, requiring both performance and quality. Developers often resort to asynchronous thread pools, which introduce additional uncertainty.

To address this, I designed a generic asynchronous processing SDK that can easily handle various async tasks.

2 Purpose

The SDK ensures that methods are executed reliably without blocking the primary workflow.

More importantly, fallback mechanisms guarantee data is not lost, achieving eventual consistency.

3 Advantages

Non‑intrusive design with independent database, scheduled jobs, message queue, and manual execution UI (single sign‑on).

Leverages Spring transaction event mechanism; even if async parsing fails, business logic remains unaffected.

If a method runs inside a transaction, the event is processed after commit or rollback.

Should parsing fail after commit, a fallback path executes unless the database, message queue, or method itself is broken.

4 Principle

After container initialization, all beans are scanned and methods annotated with @AsyncExec are cached.

At runtime, an AOP aspect publishes an event for the annotated method.

A transaction‑event listener then handles the asynchronous execution strategy.

@TransactionalEventListener(fallbackExecution = true, phase = TransactionPhase.AFTER_COMPLETION)

fallbackExecution=true

– processes the event even when no transaction is active. TransactionPhase.AFTER_COMPLETION – handles the event after both commit and rollback.

5 Components

Kafka message queue

XXL‑Job scheduled tasks

MySQL database

Spring AOP

Vue front‑end UI

6 Design Patterns

Strategy

Template Method

Dynamic Proxy

7 Flowchart

8 Database Scripts

CREATE TABLE `async_req` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'Primary Key',
  `application_name` varchar(100) NOT NULL DEFAULT '' COMMENT 'Application Name',
  `sign` varchar(50) NOT NULL DEFAULT '' COMMENT 'Method Signature',
  `class_name` varchar(200) NOT NULL DEFAULT '' COMMENT 'Full Class Name',
  `method_name` varchar(100) NOT NULL DEFAULT '' COMMENT 'Method Name',
  `async_type` varchar(50) NOT NULL DEFAULT '' COMMENT 'Async Strategy Type',
  `exec_status` tinyint NOT NULL DEFAULT '0' COMMENT 'Execution Status 0:Init 1:Fail 2:Success',
  `exec_count` int NOT NULL DEFAULT '0' COMMENT 'Execution Count',
  `param_json` longtext COMMENT 'Request Parameters',
  `remark` varchar(200) NOT NULL DEFAULT '' COMMENT 'Business Description',
  `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Create Time',
  `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Update Time',
  PRIMARY KEY (`id`) USING BTREE,
  KEY `idx_applocation_name` (`application_name`) USING BTREE,
  KEY `idx_exec_status` (`exec_status`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Async Request';

CREATE TABLE `async_log` (
  `id` bigint NOT NULL AUTO_INCREMENT COMMENT 'Primary Key',
  `async_id` bigint NOT NULL DEFAULT '0' COMMENT 'Async Request ID',
  `error_data` longtext COMMENT 'Error Information',
  `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Create Time',
  PRIMARY KEY (`id`) USING BTREE,
  KEY `idx_async_id` (`async_id`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Async Log';

9 Async Strategy

10 Security Levels

11 Execution Status

12 Additional Flowcharts

13 Apollo Configuration

# Switch (default off)
async.enabled=true

# Application name
spring.application.name=xxx

# Data source (Druid)
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://127.0.0.1:3306/fc_async?useUnicode=true&characterEncoding=utf-8&zeroDateTimeBehavior=convertToNull&useSSL=false&allowMultiQueries=true&rewriteBatchedStatements=true
spring.datasource.username=user
spring.datasource.password=xxxx
spring.datasource.filters=config
spring.datasource.connectionProperties=config.decrypt=true;config.decrypt.key=yyy
spring.resources.add-mappings=true
spring.resources.static-locations=classpath:/static/

# Core thread pool settings (defaults shown)
async.executor.thread.corePoolSize=10
async.executor.thread.maxPoolSize=50
async.executor.thread.queueCapacity=10000
async.executor.thread.keepAliveSeconds=600

# Delete record after successful execution (default true)
async.exec.deleted=true

# Custom queue name prefix (default application name)
async.topic=${spring.application.name}

# Retry count (default 5)
async.exec.count=5

# Retry query limit
async.retry.limit=100

# Compensation query limit
async.comp.limit=100

# Login interception (default false)
async.login=false

14 Usage

1. Enable async switch scm.async.enabled=true 2. Annotate methods that need async execution (must be Spring‑proxied)

@AsyncExec(type = AsyncExecEnum.SAVE_ASYNC, remark = "Data Dictionary")

3. Manual handling address

http://localhost:8004/async/index.html

15 Cautions

1. Application name spring.application.name 2. Queue name

${async.topic:${spring.application.name}}_async_queue

Custom topic can be set via async.topic=xxx.

3. Ensure business logic is idempotent

4. One queue per application

Self‑producing and self‑consuming.

5. Scheduled tasks

Async retry task (retries every 2 minutes, configurable retry count)

Async compensation task (runs hourly for records older than one hour)

16 Demo

17 GitHub Repository

https://github.com/xiongyanokok/fc-async.git