Skip to content
Spring Boot + Doris Stream Load: Real-Time Massive Data Updates with Parallel Processing

Spring Boot + Doris Stream Load: Real-Time Massive Data Updates with Parallel Processing

This article presents a complete Spring Boot integration with Apache Doris Stream Load for real-time massive data updates, featuring annotation-driven field mapping, parallel batch processing with exponential backoff retry, GZIP compression, and configurable performance tuning.

Java Architect Handbook
Java Architect Handbook
Java Architect Handbook

Overall Architecture

The project follows a standard Spring Boot structure with dedicated packages for annotations, configuration, core loading logic, entities, services, and utilities.

src/main/java/com/example/doris/annotation/DorisField.java
src/main/java/com/example/doris/config/DorisConfig.java
src/main/java/com/example/doris/config/DorisStreamLoadProperties.java
src/main/java/com/example/doris/core/DorisStreamLoader.java
src/main/java/com/example/doris/entity/User.java
src/main/java/com/example/doris/service/UserService.java
src/main/java/com/example/doris/util/DorisMapper.java
src/main/java/com/example/doris/util/TableUtils.java
src/main/resources/application.yml

1. Maven Dependencies (pom.xml)

Key dependencies include Spring Boot Web, Apache HttpClient 4.5.13, Jackson with JSR310 module, Lombok, Guava 31.1-jre, and Reflections 0.10.2 for annotation scanning.

2. Configuration Properties

DorisStreamLoadProperties

binds to doris.stream-load prefix with defaults: connectTimeout=30000ms, socketTimeout=60000ms, batchSize=50000, maxRetries=3, compression=none, maxParallel=4, columnSeparator=\x01, lineSeparator=\n.

3. Field Mapping Annotation

@DorisField

annotation supports three attributes: value for custom column name (defaults to field name), order for column ordering (lower values first), and ignore to exclude fields from mapping.

4. Automatic Mapping Utility

DorisMapper

provides two conversion methods: convertToDorisData: Uses reflection to read annotated fields in order, formats Date fields as yyyy-MM-dd HH:mm:ss, supports additional fields injection. convertWithJackson: Uses Jackson ObjectMapper for serialization to LinkedHashMap.

Field ordering is cached via ConcurrentHashMap to avoid repeated reflection.

5. Core Stream Loader

DorisStreamLoader

handles parallel batch loading with retry logic:

Initializes HTTP client with connection pool sized to maxParallel * 2 total connections. parallelStreamLoad partitions data into batches of batchSize, submits each batch to a fixed thread pool via CompletableFuture.runAsync, and waits for all to complete. sendBatch converts batch to JSON or CSV (using configured separators) and calls attemptSendWithRetry.

Retry uses exponential backoff: delay = 1000 * 2^retryCount ms, up to maxRetries.

Supports GZIP compression via custom GzipCompressingEntity that sets Content-Encoding: gzip header.

Sets Doris-specific headers: Authorization (Basic auth), format (json/csv), column_separator, line_delimiter, strip_outer_array=true for JSON.

Parses response into LoadResponse and logs NumberLoadedRows and LoadTimeMs.

6. Load Response Model

LoadResponse

captures Doris Stream Load result fields: TxnId, Label, Status, NumberTotalRows, NumberLoadedRows, NumberFilteredRows, NumberUnselectedRows, LoadBytes, and detailed timing metrics ( LoadTimeMs, BeginTxnTimeMs, StreamLoadPutTimeMs, ReadDataTimeMs, WriteDataTimeMs, ReceiveDataTimeMs, CommitAndPublishTimeMs).

7. Table Name Resolution

TableUtils.getTableName

reads MyBatis-Plus @TableName annotation; falls back to camelCase-to-snake_case conversion (e.g., User → user).

8. Entity Example

User

entity demonstrates annotation usage: id (order=1), name mapped to user_name (order=2), age (order=3), status mapped to account_status (order=4), password ignored, email excluded (no annotation), createTime (order=5).

9. Service Layer

UserService

shows two operations: batchUpdateUserStatus and batchCreateUsers. Both inject timestamp fields ( update_time or create_time), convert entities via DorisMapper.convertToDorisData, and invoke dorisStreamLoader.parallelStreamLoad with format "json", database name from UserProvider, and table name from TableUtils.

10. Spring Configuration

DorisConfig

enables configuration properties and exposes DorisStreamLoader as a bean.

11. Application Configuration (application.yml)

Example production settings: URL template http://doris-fe:8030/api/user_db/user_table/_stream_load, batch-size=100000, max-retries=5, compression=gzip, max-parallel=8, connect-timeout=60000, socket-timeout=300000. Jackson date format set to yyyy-MM-dd HH:mm:ss with GMT+8 timezone.

Key Features and Advantages

1. Automated Field Mapping

Annotation-driven mapping eliminates manual field-to-column wiring. Supports renaming ( value), ordering ( order), and exclusion ( ignore=true).

2. High-Performance Processing

Batching: 100,000 records per batch (configurable).

Parallelism: 8 threads (configurable via max-parallel).

Compression: GZIP reduces network payload.

Resilience: Exponential backoff retry (1s, 2s, 4s, ...).

3. Reflection Caching

Annotated fields are cached in ConcurrentHashMap<Class<?>, List<Field>> to avoid repeated reflection overhead.

4. Result Monitoring

Response parsing logs success metrics: loaded row count and load time in milliseconds. Non-Success status throws IOException with full response details.

Performance Optimization Recommendations

Batch size : Tune batch-size (e.g., 50,000) based on Doris cluster capacity.

Parallelism : Set max-parallel to CPU_CORES * 2.

Compression : Enable compression: gzip to reduce network transfer.

Timeouts : Increase connect-timeout to 120s and socket-timeout to 600s for large payloads.

Doris cluster settings : Raise streaming_load_max_mb to 4096 and max_running_txn_num_per_db to 1024 via SQL.

The article claims 20-50x performance improvement over JDBC batch updates for massive data scenarios, achieved by eliminating manual mapping and leveraging Doris native Stream Load parallelism.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

Real-time AnalyticsBatch ProcessingSpring BootParallel ProcessingApache DorisGZIP CompressionStream LoadAnnotation-driven Mapping
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.