Spring Boot + Doris Stream Load: Real-Time Massive Data Updates with Parallel Processing
This article presents a complete Spring Boot integration with Apache Doris Stream Load for real-time massive data updates, featuring annotation-driven field mapping, parallel batch processing with exponential backoff retry, GZIP compression, and configurable performance tuning.
Overall Architecture
The project follows a standard Spring Boot structure with dedicated packages for annotations, configuration, core loading logic, entities, services, and utilities.
src/main/java/com/example/doris/annotation/DorisField.java
src/main/java/com/example/doris/config/DorisConfig.java
src/main/java/com/example/doris/config/DorisStreamLoadProperties.java
src/main/java/com/example/doris/core/DorisStreamLoader.java
src/main/java/com/example/doris/entity/User.java
src/main/java/com/example/doris/service/UserService.java
src/main/java/com/example/doris/util/DorisMapper.java
src/main/java/com/example/doris/util/TableUtils.java
src/main/resources/application.yml1. Maven Dependencies (pom.xml)
Key dependencies include Spring Boot Web, Apache HttpClient 4.5.13, Jackson with JSR310 module, Lombok, Guava 31.1-jre, and Reflections 0.10.2 for annotation scanning.
2. Configuration Properties
DorisStreamLoadPropertiesbinds to doris.stream-load prefix with defaults: connectTimeout=30000ms, socketTimeout=60000ms, batchSize=50000, maxRetries=3, compression=none, maxParallel=4, columnSeparator=\x01, lineSeparator=\n.
3. Field Mapping Annotation
@DorisFieldannotation supports three attributes: value for custom column name (defaults to field name), order for column ordering (lower values first), and ignore to exclude fields from mapping.
4. Automatic Mapping Utility
DorisMapperprovides two conversion methods: convertToDorisData: Uses reflection to read annotated fields in order, formats Date fields as yyyy-MM-dd HH:mm:ss, supports additional fields injection. convertWithJackson: Uses Jackson ObjectMapper for serialization to LinkedHashMap.
Field ordering is cached via ConcurrentHashMap to avoid repeated reflection.
5. Core Stream Loader
DorisStreamLoaderhandles parallel batch loading with retry logic:
Initializes HTTP client with connection pool sized to maxParallel * 2 total connections. parallelStreamLoad partitions data into batches of batchSize, submits each batch to a fixed thread pool via CompletableFuture.runAsync, and waits for all to complete. sendBatch converts batch to JSON or CSV (using configured separators) and calls attemptSendWithRetry.
Retry uses exponential backoff: delay = 1000 * 2^retryCount ms, up to maxRetries.
Supports GZIP compression via custom GzipCompressingEntity that sets Content-Encoding: gzip header.
Sets Doris-specific headers: Authorization (Basic auth), format (json/csv), column_separator, line_delimiter, strip_outer_array=true for JSON.
Parses response into LoadResponse and logs NumberLoadedRows and LoadTimeMs.
6. Load Response Model
LoadResponsecaptures Doris Stream Load result fields: TxnId, Label, Status, NumberTotalRows, NumberLoadedRows, NumberFilteredRows, NumberUnselectedRows, LoadBytes, and detailed timing metrics ( LoadTimeMs, BeginTxnTimeMs, StreamLoadPutTimeMs, ReadDataTimeMs, WriteDataTimeMs, ReceiveDataTimeMs, CommitAndPublishTimeMs).
7. Table Name Resolution
TableUtils.getTableNamereads MyBatis-Plus @TableName annotation; falls back to camelCase-to-snake_case conversion (e.g., User → user).
8. Entity Example
Userentity demonstrates annotation usage: id (order=1), name mapped to user_name (order=2), age (order=3), status mapped to account_status (order=4), password ignored, email excluded (no annotation), createTime (order=5).
9. Service Layer
UserServiceshows two operations: batchUpdateUserStatus and batchCreateUsers. Both inject timestamp fields ( update_time or create_time), convert entities via DorisMapper.convertToDorisData, and invoke dorisStreamLoader.parallelStreamLoad with format "json", database name from UserProvider, and table name from TableUtils.
10. Spring Configuration
DorisConfigenables configuration properties and exposes DorisStreamLoader as a bean.
11. Application Configuration (application.yml)
Example production settings: URL template http://doris-fe:8030/api/user_db/user_table/_stream_load, batch-size=100000, max-retries=5, compression=gzip, max-parallel=8, connect-timeout=60000, socket-timeout=300000. Jackson date format set to yyyy-MM-dd HH:mm:ss with GMT+8 timezone.
Key Features and Advantages
1. Automated Field Mapping
Annotation-driven mapping eliminates manual field-to-column wiring. Supports renaming ( value), ordering ( order), and exclusion ( ignore=true).
2. High-Performance Processing
Batching: 100,000 records per batch (configurable).
Parallelism: 8 threads (configurable via max-parallel).
Compression: GZIP reduces network payload.
Resilience: Exponential backoff retry (1s, 2s, 4s, ...).
3. Reflection Caching
Annotated fields are cached in ConcurrentHashMap<Class<?>, List<Field>> to avoid repeated reflection overhead.
4. Result Monitoring
Response parsing logs success metrics: loaded row count and load time in milliseconds. Non-Success status throws IOException with full response details.
Performance Optimization Recommendations
Batch size : Tune batch-size (e.g., 50,000) based on Doris cluster capacity.
Parallelism : Set max-parallel to CPU_CORES * 2.
Compression : Enable compression: gzip to reduce network transfer.
Timeouts : Increase connect-timeout to 120s and socket-timeout to 600s for large payloads.
Doris cluster settings : Raise streaming_load_max_mb to 4096 and max_running_txn_num_per_db to 1024 via SQL.
The article claims 20-50x performance improvement over JDBC batch updates for massive data scenarios, achieved by eliminating manual mapping and leveraging Doris native Stream Load parallelism.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
