Skip to content
Tagged articles

streaming

500 articles · Page 2 of 5
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Jul 22, 2025 · Big Data

How Apache SeaTunnel Revolutionizes Heterogeneous Data Integration with Decoupled Connectors

This article explores how Apache SeaTunnel addresses modern data integration challenges by providing a high‑performance, distributed, plugin‑based platform that decouples connectors from execution engines, enabling seamless batch and streaming synchronization across heterogeneous sources such as databases, message queues, and data lakes.

Apache SeaTunnelConnector Architecturebatch processing
0 likes · 24 min read
How Apache SeaTunnel Revolutionizes Heterogeneous Data Integration with Decoupled Connectors
DataFunSummit
DataFunSummit
Jul 12, 2025 · Big Data

How Fluss Unifies Stream and Lake to Power AI Data Pipelines

In the era of rapid AI growth, Fluss offers a unified lake‑stream architecture that tackles data quality, timeliness, scale, and multimodal challenges by tightly integrating Flink streaming with a high‑performance data lake, enabling seamless real‑time and batch analytics for AI workloads.

AIFlinkFluss
0 likes · 12 min read
How Fluss Unifies Stream and Lake to Power AI Data Pipelines
Big Data Technology & Architecture
Big Data Technology & Architecture
Jul 8, 2025 · Big Data

Flink’s AI Agents and Disaggregated State: Transforming Big Data

The article reviews key topics from the FFA2025 Singapore conference, highlighting Flink’s new AI‑focused Agents framework, the breakthrough Flink 2.0 disaggregated state architecture, emerging lake storage solutions like Paimon, and the Fluss streaming table store, illustrating how big‑data platforms are evolving for AI workloads.

AI agentsDisaggregated StateFlink
0 likes · 6 min read
Flink’s AI Agents and Disaggregated State: Transforming Big Data
FunTester
FunTester
Jul 5, 2025 · Big Data

Master Kafka: Core Concepts and Performance Testing Strategies

This article explains Kafka’s high‑performance distributed streaming architecture, key components such as topics, partitions, producers, consumers, brokers, offsets, and ZooKeeper, and provides step‑by‑step workflows for producers and consumers along with performance‑testing tips and Maven setup.

JavaKafkaMessage Queue
0 likes · 9 min read
Master Kafka: Core Concepts and Performance Testing Strategies
Big Data Tech Team
Big Data Tech Team
Jul 3, 2025 · Big Data

Master Kafka: A Complete Learning Roadmap from Basics to Advanced Projects

This guide presents a step‑by‑step Kafka learning roadmap covering core concepts, architecture, configuration, monitoring tools, practical project ideas, advanced components like Streams and KSQL, plus code samples and resource recommendations to help beginners become proficient in real‑time data streaming.

Code ExamplesKafkaTutorial
0 likes · 14 min read
Master Kafka: A Complete Learning Roadmap from Basics to Advanced Projects
FunTester
FunTester
Jul 2, 2025 · Backend Development

Master Non‑Blocking & Streaming gRPC Clients in Java for High‑Performance Testing

Learn how to create and use non‑blocking (asynchronous) and streaming gRPC clients in Java, covering stub creation, request handling with ListenableFuture, blocking vs listener approaches, and stream observer implementation, enabling high‑concurrency performance testing and real‑time data processing scenarios.

JavaasynchronousgRPC
0 likes · 7 min read
Master Non‑Blocking & Streaming gRPC Clients in Java for High‑Performance Testing
Architecture & Thinking
Architecture & Thinking
Jun 11, 2025 · Artificial Intelligence

Accelerate LLM App Development with Eino: A Go Framework Walkthrough

Eino is an open‑source Golang framework for building large‑model applications, offering reusable components, robust orchestration, clean APIs, best‑practice templates, and full‑cycle DevOps tools, with code examples for both Ollama and OpenAI modes, plus streaming and normal output options.

AI developmentGoLLM
0 likes · 10 min read
Accelerate LLM App Development with Eino: A Go Framework Walkthrough
Linux Cloud Computing Practice
Linux Cloud Computing Practice
May 29, 2025 · Big Data

Why Learn Kafka? Core Benefits, Use Cases, and Key Interview Topics

This article explains why Kafka is essential for modern data engineering, highlighting its widespread adoption, high throughput, scalability, durability, integration with streaming ecosystems, and common real‑time use cases, while also providing a concise list of interview topics for aspiring engineers.

data pipelinesreal-time processingstreaming
0 likes · 6 min read
Why Learn Kafka? Core Benefits, Use Cases, and Key Interview Topics
Code Mala Tang
Code Mala Tang
May 21, 2025 · Backend Development

Master FastAPI Responses: JSON, HTML, Files, Streaming & Custom Classes

Learn how FastAPI handles various response types—including default JSON, custom status codes and headers, HTMLResponse, FileResponse, StreamingResponse, and user‑defined response classes—while covering best practices, common pitfalls, and code examples for building robust APIs.

API responsesFastAPIPython
0 likes · 8 min read
Master FastAPI Responses: JSON, HTML, Files, Streaming & Custom Classes
Huolala Tech
Huolala Tech
May 14, 2025 · Big Data

How Lalamove Scaled Real‑Time Data Warehousing with Flink and Paimon

Lalamove’s international logistics platform transformed its real‑time data warehouse by leveraging Apache Flink and the Paimon lakehouse, addressing challenges of multi‑region data centers, time‑zone diversity, frequent upstream changes, and high costs, while improving scalability, latency, and operational efficiency across global markets.

FlinkPaimonbig data
0 likes · 13 min read
How Lalamove Scaled Real‑Time Data Warehousing with Flink and Paimon
DataFunSummit
DataFunSummit
May 4, 2025 · Big Data

Iceberg Table Format Practice in Huawei Terminal Cloud

This article explains how Huawei's terminal cloud adopts the Apache Iceberg table format to efficiently manage large-scale datasets, detailing its architecture, feature engineering, merge operations, LSM-based storage, schema versioning, AB testing support, catalog enhancements, and future roadmap for full lifecycle data governance.

Huawei CloudIcebergSchema Management
0 likes · 13 min read
Iceberg Table Format Practice in Huawei Terminal Cloud
Su San Talks Tech
Su San Talks Tech
May 4, 2025 · Backend Development

How to Export Millions of Excel Rows in Seconds: High‑Performance Java Strategies

Learn how to overcome memory and speed bottlenecks when exporting massive datasets to Excel by using streaming APIs like SXSSFWorkbook and EasyExcel, optimizing database pagination, tuning JVM and connection pools, and applying asynchronous shard processing to achieve stable sub‑200 MB memory usage for millions of rows.

Asynchronous ProcessingDatabase paginationExcel Export
0 likes · 10 min read
How to Export Millions of Excel Rows in Seconds: High‑Performance Java Strategies
Java Architect Essentials
Java Architect Essentials
Apr 14, 2025 · Backend Development

Using Spring's ResponseBodyEmitter for Real-Time Log Streaming

This article introduces Spring Framework's ResponseBodyEmitter, explains its role in asynchronous HTTP responses, outlines typical use cases such as long polling and real‑time log streaming, provides a complete Spring Boot example with code, and compares it with SSE and raw OutputStream approaches.

JavaResponseBodyEmitterSpring
0 likes · 11 min read
Using Spring's ResponseBodyEmitter for Real-Time Log Streaming
DataFunTalk
DataFunTalk
Apr 9, 2025 · Big Data

Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies

The Kuaishou‑hosted Apache Hudi Asia technical salon gathered over 230 attendees and featured seven experts from Kuaishou, Meituan, TikTok, Huawei, JD and others, who shared best practices, architecture designs, and performance optimizations for large‑scale data lake applications across AI, BI, and real‑time workloads.

AIApache Hudibatch processing
0 likes · 14 min read
Highlights of the Apache Hudi Asia Technical Salon Hosted by Kuaishou – Practices and Innovations from Leading Companies
Architecture Digest
Architecture Digest
Apr 3, 2025 · Backend Development

Real‑Time Streaming with Spring’s ResponseBodyEmitter: Concepts, Use Cases, and Code Example

This article explains the purpose, core methods, and practical scenarios of Spring Framework’s ResponseBodyEmitter, compares it with SSE and raw streaming, and provides a complete Spring Boot controller example that demonstrates how to implement real‑time log streaming and other asynchronous HTTP responses.

BackendReal-timeResponseBodyEmitter
0 likes · 9 min read
Real‑Time Streaming with Spring’s ResponseBodyEmitter: Concepts, Use Cases, and Code Example
DataFunSummit
DataFunSummit
Apr 3, 2025 · Big Data

Apache Hudi Asia Technical Salon Highlights: Practices and Innovations from Kuaishou, Meituan, Douyin, Huawei, and JD

The Apache Hudi Asia technical salon held in Beijing on March 29 gathered over 230 on‑site participants and 16,000 online viewers, featuring expert talks from leading Chinese tech companies that showcased real‑world Hudi implementations, performance optimizations, and future roadmap for data‑lake technologies.

Apache HudiFlinkSpark
0 likes · 13 min read
Apache Hudi Asia Technical Salon Highlights: Practices and Innovations from Kuaishou, Meituan, Douyin, Huawei, and JD
DataFunSummit
DataFunSummit
Apr 1, 2025 · Big Data

Understanding Flink CDC 3.3: Features, Improvements, and Future Plans

This article provides a comprehensive overview of Flink CDC 3.3, detailing its CDC fundamentals, new connectors, Transform module enhancements, asynchronous snapshot splitting, community adoption, and upcoming roadmap for broader ecosystem support and batch‑mode execution.

CDCFlinkbig data
0 likes · 15 min read
Understanding Flink CDC 3.3: Features, Improvements, and Future Plans
iQIYI Technical Product Team
iQIYI Technical Product Team
Mar 27, 2025 · Big Data

Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg

iQIYI transformed its real‑time data warehouse by replacing a costly Kafka‑based Lambda stack with a unified stream‑batch Iceberg lake, cutting storage expenses by 90%, halving compute costs, extending data retention, and delivering minute‑level freshness for 90% of use cases while preserving second‑level processing where needed.

FlinkIcebergKafka
0 likes · 11 min read
Cost‑Effective Real‑Time Data Warehouse 2.0: Migrating from Kafka to Iceberg
Airbnb Technology Team
Airbnb Technology Team
Mar 24, 2025 · Artificial Intelligence

Chronon: Open‑Source Feature Platform for Machine Learning – Architecture, Workflow, and Code Examples

Chronon is an open‑source ML feature platform that lets engineers declaratively define, compute, and serve both batch and real‑time features with built‑in observability, data‑quality checks, and a low‑latency retrieval API, ensuring online‑offline consistency while simplifying pipeline management and enabling future automation.

Chronondata pipelinesobservability
0 likes · 13 min read
Chronon: Open‑Source Feature Platform for Machine Learning – Architecture, Workflow, and Code Examples
Top Architecture Tech Stack
Top Architecture Tech Stack
Mar 24, 2025 · Backend Development

Using Spring's ResponseBodyEmitter for Real‑Time Log Streaming

This article explains how Spring Framework's ResponseBodyEmitter enables real‑time, chunked HTTP responses for use cases such as log streaming, progress updates, chat, and AI output, detailing its advantages over SSE, usage scenarios, core methods, a complete controller example, and best‑practice considerations.

BackendJavaResponseBodyEmitter
0 likes · 9 min read
Using Spring's ResponseBodyEmitter for Real‑Time Log Streaming
Big Data Technology & Architecture
Big Data Technology & Architecture
Mar 24, 2025 · Big Data

Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More

Apache Kafka 4.0 introduces a groundbreaking KRaft architecture that removes ZooKeeper, a revamped consumer‑group protocol that dramatically speeds up rebalancing, a new queue mode for point‑to‑point messaging, upgraded Java requirements, streamlined APIs, and numerous performance and security enhancements, reshaping both development and operations for large‑scale streaming workloads.

Java 11Java 17KRaft
0 likes · 12 min read
Apache Kafka 4.0: Major New Features – KRaft Architecture, Consumer Group Protocol, Queue Mode, Java Upgrade, API Simplifications and More
Laravel Tech Community
Laravel Tech Community
Mar 23, 2025 · Big Data

Apache Kafka 4.0 Released: First Version Without ZooKeeper and New Features

Apache Kafka 4.0 has been officially released as the first major version that runs entirely without Apache ZooKeeper, introducing KRaft mode, a new consumer group protocol (KIP‑848), early‑access queue support (KIP‑932), updated Java requirements, and other enhancements aimed at improving scalability, operability, and messaging versatility.

Apache KafkaKIP-848KIP-932
0 likes · 3 min read
Apache Kafka 4.0 Released: First Version Without ZooKeeper and New Features
Alimama Tech
Alimama Tech
Mar 12, 2025 · Big Data

Design and Evolution of Alibaba Advertising Real-Time Data Warehouse

Alibaba Mama’s advertising platform migrated from a monolithic Flink‑Kafka pipeline to a layered Paimon lakehouse, adding DWS upsert support and multi‑layer storage, which delivers minute‑level data freshness, cuts latency by 2.5 hours, reduces resource use over 40 %, halves development effort and achieves ≥99.9 % availability.

AdvertisingAlibabaFlink
0 likes · 18 min read
Design and Evolution of Alibaba Advertising Real-Time Data Warehouse
Radish, Keep Going!
Radish, Keep Going!
Mar 12, 2025 · Backend Development

Why gRPC Beats REST: Performance, Contracts, and Streaming Explained

gRPC offers superior performance, efficient Protobuf encoding, strong API contracts, seamless streaming, cross‑language support, and HTTP/2 advantages, making it a compelling alternative to REST for modern web services, while tools like gRPC‑Gateway, ConnectRPC, and Buf streamline migration and ecosystem integration.

API contractsgRPCstreaming
0 likes · 10 min read
Why gRPC Beats REST: Performance, Contracts, and Streaming Explained
Java Architect Essentials
Java Architect Essentials
Mar 7, 2025 · Artificial Intelligence

Introducing DeepSeek4j 1.4: A Java Spring Boot Integration for DeepSeek AI with Chain‑of‑Thought and Streaming Support

The article introduces DeepSeek4j 1.4, a Java Spring Boot library that overcomes existing framework limitations by preserving DeepSeek's chain‑of‑thought capabilities, adding full reactive streaming, and providing a simple one‑line API along with quick‑start instructions and code examples.

AI IntegrationChain-of-ThoughtDeepSeek
0 likes · 5 min read
Introducing DeepSeek4j 1.4: A Java Spring Boot Integration for DeepSeek AI with Chain‑of‑Thought and Streaming Support
Code Ape Tech Column
Code Ape Tech Column
Mar 7, 2025 · Backend Development

Using Spring's ResponseBodyEmitter for Real‑Time Streaming Responses

This article introduces Spring Framework's ResponseBodyEmitter, explains its advantages over SSE for asynchronous HTTP responses, demonstrates practical usage with a real‑time log‑streaming example, outlines core methods, working principles, best‑practice considerations, and compares it with traditional Streaming and Server‑Sent Events.

BackendHTTPJava
0 likes · 10 min read
Using Spring's ResponseBodyEmitter for Real‑Time Streaming Responses
Alibaba Cloud Infrastructure
Alibaba Cloud Infrastructure
Mar 6, 2025 · Big Data

Leveraging Apache Iceberg and AutoMQ for Real-Time Data Lake Ingestion: Architecture, Best Practices, and Cost Optimization

This article examines how Apache Iceberg’s snapshot‑based ACID transactions, logical‑physical partition evolution, and COW/MOR update modes enable efficient real‑time data lake ingestion, and demonstrates AutoMQ’s Kafka‑to‑Iceberg Table Topic solution that simplifies schema management, reduces latency, and cuts operational costs.

Apache IcebergAutoMQCloud Native
0 likes · 14 min read
Leveraging Apache Iceberg and AutoMQ for Real-Time Data Lake Ingestion: Architecture, Best Practices, and Cost Optimization
Python Programming Learning Circle
Python Programming Learning Circle
Feb 28, 2025 · Fundamentals

Techniques for Efficient Large File Processing in Python

Processing large files efficiently in Python requires techniques such as line-by-line iteration, chunked reads, generators, buffered I/O, and streaming, which help avoid memory errors, improve speed, and optimize resources for tasks like log analysis, data scraping, and real-time API handling.

File I/OLarge Filesperformance
0 likes · 5 min read
Techniques for Efficient Large File Processing in Python
Cognitive Technology Team
Cognitive Technology Team
Feb 28, 2025 · Artificial Intelligence

Design and High‑Availability Architecture of Alibaba LangEngine AI Application Framework

This article introduces Alibaba's LangEngine, a pure Java AI application framework, detailing its high‑availability gateway architecture, communication protocols, streaming and non‑streaming output, multi‑level metadata caching, asynchronous and serverless designs, and future open‑source roadmap, offering practical guidance for building robust AI services.

AI frameworkAsynchronous DesignHigh Availability
0 likes · 11 min read
Design and High‑Availability Architecture of Alibaba LangEngine AI Application Framework
Top Architect
Top Architect
Feb 21, 2025 · Artificial Intelligence

DeepSeek4j 1.4: Java Integration Framework for DeepSeek with Full Chain‑of‑Thought and Streaming Support

The article introduces DeepSeek4j 1.4, a Java‑based framework that overcomes Spring AI’s limitations by fully preserving DeepSeek’s chain‑of‑thought and billing features, adding reactive streaming, providing Spring Boot starter integration, and offering quick‑start code samples and configuration guidance.

AIChain-of-ThoughtDeepSeek
0 likes · 8 min read
DeepSeek4j 1.4: Java Integration Framework for DeepSeek with Full Chain‑of‑Thought and Streaming Support
Java Architecture Diary
Java Architecture Diary
Feb 10, 2025 · Artificial Intelligence

deepseek4j 1.3: Java SDK adds web search, streaming & multi‑channel AI

deepseek4j 1.3 introduces web‑search capability, streaming responses, system prompts, expanded multi‑platform support, enhanced SSE debugging, and upcoming features like API‑key rotation and resilience, enabling Java developers to integrate DeepSeek models effortlessly while focusing on business logic.

AIDeepSeekSDK
0 likes · 8 min read
deepseek4j 1.3: Java SDK adds web search, streaming & multi‑channel AI
macrozheng
macrozheng
Jan 24, 2025 · Backend Development

Boost Java Excel Performance with FastExcel: Features, Usage, and Comparison

This article introduces FastExcel, an upgraded Java library for high‑performance Excel read/write, outlines its key features, provides step‑by‑step code examples for entity creation, event listeners, writing, reading, PDF conversion, compares it with EasyExcel, and concludes with its suitability for large‑scale data processing.

ExcelFastExcelPDF
0 likes · 8 min read
Boost Java Excel Performance with FastExcel: Features, Usage, and Comparison
Alibaba Cloud Big Data AI Platform
Alibaba Cloud Big Data AI Platform
Jan 14, 2025 · Big Data

How Fluss Unifies Lake and Stream for Real‑Time Analytics: Architecture, Benefits, and Future Roadmap

This article summarizes a talk by Alibaba Cloud senior engineer and Flink Committer Luo Yuxia on the challenges of separating lake and stream storage, introduces the Fluss lake‑stream unified architecture, explains its technical benefits such as second‑level data freshness, unified metadata, efficient changelog generation, and outlines future plans for broader ecosystem integration.

FlinkFlussdata lake
0 likes · 13 min read
How Fluss Unifies Lake and Stream for Real‑Time Analytics: Architecture, Benefits, and Future Roadmap
Ctrip Technology
Ctrip Technology
Jan 3, 2025 · Big Data

Design and Implementation of a Kafka Gatekeeper for FinOps Billing Data Quality Governance

This article describes the challenges of data quality in Ctrip’s hybrid‑cloud FinOps billing system and presents the design, implementation, and high‑availability deployment of a custom Kafka Gatekeeper proxy that performs pre‑validation, configurable rules, self‑service dashboards, and automated alerts to improve coverage, timeliness, and responsibility attribution.

Cloud NativeFinOpsKafka
0 likes · 17 min read
Design and Implementation of a Kafka Gatekeeper for FinOps Billing Data Quality Governance
Big Data Technology & Architecture
Big Data Technology & Architecture
Jan 2, 2025 · Big Data

Apache Paimon: Core Capabilities, Table Types, LSM Tree, Buckets, Merge Engines, and Operational Details

This article provides a comprehensive overview of Apache Paimon, covering its real‑time lake ingestion, unified stream‑batch processing, table types (primary‑key and append‑only), LSM‑tree storage, bucket mechanisms, merge‑engine options, compaction strategies, concurrency control, consumption methods, tag management, data cleanup, and system tables for big‑data workloads.

Apache PaimonFlinkLSM-tree
0 likes · 25 min read
Apache Paimon: Core Capabilities, Table Types, LSM Tree, Buckets, Merge Engines, and Operational Details
Zhihu Tech Column
Zhihu Tech Column
Dec 31, 2024 · Cloud Native

Cloud Native Innovation Forum: AutoMQ Table Topic, OceanBase Integrated Database, and Observability Practices

The article recaps Zhihu's Cloud Native Innovation Forum where experts from AutoMQ, OceanBase, and Flashcat shared practical solutions on streaming data ingestion, unified database architectures, and AI‑driven observability, highlighting real‑world deployments, performance optimizations, and cost‑saving strategies.

AIAutoMQCloud Native
0 likes · 10 min read
Cloud Native Innovation Forum: AutoMQ Table Topic, OceanBase Integrated Database, and Observability Practices
Architect
Architect
Dec 15, 2024 · Databases

Efficient MySQL Queries for Millions of Rows: Regular, Stream, and Cursor

When processing massive MySQL result sets, loading all rows into JVM memory can cause OOM and slow performance, so this guide compares three approaches—regular pagination, streaming queries using server-side cursors, and cursor‑based fetchSize control—detailing their implementations, MyBatis configurations, and trade‑offs.

CursorLarge DataMyBatis
0 likes · 10 min read
Efficient MySQL Queries for Millions of Rows: Regular, Stream, and Cursor
MaGe Linux Operations
MaGe Linux Operations
Dec 14, 2024 · Big Data

Master Kafka: From Core Concepts to Real-World Deployment

This comprehensive guide explains Kafka’s architecture, core APIs, topics and partitions, deployment steps, multi‑broker clustering, and practical use cases such as messaging, log aggregation, stream processing, and data import/export with Kafka Connect, providing a hands‑on tutorial for developers and engineers.

Distributed SystemsKafkaTutorial
0 likes · 30 min read
Master Kafka: From Core Concepts to Real-World Deployment
Tencent Advertising Technology
Tencent Advertising Technology
Dec 6, 2024 · Big Data

Building a High‑Performance Advertising Feature Data Lake with Apache Iceberg at Tencent

Tencent's advertising team replaced a traditional HDFS‑Hive warehouse with an Apache Iceberg‑based data lake, adding primary‑key tables, multi‑stream merging, adaptive compaction, and Spark SPJ optimizations to achieve minute‑level feature update latency, 10× back‑fill speed, and up to 60% storage savings.

CDCIcebergbig data
0 likes · 25 min read
Building a High‑Performance Advertising Feature Data Lake with Apache Iceberg at Tencent
Bilibili Tech
Bilibili Tech
Nov 26, 2024 · Big Data

Bilibili’s Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practices

Bilibili migrated its massive user‑behavior, commercial AI training, and database synchronization pipelines from Hive and Kafka to an Iceberg‑based streaming‑batch architecture, using Flink and the Magnus optimizer to achieve minute‑level freshness, reduce CPU and memory usage by about 20‑22 %, save roughly 3.55 M CNY annually, and dramatically improve query latency and join performance.

FlinkIcebergOptimization
0 likes · 20 min read
Bilibili’s Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practices
DataFunSummit
DataFunSummit
Nov 23, 2024 · Big Data

Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice

This article presents Bilibili's end‑to‑end exploration of a streaming‑batch unified data pipeline built on Apache Iceberg, detailing the original and iterated architectures for massive user behavior transmission, online AI training, DB synchronization, and dimension‑join, along with performance gains, cost savings, and future plans.

FlinkIcebergOptimization
0 likes · 20 min read
Bilibili's Iceberg‑Based Streaming‑Batch Integration: Architecture, Optimizations, and Practice
CSS Magic
CSS Magic
Nov 8, 2024 · Artificial Intelligence

LLM Application Development Tips (3): Exploring LLM API Inputs and Outputs

This article explains how to configure key OpenAI chat completion parameters—such as temperature, top_p, streaming, response format, and tool selection—and walks through the structure of the API's JSON response, highlighting fields like id, model, choices, finish_reason, and usage for better control and cost estimation.

AI agentsAPI parametersJSON response
0 likes · 8 min read
LLM Application Development Tips (3): Exploring LLM API Inputs and Outputs
360 Zhihui Cloud Developer
360 Zhihui Cloud Developer
Oct 31, 2024 · Backend Development

Boosting Ozone Block Reads with gRPC Streaming: Up to 30% Faster

This article explains how a gRPC bidirectional streaming read method was added to Ozone to reduce chunk‑by‑chunk request gaps, describes the client‑side implementation, presents single‑ and multi‑threaded performance tests showing roughly 30% faster reads, and outlines future enhancements such as pre‑fetching.

Ozoneblock storagegRPC
0 likes · 7 min read
Boosting Ozone Block Reads with gRPC Streaming: Up to 30% Faster
DataFunSummit
DataFunSummit
Sep 30, 2024 · Big Data

Apache Hudi from Zero to One: The Swiss Army Knife for Data Ingestion – Hudi Streamer (Part 9)

This article introduces Apache Hudi Streamer, a versatile Spark‑based data ingestion tool likened to a Swiss Army knife, detailing its core options—including table configuration, continuous mode, source classes, transformers, table services, catalog synchronization, and advanced features—while guiding users on practical pipeline setup.

Apache HudiData IngestionSpark
0 likes · 10 min read
Apache Hudi from Zero to One: The Swiss Army Knife for Data Ingestion – Hudi Streamer (Part 9)
JD Retail Technology
JD Retail Technology
Sep 3, 2024 · Backend Development

Design and Architecture of a New Video Review System with Streamlined Frame Extraction and Parallel Processing

This article presents the design goals, architecture, technology selection, and component details of a unified video review system that leverages FFmpeg for frame extraction, stream‑based parallel processing, and flexible synchronous/asynchronous workflows to achieve low latency and high scalability.

Content moderationFFmpegParallel Processing
0 likes · 10 min read
Design and Architecture of a New Video Review System with Streamlined Frame Extraction and Parallel Processing
Java Tech Enthusiast
Java Tech Enthusiast
Sep 2, 2024 · Industry Insights

Why Major Pirate Streaming Sites Are Closing: Industry Trends and Copyright Crackdowns

A wave of shutdowns affecting popular free video and anime piracy platforms such as RARBG and Animeflix reveals how pandemic costs, legal pressures, court rulings, and coordinated anti‑piracy actions by industry alliances are reshaping the digital media landscape and pushing users toward legitimate services.

copyright enforcementdigital mediaindustry trends
0 likes · 7 min read
Why Major Pirate Streaming Sites Are Closing: Industry Trends and Copyright Crackdowns
Big Data Technology & Architecture
Big Data Technology & Architecture
Aug 20, 2024 · Big Data

Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios

This article shares a personal, experience‑driven overview of Apache Paimon, highlighting its design simplicity, key capabilities such as schema evolution, stream‑batch unified processing, primary‑key support, and closed‑loop data handling, while discussing when its features are appropriate for production environments.

Apache PaimonSchema Evolutionbatch processing
0 likes · 5 min read
Practical Insights on Using Apache Paimon for Real-World Data Lake Scenarios
DeWu Technology
DeWu Technology
Jul 31, 2024 · Big Data

Custom Flink Scheduler Enhancements: Resource Balancing, Task Migration, and TmRestart Strategy

The article details Dewu’s custom Flink scheduler, DwScheduler, which adds JSON‑based resource specifications, per‑TaskManager slot sharing for balanced CPU use, hot TaskManager migration callbacks, and a new TmRestart strategy for rapid pod‑process recovery, offering practical techniques to enhance real‑time stream processing stability and performance.

Apache FlinkPerformance OptimizationTask Migration
0 likes · 9 min read
Custom Flink Scheduler Enhancements: Resource Balancing, Task Migration, and TmRestart Strategy
Soul Technical Team
Soul Technical Team
Jul 23, 2024 · Big Data

Kafka Stability Challenges and Governance Framework at Soul

This article analyzes the role, application scenarios, stability challenges, and comprehensive governance framework of Apache Kafka at Soul, covering deployment, configuration, monitoring, standard controls, common misuse, and future directions toward cloud‑native solutions.

Kafkamonitoringoperations
0 likes · 30 min read
Kafka Stability Challenges and Governance Framework at Soul
Rare Earth Juejin Tech Community
Rare Earth Juejin Tech Community
Jul 22, 2024 · Big Data

Comprehensive Guide to Kafka: Architecture, Core Concepts, and Configuration

This article provides an in‑depth overview of Apache Kafka, covering its use cases, comparison with other message queues, versioning, performance mechanisms, core concepts such as topics, partitions, offsets, consumer groups, rebalancing, replication, leader election, idempotence, transactions, compression, interceptors, request handling, and practical configuration tips for reliable streaming applications.

ConsumerKafkaMessage Queue
0 likes · 25 min read
Comprehensive Guide to Kafka: Architecture, Core Concepts, and Configuration
Tencent Cloud Developer
Tencent Cloud Developer
Jul 16, 2024 · Big Data

In‑Depth Exploration of Apache Kafka: Architecture, High Reliability, and High Performance

Apache Kafka achieves high‑throughput, fault‑tolerant messaging by combining a partitioned log architecture with leader‑follower replication, asynchronous producer pipelines, configurable acknowledgments, page‑cache‑based sequential writes, zero‑copy transfers, batching, compression, and a multi‑reactor network model that together ensure scalability, reliability, and performance.

Apache Kafkareliabilitystreaming
0 likes · 30 min read
In‑Depth Exploration of Apache Kafka: Architecture, High Reliability, and High Performance
Tencent Cloud Developer
Tencent Cloud Developer
Jul 2, 2024 · Big Data

Apache Flink Deployment with Pulsar Connector: Setup, Demos, and Best Practices

This guide shows how to deploy Apache Flink 1.17 in Docker, configure off‑heap memory, connect it to Pulsar via the 4.1.0‑1.17 connector, run example jobs that copy topics and perform windowed word‑count, and provides Maven dependencies, custom serialization tips, batching settings, and version‑specific best‑practice notes.

Apache FlinkDataStreamDocker deployment
0 likes · 20 min read
Apache Flink Deployment with Pulsar Connector: Setup, Demos, and Best Practices
Code Mala Tang
Code Mala Tang
Jun 29, 2024 · Frontend Development

Master WritableStream: Real-World Uses, Best Practices, and Common Pitfalls

This article introduces the JavaScript WritableStream API, explains its core methods and construction, demonstrates practical scenarios such as file uploads, logging, data transformation, and media handling, and discusses advanced considerations like chunk sizing, error recovery, concurrency control, and performance optimization.

Web APIWritableStreamstreaming
0 likes · 10 min read
Master WritableStream: Real-World Uses, Best Practices, and Common Pitfalls
Code Mala Tang
Code Mala Tang
Jun 27, 2024 · Frontend Development

Mastering ReadableStream: A Deep Dive into Web Streams API

This article introduces the concept of streams, explains the Web Streams API and its ReadableStream component, details constructors, methods, queuing strategies, back‑pressure handling, BYOB and byte streams, and provides practical code examples and usage scenarios for modern web development.

Front-endReadableStreamWeb Streams API
0 likes · 20 min read
Mastering ReadableStream: A Deep Dive into Web Streams API
DataFunTalk
DataFunTalk
Jun 18, 2024 · Big Data

Real-time Data Warehouse Evolution with Data Lake: Architecture, Challenges, and Solutions

This article presents a comprehensive overview of the evolution from traditional Lambda‑based real‑time data warehouse solutions to a data‑lake‑integrated architecture, detailing the shortcomings of legacy designs, the iterative improvements made at JD Technology, and the technical and operational challenges encountered during implementation.

Lambda architecturearchitecturedata lake
0 likes · 24 min read
Real-time Data Warehouse Evolution with Data Lake: Architecture, Challenges, and Solutions
Big Data Technology & Architecture
Big Data Technology & Architecture
Jun 16, 2024 · Big Data

Real-time Big Data Analytics with Apache Paimon and the Streaming Lakehouse Architecture

This article summarizes Wang Feng's presentation on the next‑generation Lakehouse architecture, explaining how Apache Paimon provides a unified, real‑time data lake format that bridges batch and streaming workloads, enabling low‑latency analytics and AI integration for modern big‑data applications.

Apache Paimonbig datareal-time analytics
0 likes · 9 min read
Real-time Big Data Analytics with Apache Paimon and the Streaming Lakehouse Architecture
Sohu Tech Products
Sohu Tech Products
Jun 5, 2024 · Big Data

Why Kafka Is the Backbone of Modern Data Pipelines: Core Architecture and Use Cases

This article explains Kafka's role as a high‑throughput distributed message queue, detailing its core components, topic‑partition model, consumer groups, storage mechanisms, fault‑tolerance features, delivery guarantees, ZooKeeper coordination, and scalability strategies for building reliable real‑time data pipelines.

Distributed SystemsKafkaMessage Queue
0 likes · 14 min read
Why Kafka Is the Backbone of Modern Data Pipelines: Core Architecture and Use Cases
Su San Talks Tech
Su San Talks Tech
Jun 2, 2024 · Big Data

Mastering Kafka: Core Architecture, Use Cases, and Design Principles

This article provides a comprehensive overview of Apache Kafka, covering its role as a message queue, core components, topic and partition design, consumer groups, storage mechanisms, high‑availability features, delivery guarantees, ZooKeeper coordination, and scalability strategies for building robust real‑time data pipelines.

Kafkabig datastreaming
0 likes · 15 min read
Mastering Kafka: Core Architecture, Use Cases, and Design Principles
DataFunTalk
DataFunTalk
May 16, 2024 · Big Data

Streaming Data Lake Warehouse Solution Based on USDP with Flink and Paimon

This article presents UCloud's USDP‑based streaming data lake warehouse solution that leverages Flink for real‑time processing and Paimon for lake storage, detailing its architecture, advantages, practical scenarios, and providing complete SQL and Flink CDC code snippets for end‑to‑end implementation.

CDCFlinkPaimon
0 likes · 27 min read
Streaming Data Lake Warehouse Solution Based on USDP with Flink and Paimon
Sohu Tech Products
Sohu Tech Products
May 15, 2024 · Artificial Intelligence

OpenAI Assistants API Quickstart Project for Next.js

OpenAI’s open‑source openai‑assistants‑quickstart project shows how to integrate the Assistants API into a Next.js app, offering streaming chat, code‑interpreter, file‑search, and function‑calling tools, and provides step‑by‑step setup instructions so developers can quickly build and customize AI assistants.

AI assistantAssistants APICode interpreter
0 likes · 4 min read
OpenAI Assistants API Quickstart Project for Next.js
Mike Chen's Internet Architecture
Mike Chen's Internet Architecture
May 11, 2024 · Big Data

Comprehensive Introduction to Apache Kafka: Architecture, Features, and Use Cases

This article provides a detailed overview of Apache Kafka, covering its core characteristics, distributed architecture, key components such as topics, partitions, brokers, producers, consumers, ZooKeeper, and common application scenarios like log collection, event‑driven architecture, real‑time analytics, and monitoring.

Distributed SystemsKafkaMessage Queue
0 likes · 7 min read
Comprehensive Introduction to Apache Kafka: Architecture, Features, and Use Cases
Big Data Technology & Architecture
Big Data Technology & Architecture
Apr 30, 2024 · Big Data

Apache Paimon Becomes a Top-Level Project: A Comprehensive Overview of Lakehouse Framework Capabilities and Future Trends

The article reviews Apache Paimon's graduation to an Apache Top-Level Project, outlines the essential capabilities of modern lakehouse frameworks—including streaming and batch I/O, multi‑engine integration, and advanced features—and discusses the problems they solve and the promising direction of the lakehouse ecosystem.

Apache PaimonReal-time Data Lakebatch processing
0 likes · 5 min read
Apache Paimon Becomes a Top-Level Project: A Comprehensive Overview of Lakehouse Framework Capabilities and Future Trends
Bilibili Tech
Bilibili Tech
Apr 26, 2024 · Artificial Intelligence

2024 Bilibili Technology Patent Awards – Highlights of Ten Winning Innovations

On World Intellectual Property Day, Bilibili honored ten breakthrough patents that together enable billion‑scale video duplicate detection, AI‑driven story generation, synchronized live rhythm‑games, automatic OTT casting, knowledge‑graph‑based content moderation, glitch‑free multi‑audio streaming, modular playback integration, neural‑network resolution encoding, AV1 reference‑frame pruning, and fine‑grained GPU isolation.

artificial intelligenceencodinginteractive media
0 likes · 6 min read
2024 Bilibili Technology Patent Awards – Highlights of Ten Winning Innovations
21CTO
21CTO
Apr 22, 2024 · Big Data

Inside Uber’s Real‑Time Data Infrastructure: How They Scale Streaming at Massive Scale

This article explores Uber’s sophisticated real‑time data infrastructure, detailing how the company leverages open‑source technologies such as Apache Kafka, Flink, Pinot, and Presto, and describing the architectural components, scaling challenges, multi‑region resilience, data back‑filling, and operational practices that enable low‑latency analytics for millions of daily rides and deliveries.

FlinkKafkaPinot
0 likes · 25 min read
Inside Uber’s Real‑Time Data Infrastructure: How They Scale Streaming at Massive Scale
Bilibili Tech
Bilibili Tech
Apr 12, 2024 · Backend Development

Design and Optimization of a High‑Throughput Long‑Connection Service for Live Streaming

The article details a Golang‑based high‑throughput long‑connection service for live‑streaming, describing its five‑layer architecture, multi‑protocol support, load‑balancing, message‑queue decoupling, aggregation with brotli compression, multi‑region deployment, priority channels, and future enhancements for observability and intelligent endpoint selection.

Backend ArchitectureLong Connectiongolang
0 likes · 16 min read
Design and Optimization of a High‑Throughput Long‑Connection Service for Live Streaming
Bilibili Tech
Bilibili Tech
Apr 9, 2024 · Big Data

Optimizing Flink State Performance with RocksDB KV Separation and BlobDB

In large‑scale Flink double‑stream joins, terabyte‑sized RocksDB state caused severe compaction latency and CPU spikes, but enabling RocksDB BlobDB KV‑separation (and an inner‑compaction patch) dramatically shrank SST files, reduced read/write latencies to sub‑millisecond levels, and cut CPU spikes by about half.

FlinkKV SeparationPerformance Optimization
0 likes · 12 min read
Optimizing Flink State Performance with RocksDB KV Separation and BlobDB
DataFunSummit
DataFunSummit
Apr 7, 2024 · Big Data

Li Auto’s Flink on Kubernetes Data Integration Practice

This article presents Li Auto’s end‑to‑end data integration journey, detailing the evolution of its data platform, the challenges of heterogeneous sources, and how a unified Flink‑on‑K8s solution with cloud‑native architecture, operator management, monitoring, and checkpointing addresses batch‑stream convergence and future scalability.

FlinkKubernetesbatch processing
0 likes · 12 min read
Li Auto’s Flink on Kubernetes Data Integration Practice
Ctrip Technology
Ctrip Technology
Mar 22, 2024 · Mobile Development

Design and Implementation of the Cloud Touch Platform for Remote Mobile Device Control and Testing

The article presents the background, full‑scenario construction, core architecture, device‑pool strategy, remote iOS control via WebDriverAgent, screen‑sync using ffmpeg, streaming pipeline, data collection, and practical lessons of the Cloud Touch platform that enables unified remote testing and customer‑support workflows for mobile applications.

Cloud TouchFFmpegRemote Device Control
0 likes · 14 min read
Design and Implementation of the Cloud Touch Platform for Remote Mobile Device Control and Testing
Didi Tech
Didi Tech
Mar 12, 2024 · Big Data

Understanding Flink Metrics System: Core Concepts, Elastic Design, and Practical Usage

The article explains Flink’s metrics architecture—core concepts, reporter interfaces, built‑in and custom metric types, elastic plugin design, and scheduled reporting—illustrated with a consumption‑latency example, and shows how Didi uses these metrics for real‑time UI curves, alerts, and intelligent task diagnosis.

FlinkMetricsbig data
0 likes · 11 min read
Understanding Flink Metrics System: Core Concepts, Elastic Design, and Practical Usage
Architect's Guide
Architect's Guide
Mar 2, 2024 · Fundamentals

RabbitMQ vs Kafka: Core Differences and When to Use Each

This article compares RabbitMQ and Apache Kafka across architecture, message ordering, routing, timing, retention, fault handling, scalability, and consumer complexity, and provides guidance on which platform suits specific use‑cases such as flexible routing, strict ordering, long‑term retention, or high throughput.

Fault ToleranceKafkaMessage Queue
0 likes · 19 min read
RabbitMQ vs Kafka: Core Differences and When to Use Each
Airbnb Technology Team
Airbnb Technology Team
Mar 1, 2024 · Big Data

Riverbed: A Scalable Data Framework for Real‑time and Batch Processing at Airbnb

Airbnb’s Riverbed framework unifies streaming CDC events and batch Spark jobs behind a GraphQL‑based declarative API to automatically build and maintain distributed materialized views, using Kafka‑partitioned ordering and version control to deliver billions of daily updates with low‑latency reads for features such as payments and search.

AirbnbApache SparkData Engineering
0 likes · 8 min read
Riverbed: A Scalable Data Framework for Real‑time and Batch Processing at Airbnb
MaGe Linux Operations
MaGe Linux Operations
Feb 20, 2024 · Big Data

Redis Streams vs Kafka: Which Is Better for Real‑Time Event Processing?

This article compares Redis Streams and Kafka, examining their architectures, ordering guarantees, consumer group models, scalability, and trade‑offs, and shows how Redis can emulate Kafka‑like semantics using the Runnel library, while highlighting memory‑speed benefits versus Kafka’s durable, unlimited log storage.

Event ProcessingKafkaRedis
0 likes · 9 min read
Redis Streams vs Kafka: Which Is Better for Real‑Time Event Processing?
Open Source Tech Hub
Open Source Tech Hub
Jan 31, 2024 · Artificial Intelligence

How to Build Async OpenAI PHP Clients with Workerman & Webman

This guide shows how to install the OpenAI PHP async client and implement streaming and non‑streaming chat, image generation, audio speech, and embedding features using Workerman and Webman, including Azure OpenAI support, with complete code examples.

APIOpenAIPHP
0 likes · 6 min read
How to Build Async OpenAI PHP Clients with Workerman & Webman
StarRocks
StarRocks
Jan 30, 2024 · Big Data

How InLong Guarantees Exactly‑Once Real‑Time Writes to StarRocks

This article explains how Apache InLong provides automatic, secure, high‑performance real‑time data transfer to StarRocks, detailing the transactional Stream Load API, the two‑phase commit process, Flink‑based ingestion architecture, exactly‑once guarantees, and performance test results across different parallelism levels.

InLongStarRocksbig data
0 likes · 11 min read
How InLong Guarantees Exactly‑Once Real‑Time Writes to StarRocks