Tagged articles
checkpoint
100 articles · Page 1 of 1
Backstage 1.55 adds task recovery for Scaffolder, but recovery only resumes from saved step state and workspace snapshots; external operations can still repeat if they succeed before checkpoint persistence, requiring action authors to implement idempotency and resource verification.
BackstageIdempotencyScaffolder
0 likes · 14 min read

This article explains exactly-once processing semantics in data engineering, contrasting it with at-most-once and at-least-once, detailing why retries cause duplicates, the roles of checkpoints, idempotency, and transactions, why technical exactly-once doesn't guarantee business correctness, and the four critical failure tests needed before production deployment.
Data ConsistencyIdempotencycheckpoint
0 likes · 16 min read

This tutorial teaches how to build stateful AI agents using LangGraph, covering core abstractions (StateGraph, nodes, edges, checkpoints), adding long-term memory with thread IDs, integrating tools like Tavily search, customizing state with reducers, debugging with state history, and choosing production-ready checkpoint backends.
AI agentsLangGraphLong-term Memory
0 likes · 18 min read

This article systematically covers 10 essential LangGraph interview questions, explaining core concepts like State, Node, Edge, Checkpoint, and Human-in-the-loop with code examples, and compares LangGraph with AutoGen and CrewAI for enterprise AI agent orchestration.
AI AgentAutoGenCrewAI
0 likes · 16 min read

This article provides a deep technical analysis of Apache Flink's checkpoint mechanism, covering barrier alignment and unaligned snapshots, heap vs RocksDB state backends with incremental checkpoints, savepoint vs checkpoint differences, failure recovery flows, and practical troubleshooting for checkpoint timeouts, state bloat, and recovery slowness.
Apache FlinkBarrier AlignmentIncremental Checkpoint
0 likes · 26 min read


ITPUBSep 7, 2026 · Databases
This article explains InnoDB's checkpoint mechanism, contrasting synchronous and fuzzy checkpoints, detailing the pre-MySQL 8.0 fuzzy checkpoint implementation using flush lists and page cleaners, and describing two principles for detecting abnormal shutdowns plus four post-restart crash scenarios with their recovery requirements.
Crash RecoveryInnoDBLSN
0 likes · 11 min read

This article analyzes Pi Agent's built-in context compression mechanism—detailing its trigger, cutoff, structured summary fields, and relay design—and introduces a custom Handoff skill that replicates the summary structure to create persistent checkpoints for seamless cross-session and cross-agent task handoff, enabling continuous long-running agent workflows.
AI AgentHandoff skillPi agent
0 likes · 10 min read

This article details how to build a production-ready real-time data warehouse using Flink, covering CDC ingestion, event-time processing, dimension joins, upsert aggregation, state TTL, checkpoint tuning, exactly-once verification, and a pre-launch checklist for reliability.
CDCDimension JoinEvent Time
0 likes · 22 min read

The article examines how to construct a container‑based sandbox infrastructure for multi‑agent AI systems, covering isolation mechanisms, lifecycle management, resource scaling, checkpoint/commit techniques, the OpenKruise Agents project, ecosystem integration, and production case studies with performance metrics.
AI agentsContainer SandboxKubernetes
0 likes · 16 min read

When migrating tens of billions of rows, the article explains how a simple progress‑tracking table and failure‑log table enable automatic checkpoint‑based resumption, stateless execution, and dynamic batch tuning without restarting or rewriting code.
MySQLNacosbatch processing
0 likes · 5 min read

This guide details how to achieve end-to-end exactly-once semantics with Apache Flink and Apache Paimon, covering consistency models, three-layer coordination mechanisms, production-grade configurations, concurrency rules, verification tests, failure injection scenarios, performance trade-offs, and operational SOPs for lakehouse streaming workloads.
Apache FlinkApache PaimonProduction Operations
0 likes · 26 min read

The article explains why AI pipelines often freeze due to external rate limiting or data corruption, and presents a three‑step checkpoint and rollback protocol plus partial‑retry routing that cuts full rerun time from hours to minutes, reduces compute waste by 85% and dramatically improves reliability.
AIRetryautomation
0 likes · 7 min read

In a 90‑minute technical livestream, three experts dissect ten core challenges of bringing AI agents from demo to production, covering execution control, sandbox versus permission boundaries, checkpoint design, rollback strategies, tool‑call safety, human‑in‑the‑loop interaction, multi‑agent coordination, observability, and memory management.
Agent EngineeringObservabilitycheckpoint
0 likes · 17 min read

In a 90‑minute live technical session, three experts dissect ten core challenges of Agent engineering—sandbox vs permission boundaries, checkpoints, rollback, tool‑call safety, human‑in‑the‑loop, multi‑agent coordination, observability, and memory—showing that moving agents from "usable" to "trustworthy" requires fine‑grained execution controls rather than broader permissions.
Agent EngineeringObservabilitycheckpoint
0 likes · 18 min read

The live discussion explores how to move agents from demo to production by establishing execution controls, safety boundaries, checkpoints, rollback mechanisms, tool‑call auditing, human‑in‑the‑loop handling, multi‑agent coordination, observability, and memory management, forming a comprehensive harness engineering framework.
Agent EngineeringPermission Boundarycheckpoint
0 likes · 15 min read

The article breaks down the unique hard and soft failure modes of LLM‑driven agents and proposes a four‑layer defense—LLM call handling, tool execution isolation, execution‑chain checkpointing, and semantic‑level safeguards—plus observability practices to keep production agents stable and reliable.
AgentLLMLangfuse
0 likes · 15 min read

The article details ten production‑grade pitfalls encountered when using LangGraph—ranging from misusing thread IDs and unbounded state growth to uncaught tool errors, infinite loops, concurrency conflicts, subgraph field mismatches, HITL timeouts, and misconfigured LangSmith tracing—each illustrated with concrete code, root‑cause analysis, and concrete remediation steps.
AI agentsLLMLangGraph
0 likes · 14 min read

This article explains LangGraph's checkpoint persistence, detailing its data structure, the role of thread_id for multi‑session isolation, the three available checkpointer backends, and how to use checkpoints for conversation memory, resumable workflows, and manual state updates, while highlighting common pitfalls.
LangGraphMemorySaverPostgresSaver
0 likes · 9 min read

This article explains why subgraphs are needed for complex tasks, shows how to define a subgraph as a node, use named channels for data flow, isolate namespaces, reuse compiled subgraphs, add checkpoints, and presents a complete multi‑agent orchestration example in Python using LangGraph.
LangGraphNamed ChannelsPython
0 likes · 6 min read

The article explains LangGraph’s checkpoint system that lets agents retain dialogue context, detailing the InMemorySaver for development and PostgresSaver for production, how to use checkpointer.put/get, thread_id for session isolation, manual state manipulation, and time‑travel replay, with full Python examples.
Agent MemoryInMemorySaverLangGraph
0 likes · 5 min read

This article explains how LangGraph’s graph‑based StateGraph lets you model agent workflows visually, contrasting it with LangChain’s high‑level API, detailing the three core components, showing complete Python examples, and highlighting benefits such as easier debugging, extensibility, and checkpoint support.
AgentGraphLangGraph
0 likes · 7 min read

LoongCollector introduces a One‑Time file collection mode that scans matching files once, records a snapshot, and exits, enabling efficient historic log migration, data back‑fill, and temporary batch processing while providing fine‑grained checkpoints, execution windows, and throttling controls to avoid quota issues and ensure reliable completion.
Log Collectioncheckpointdata migration
0 likes · 12 min read

LoongCollector’s OneTime file collection feature enables fast, reliable migration of historical logs, data back‑filling, and batch processing by scanning files once, using checkpoints for fault tolerance, configurable execution windows, and rate‑limiting to avoid impacting live data streams.
Log CollectionOneTimecheckpoint
0 likes · 12 min read

This article presents a production‑grade Flink ClickHouse sink that solves common pain points such as lack of size‑based batching, static table schemas, and distributed‑table latency by introducing data‑size batching, dynamic table routing, local‑table writes, load‑balanced node discovery, back‑pressure queues, dual‑trigger flush, and recursive retry with node exclusion, all integrated with Flink checkpoint semantics for at‑least‑once guarantees.
ClickHouseDynamic ShardingFlink
0 likes · 25 min read


Java OneJan 24, 2026 · Artificial Intelligence
This guide explains Claude Code’s agent loop, model choices, built‑in tool categories, project access scope, session handling, checkpoint and permission controls, and practical tips for efficiently using the AI‑driven terminal assistant to write, test, and refactor code.
AI coding assistantClaude Codeagent loop
0 likes · 15 min read

This article walks through three advanced LangGraph 1.0 features—using the Command object for conditional routing, checkpoint‑based memory for state persistence across invocations, and interrupt‑driven human‑in‑the‑loop control—providing concrete code examples, execution traces, and a comparison of design trade‑offs.
AI agentsCommandLangGraph
0 likes · 15 min read

This guide walks through adding the Elasticsearch sink to a Flink job, configuring connection and bulk settings, implementing custom failure handlers, and enabling checkpoint‑based retry mechanisms to ensure reliable streaming writes.
ElasticsearchFlinkJava
0 likes · 9 min read

This article details practical Spark performance optimizations—including RDD reuse, broadcast variables, Kryo serialization, parallelism settings, shuffle tuning, and JVM tweaks—while also presenting systematic solutions for data skew, shuffle failures, serialization errors, and YARN mode issues, all illustrated with concrete code snippets and examples.
BroadcastData SkewJVM
0 likes · 25 min read

Learn the six typical scenarios that cause data loss when writing to Paimon—ranging from checkpoint failures and misconfigured partial‑update mode to incorrect sequence fields, snapshot retention issues, concurrent bucket writes, and outdated Spark versions—and how to prevent each problem.
FlinkPaimonSpark
0 likes · 5 min read

This article explains how to re‑implement ByteDance's DeerFlow deep‑research framework in Go (Deer‑Go), covering the multi‑agent architecture, control‑hand‑off, interrupt & checkpoint mechanisms, integration with the Hertz SSE server, and step‑by‑step deployment instructions.
DeerFlowEinoGo
0 likes · 16 min read

This article explains Apache Flink's checkpoint and savepoint mechanisms, detailing their core differences, ideal use cases for automatic failure recovery versus manual structural changes, and key operational considerations for stateful stream processing applications.
FlinkRecoverySavepoint
0 likes · 8 min read

This article explains how os‑criu provides transparent, OS‑level GPU checkpoint/restore, compares its performance with NVIDIA's cuda‑checkpoint, walks through building and installing the PhOS framework, demonstrates migration of a Llama2‑13b‑chat workload in Docker, and discusses current limitations and future Kubernetes integration plans.
CRIUDockerGPU
0 likes · 9 min read

An in‑depth guide explains MySQL’s write‑ahead log mechanism, covering buffer pool, redo and undo logs, checkpoint types, and how the system recovers from power failures, with step‑by‑step examples and practical configuration tips for reliable data consistency.
MySQLWALcheckpoint
0 likes · 12 min read


ITPUBMay 6, 2024 · Databases
This article explains MySQL InnoDB’s write‑ahead logging, detailing the roles of Buffer Pool, Redo and Undo logs, checkpoint mechanisms, and how they ensure data consistency and atomicity when a sudden power loss occurs.
InnoDBMySQLWAL
0 likes · 12 min read

The article reviews Apache Flink's rapid development over the past decade, highlighting its 2023 community growth, SIGMOD award, major releases, streaming SQL enhancements, incremental checkpointing, batch maturity, cloud‑native scaling, and integration with the emerging Lakehouse architecture.
Apache FlinkSQLbig data
0 likes · 11 min read

This article explains how to enable and use the OpenJDK CRaC project with SpringBoot 3.2, covering prerequisites, dependency setup, automatic and manual checkpoint creation, and demonstrates significant startup‑time reductions through detailed examples and performance results.
BackendCRaCJava
0 likes · 9 min read

This guide explains how Spring Boot 3.2 leverages the OpenJDK CRaC project to create and restore JVM checkpoints, dramatically reducing application startup time through automatic and manual checkpoint techniques, complete with required dependencies, installation steps, and performance results.
BackendCRaCJava
0 likes · 8 min read

This article demonstrates how to enable and use Coordinated Restore at Checkpoint (CRaC) in Spring Boot 3.2 with the Spring Petclinic example, covering required dependencies, JVM setup, automatic and manual checkpoint creation, performance measurements, and best‑practice considerations for backend developers.
BackendCRaCJava
0 likes · 8 min read

This article demonstrates how Spring Boot 3.2 leverages the OpenJDK CRaC project to dramatically cut application startup time, covering required JVM, dependencies, Zulu JDK setup, automatic and manual checkpoint creation, performance benchmarks, and restoration steps, all using the Petclinic example.
CRaCcheckpointperformance
0 likes · 8 min read

When training large‑batch deep learning models, developers often use recompute to trade computation for memory, but in dynamic graph frameworks this can trigger synchronization errors in distributed data parallel training; the article explains the underlying DDP mechanics, illustrates the error, and offers a practical no_sync workaround with code examples.
PyTorchcheckpointdistributed training
0 likes · 14 min read

The article reviews Apache Flink 1.17's major batch and streaming improvements, including new Delete/Update APIs, performance boosts, SQL client gateway, checkpoint and watermark enhancements, StateBackend upgrades, and practical use‑case scenarios for data engineers.
Apache FlinkSQLStateBackend
0 likes · 7 min read


ITPUBMar 24, 2023 · Big Data
Apache Flink 1.17 introduces a suite of batch and streaming enhancements—including a new Streaming Warehouse API, significant TPC‑DS performance boosts, adaptive batch scheduling, improved checkpointing, expanded SQL capabilities, Hive connector upgrades, and broader filesystem support—while also delivering upgrades to FRocksDB, Calcite, and the token framework to strengthen its position as a leading unified data‑processing engine.
Apache FlinkRocksDBSQL
0 likes · 23 min read

This article compiles frequent Flink job submission problems on YARN—including WordCount jar errors, HBase dependency conflicts, MySQL timeout, checkpoint restoration failures, parallelism limits, and unexpected container termination—provides root‑cause analysis and step‑by‑step remediation instructions.
FlinkHBaseMySQL
0 likes · 21 min read

An incident report details how a scheduled machine reboot on Alibaba Cloud triggered a Flink TaskManager failover, leading to excessive data replay, increased ES pressure, and significant business latency, and explains the root cause involving disabled checkpoints and timestamp‑based offset consumption.
FailoverFlinkKafka
0 likes · 10 min read

Flink 1.16 introduces adaptive batch scheduling, speculative execution, hybrid shuffle, dynamic partition pruning, improved Hive SQL compatibility, advanced checkpoint mechanisms including changelog backend, and integrates CDC with Kafka and Table Store, offering faster, more stable, and easier-to-use stream‑batch processing capabilities.
CDCFlinkHive SQL
0 likes · 8 min read


ITPUBDec 21, 2022 · Big Data
This article details Bilibili's extensive enhancements to the Flink runtime—including checkpoint recoverability, max‑parallelism calculations, State Processor API extensions, Full and Regional Checkpoints, hybrid HA, task‑level recovery, load‑balanced partitioners, and large‑scale cluster maintenance—to improve reliability and performance of its billion‑scale streaming workloads.
FlinkHigh Availabilitybig data
0 likes · 33 min read


政采云技术Dec 8, 2022 · Big Data
This article explains the Asynchronous Barrier Snapshotting algorithm used by Apache Flink for checkpointing, detailing its origins from the Chandy‑Lamport algorithm, its operation in both acyclic and cyclic dataflow graphs, barrier alignment, and the fault‑recovery process.
Asynchronous Barrier SnapshottingFlinkcheckpoint
0 likes · 10 min read

This article details Bilibili's extensive enhancements to Flink's runtime—including checkpoint recoverability, operator ID stability, state processor extensions, hybrid high‑availability, regional checkpointing, and load‑based channel selection—to improve scalability, reliability, and operational efficiency of large‑scale streaming jobs.
FlinkHigh AvailabilityRuntime Optimization
0 likes · 32 min read

This article presents a detailed guide for optimizing large‑scale Apache Flink streaming jobs on the JD Real‑Time Computing platform, covering TaskManager memory model tuning, network stack configuration, RocksDB state management, checkpoint strategies, and additional performance tips with practical examples and calculations.
Apache FlinkPerformance TuningRocksDB
0 likes · 22 min read

This article explains how Neutrino’s SerializableProvider API enables passing final classes, managing mutable object state, and supporting Spark checkpoint recovery through dependency injection, while also showing practical code patterns and injection of core Spark components.
NeutrinoSerializationSpark
0 likes · 8 min read

The article explains how Spark Streaming's checkpoint mechanism works, detailing the four-step process—from setting the checkpoint directory to writing RDD data and finalizing the checkpoint—highlighting its role in ensuring fault‑tolerant, fast recovery for real‑time recommendation feature pipelines.
Spark Streamingbig datacheckpoint
0 likes · 7 min read

This article explains how Apache Flink achieves end‑to‑end exactly‑once semantics by using source replay support, checkpoint‑based snapshots, asynchronous incremental checkpoints, and two‑phase commit sinks, and describes the interaction with external systems such as Kafka to ensure transactional writes.
FlinkKafkaTwo-Phase Commit
0 likes · 7 min read

This article provides an extensive overview of Apache Flink, covering deployment modes, cluster sizing, job submission workflows, state programming concepts, checkpoint mechanisms, backpressure handling, comparison with Spark, and practical code snippets for configuration and optimization.
FlinkJob DeploymentState Management
0 likes · 48 min read

This article explains Flink's fundamental checkpoint mechanism, its coupling with backpressure, and how the introduction of Unaligned Checkpoint in Flink 1.11 decouples checkpointing from backpressure to improve latency and resource utilization in high‑backpressure streaming jobs.
FlinkUnaligned Checkpointbackpressure
0 likes · 14 min read

This article compiles a comprehensive list of common production problems encountered with Apache Flink, covering cluster sizing, checkpoint failures, backpressure analysis, resource allocation, deployment errors, UDF definitions, data skew, Kafka configurations, and provides detailed troubleshooting steps and best‑practice recommendations.
Apache FlinkKafkaUDF
0 likes · 39 min read

This article details ByteDance's practical experience with Apache Flink, covering SQL extensions, a visual SQL platform, performance tweaks such as window mini‑batching and custom windows, join and checkpoint recovery improvements, stream‑batch integration experiments, and future roadmap plans.
Batch IntegrationFlinkSQL
0 likes · 16 min read


政采云技术Nov 30, 2021 · Databases
This article provides a comprehensive overview of MySQL, detailing its configuration file search order, component architecture, various storage engines such as MyISAM, NDB, Memory, and an in‑depth examination of InnoDB’s internal structures, memory management, background threads, LRU handling, redo log buffering, and checkpoint mechanisms.
Database ArchitectureInnoDBMySQL
0 likes · 26 min read

This article provides an extensive technical guide to Apache Flink, covering its exactly‑once consumption guarantees, checkpoint and two‑phase commit mechanisms, differences from Spark, state backends, watermark handling, time semantics, window joins, CEP, backpressure, architecture layers, deployment, resource management, and common operational issues.
CEPFlinkWatermark
0 likes · 77 min read

This article provides a comprehensive guide on optimizing Flink checkpoints, diagnosing and alleviating backpressure, and fine‑tuning memory configurations—including process, heap, off‑heap, managed, and network memory—to improve job stability and performance in large‑scale streaming applications.
FlinkMemory TuningStateBackend
0 likes · 25 min read

The article systematically introduces Apache Flink’s streaming computation model, contrasting batch and real‑time processing, detailing its unified architecture, managed and raw state with key groups, checkpointing and savepoints for fault tolerance, data exchange mechanisms, time semantics, windowing, side‑outputs, and a complete Java Kafka‑based example.
Apache FlinkFlink ArchitectureKeyed State
0 likes · 46 min read

This article provides a comprehensive overview of Apache Flink's major releases from version 1.7 to 1.14, detailing new APIs, state management improvements, Kubernetes integration, SQL and Table API enhancements, checkpointing advances, and performance optimizations that together illustrate the platform's evolution for both streaming and batch processing workloads.
Apache FlinkKubernetesPython
0 likes · 78 min read

This article details Bilibili's comprehensive optimization of Flink real‑time computing, covering connector stability improvements, SQL interval‑join enhancements, runtime state and checkpoint refinements, a diagnostic tool, and future directions for high‑throughput streaming workloads.
FlinkReal-time Computingbig data
0 likes · 18 min read

This article explains the meaning of the 'Pages flushed up to' value in MySQL's InnoDB engine, how it differs from the 'Last checkpoint' LSN, the underlying logic for its calculation, and provides test results demonstrating its behavior during writes and checkpoints.
InnoDBLSNMySQL
0 likes · 6 min read

This article explains the internal mechanisms of relational databases, covering data pages, buffer pools, undo and redo logs, checkpointing, and how these components work together to provide atomicity, durability, and crash recovery while minimizing disk I/O.
Transaction Logscheckpointdata pages
0 likes · 7 min read

This article reviews the Pravega project and its Flink connector, covering Pravega's design for large‑scale streaming, the connector's evolution and exact‑once semantics, Flink 1.11 integration challenges, checkpoint mechanisms, and future plans such as schema‑registry and new Flink features.
ConnectorFlinkPravega
0 likes · 10 min read

This article summarizes frequent problems encountered when using Flink CDC with MySQL—including Kafka version conflicts, checkpoint timeouts, permission errors, global lock issues, and DDL parsing failures—and provides practical configuration tweaks and code examples to resolve them.
CDCDebeziumFlink
0 likes · 11 min read

This article compiles classic production problems encountered with Apache Flink, covering cluster sizing, checkpoint failures, backpressure diagnosis, client submission errors, resource allocation on YARN, and PyFlink UDF definitions, providing step‑by‑step troubleshooting methods and practical recommendations.
Flinkbackpressurecheckpoint
0 likes · 18 min read

This article explains Spark's cache and checkpoint mechanisms, detailing when to use each, how they are implemented internally, how cached and checkpointed RDDs are stored and retrieved, and the differences between caching, persisting, and checkpointing for reliable big‑data processing.
CacheRDDSpark
0 likes · 13 min read

This guide details comprehensive Flink performance tuning techniques, covering memory configuration, GC settings, parallelism adjustments, process parameters, partitioning strategies, Netty network tuning, checkpoint optimization, and common issues such as data skew and resource bottlenecks.
FlinkParallelismPerformance Tuning
0 likes · 18 min read

This article presents ByteDance's recent Flink enhancements, detailing a single‑point recovery mechanism for the network layer and a regional checkpoint strategy that together improve failover latency, reduce output loss, and enable scalable, high‑throughput stream processing for large‑scale real‑time recommendation workloads.
FailoverFlinkRecovery
0 likes · 12 min read

This article details common Flink streaming problems such as data skew causing task back‑pressure, oversized Kafka messages, high‑throughput ack settings, slot removal errors, checkpoint timeouts, and resource constraints, and provides concrete configuration changes and architectural adjustments to resolve them.
Data SkewFlinkcheckpoint
0 likes · 18 min read

This article presents a detailed collection of 2021 Flink interview questions covering checkpoint mechanisms, watermarks, state backends, join types, fault tolerance, resource configuration, and recent Flink 1.10 features, providing concise explanations and code examples for each topic.
FlinkInterviewState Backend
0 likes · 23 min read

This article details a production‑grade pipeline that uses Apache Flink 1.11's SQL CDC to stream MySQL changes into Elasticsearch, enabling low‑latency OLAP reporting, and shares the architecture, DDL/DML scripts, operational settings, and dozens of pitfalls encountered along the way.
big-datacheckpointsql-cdc
0 likes · 19 min read

This article explores the concurrency control strategies, persistence mechanisms, and compiled query processing techniques used by modern memory databases, comparing systems like Hekaton, Hyper, HANA, and others while highlighting performance trade‑offs and architectural considerations.
MVCCcheckpointconcurrency control
0 likes · 20 min read

Burst Buffer is a storage acceleration technology that enhances I/O bandwidth and OPS for high‑performance computing by providing fast checkpoint/restart, temporary storage, and balancing SSD and parallel file system resources, with implementations from DDN, Cray, EMC, and IBM detailed for HPC designers.
Burst BufferHPCSSD
0 likes · 5 min read

Apache Flink 1.11.0, released after four months of development, brings major ecosystem, usability, and stability improvements—including CDC support, a new JDBC catalog, real‑time Hive integration, a redesigned source API, PyFlink enhancements, application mode for Kubernetes, and checkpoint optimizations—while highlighting the growing contribution of Chinese developers.
Apache FlinkFeature ReleaseKubernetes
0 likes · 20 min read

This article explains Flink's checkpoint mechanism, outlines key performance metrics, discusses interval configuration, external state storage choices, resource allocation, and task-local recovery strategies to improve checkpoint speed and reliability in large‑scale state scenarios.
FlinkPerformance TuningState Backend
0 likes · 5 min read

This article provides a detailed examination of Elasticsearch's distributed consistency mechanisms, covering the shard write path, the PacificA replication algorithm, the role of SequenceNumber and Checkpoint, and a comparison of ES's implementation with the original algorithm, based on version 6.2.
BackendDistributed ConsistencyElasticsearch
0 likes · 23 min read

This article explains the role of HDFS SecondaryNameNode, the structure of fsimage and edits files, how checkpointing works—including configuration parameters and steps—and how the process changes when NameNode high availability is enabled.
FilesystemHDFSHadoop
0 likes · 6 min read

This article enumerates typical Apache Flink deployment, job, and checkpoint errors—such as JDK version issues, resource shortages, task manager timeouts, and state migration problems—and provides practical troubleshooting steps and configuration tips to help engineers quickly diagnose and fix these failures.
ExceptionFlinkYARN
0 likes · 8 min read

The article thoroughly explains Apache Flink’s checkpoint mechanism—including state types, coordinator workflow, exactly‑once versus at‑least‑once semantics, common failure sources such as code exceptions, storage or network issues, and practical configuration tips like interval settings, local recovery and externalized checkpoints.
Apache FlinkReal-time ComputingState Management
0 likes · 15 min read

This article explains a small‑file‑merging technique for Apache Flink checkpoints that reuses FSDataOutputStreams to combine multiple state files into a single HDFS file, detailing design considerations such as concurrent checkpoint support, reference‑counted deletion, space amplification reduction, fault handling, compatibility, and observed production performance gains.
Apache FlinkHDFSState Backend
0 likes · 13 min read

This article explains how MySQL's Multi‑Threaded Slave (MTS) processes events, manages checkpoints, and persists state using GAQ queues, bitmaps, and system tables, providing detailed code references and configuration parameters for reliable parallel replication.
GTIDMTSMulti-threaded
0 likes · 14 min read

This article explains the purpose and management of InnoDB REDO log files, the role of Log Sequence Numbers, the structure of log pages, the MTR physical transaction mechanism, and how checkpoints ensure data integrity in MySQL databases.
Database InternalsInnoDBMySQL
0 likes · 22 min read

This article explains why Spark checkpoints are needed for large or complex RDD pipelines, how they work by persisting data to reliable storage such as HDFS, and outlines practical steps and best‑practice recommendations for using checkpoints effectively in production environments.
HDFSRDDSpark
0 likes · 6 min read

This article details how Weibo’s advertising team designed and implemented a real‑time data platform capable of processing over a hundred billion daily logs, covering technology selection, Flink advantages, architecture evolution, data processing pipelines, component libraries, fault‑tolerance strategies, and the construction of a multi‑layer real‑time data warehouse.
Flinkbig datacheckpoint
0 likes · 25 min read

This article presents a comprehensive analysis of Spark PageRank performance, detailing the algorithm's basics, the original example code, and four key optimizations—caching with checkpointing, memory‑efficient data structures, handling data skew, and maximizing executor and driver resource usage—backed by experimental results and practical recommendations.
CacheData SkewPageRank
0 likes · 18 min read

This article explains how Flink checkpoints persist state, compares the three built‑in state backends (MemoryStateBackend, FsStateBackend, RocksDBStateBackend), discusses their configurations, advantages, limitations, and provides guidance on selecting the appropriate backend for different big‑data streaming scenarios.
FlinkMemoryStateBackendRocksDB
0 likes · 10 min read

This article explains Flink's checkpoint mechanism, its execution flow, common configuration options, and the benefits and considerations of incremental checkpoints using the RocksDB state backend, providing practical code examples and YAML settings for reliable stream processing.
FlinkRocksDBState Backend
0 likes · 12 min read

This article analyzes the difficulties of achieving exactly-once delivery in Apache Flink, explains the distinction between state and end‑to‑end exactly‑once, and details how Flink implements exactly‑once sinks using idempotent and transactional approaches, including a Bucketing File Sink example.
FlinkState ManagementTransactional Sink
0 likes · 12 min read

This guide walks you through enabling Docker's experimental mode, installing CRIU, building a simple Node container, creating checkpoints, and restoring containers both on the same host and on a different host, highlighting the prerequisites and limitations of live migration.
CRIUContainer MigrationLinux kernel
0 likes · 5 min read

This article explains how Apache Flink manages state, uses checkpointing for fault-tolerant recovery, and achieves exactly-once semantics when consuming Kafka streams by persisting offsets, describing the checkpoint mechanism, recovery process, and practical considerations for production deployments.
FlinkKafkaState Management
0 likes · 8 min read

Learn how to install CRIU on CentOS, create a simple Node.js program that prints its PID, capture its state with a checkpoint using CRIU, and then restore the process, demonstrating practical use cases like migration and fast startup.
LinuxNode.jscheckpoint
0 likes · 3 min read

This article introduces the design and implementation of a real‑time user behavior engineering platform at Qunar using Apache Flink, covering Flink's core characteristics, distributed runtime, DataStream programming model, fault‑tolerance, back‑pressure handling, event‑time processing, windowing, watermarks, and practical code examples for filtering, splitting, joining, and state management.
DataStreamEventTimeFlink
0 likes · 18 min read

This article explains how Alibaba adopted Apache Flink as a unified, low‑latency, high‑throughput big‑data engine, detailing its stream‑first design, state management, checkpointing, massive production deployment, community contributions, and upcoming plans for a unified API, SQL layer, broader language support, and AI integration.
AlibabaApache FlinkUnified Engine
0 likes · 13 min read

This article explains the two types of SSD power loss—normal and abnormal—detailing how SSDs preserve data, the role of capacitors and non‑volatile memory, and the checkpoint‑based strategies used to quickly reconstruct mapping tables after unexpected power interruptions.
Mapping TableNon-volatile MemorySSD
0 likes · 10 min read

This article explains InnoDB’s double‑write buffer in MySQL, compares its design and recovery handling with Oracle’s redo and control‑file mechanisms, discusses partial‑write issues, checkpoint strategies, performance impacts on SSDs, and provides practical commands and configuration tips for DBAs.
InnoDBMySQLOracle
0 likes · 21 min read

This article provides practical guidance on data cleaning, covering the importance of data wrangling, using assertions, handling incomplete records, checkpointing, testing on subsets, logging, optional raw data storage, and validating the cleaned dataset to ensure reliable downstream analysis.
Loggingassertionscheckpoint
0 likes · 7 min read


ITPUBAug 11, 2016 · Databases
This article explains Oracle’s checkpoint mechanism, contrasting full and incremental checkpoints, describing the checkpoint queue structure, the roles of DBWn and CKPT processes, recovery using redo logs, and how to tune checkpoint frequency with the fast_start_mttr_target parameter while monitoring relevant performance views.
OracleRecoverycheckpoint
0 likes · 14 min read

This article explains the design, configuration parameters, core data structures, initialization, coordinator distribution, worker execution, checkpointing, and shutdown procedures of MySQL 5.6's Multi‑Threaded Slave (MTS) parallel replication, providing a code‑level walkthrough for developers and DBAs.
Database ReplicationMTSMySQL
0 likes · 17 min read
