Tag

io-optimization

0 views collected around this technical thread.

DataFunSummit
DataFunSummit
Jun 9, 2024 · Cloud Native

Optimizing I/O for Data‑Intensive Analytics in Cloud‑Native Environments: Insights from Uber Presto

This whitepaper examines the industry trend of moving data‑intensive analytics workloads to cloud‑native platforms, analyzes how cloud storage cost models affect performance optimization, and presents a case study of Uber's Presto deployment that reveals fragmented access patterns and new I/O cost considerations.

Big DataCase Studycloud-native
0 likes · 3 min read
Optimizing I/O for Data‑Intensive Analytics in Cloud‑Native Environments: Insights from Uber Presto
政采云技术
政采云技术
Jul 14, 2022 · Operations

Diagnosing and Optimizing Elasticsearch IO Bottlenecks for Billion-Scale Product Catalogs

Facing severe IO-wait and read bottlenecks as product data grew from tens of millions to billions, this article analyzes root causes in Elasticsearch clusters and presents a comprehensive solution involving index parameter tuning, merge settings, translog async writes, query optimizations, and hardware upgrades to restore performance and stability.

Cluster OperationsElasticsearchIndex Tuning
0 likes · 14 min read
Diagnosing and Optimizing Elasticsearch IO Bottlenecks for Billion-Scale Product Catalogs
IT Architects Alliance
IT Architects Alliance
May 31, 2022 · Backend Development

Optimizing High‑Concurrency Services: Practical Strategies for QPS Over 200K

This article presents practical techniques for optimizing high‑concurrency online services—such as avoiding relational databases, employing multi‑level caching, leveraging multithreading, implementing circuit‑breaker patterns, reducing I/O, managing retries, handling edge cases, and logging efficiently—to maintain sub‑300 ms response times under massive load.

Cachingcircuit breakerhigh concurrency
0 likes · 10 min read
Optimizing High‑Concurrency Services: Practical Strategies for QPS Over 200K
Top Architect
Top Architect
May 30, 2022 · Backend Development

Key Practices for Optimizing High‑Concurrency Backend Services

This article outlines practical strategies for handling high‑QPS backend services, including avoiding relational databases, employing multi‑level caching, leveraging multithreading, implementing degradation and circuit‑breaker mechanisms, optimizing I/O, using cautious retries, handling edge cases, and logging efficiently.

Cachingbackend performancecircuit breaker
0 likes · 10 min read
Key Practices for Optimizing High‑Concurrency Backend Services
Top Architect
Top Architect
May 19, 2022 · Backend Development

Optimizing High‑Concurrency Services: Practical Strategies for 200k+ QPS

This article outlines practical techniques for handling ultra‑high‑traffic backend services—including abandoning relational databases, employing multi‑level caching, leveraging multithreading, applying degradation and circuit‑breaker patterns, optimizing I/O, using cautious retries, guarding boundary cases, and implementing efficient logging—to maintain sub‑300 ms response times at 200k+ QPS.

Cachingbackend performancecircuit breaker
0 likes · 10 min read
Optimizing High‑Concurrency Services: Practical Strategies for 200k+ QPS
Architecture Digest
Architecture Digest
May 17, 2022 · Backend Development

Optimizing High‑Concurrency Services: Practical Strategies for QPS Over 200k

This article outlines practical techniques for handling online services with QPS exceeding 200,000, including avoiding relational databases, employing multi‑level caching, leveraging multithreading, implementing degradation and circuit‑breaker patterns, optimizing I/O, using controlled retries, handling edge cases, and logging efficiently.

Cachingcircuit breakerhigh concurrency
0 likes · 9 min read
Optimizing High‑Concurrency Services: Practical Strategies for QPS Over 200k
IT Architects Alliance
IT Architects Alliance
May 16, 2022 · Backend Development

Optimizing High-Concurrency Services: Strategies for Handling Over 200k QPS

This article outlines practical techniques for optimizing high‑concurrency online services handling over 200 k QPS, covering the avoidance of relational databases, multi‑level caching, multithreading, circuit‑breaker and degradation strategies, I/O reduction, cautious retry policies, boundary checks, and efficient logging.

Cachingbackend performancecircuit breaker
0 likes · 10 min read
Optimizing High-Concurrency Services: Strategies for Handling Over 200k QPS
Architect
Architect
May 15, 2022 · Backend Development

Optimizing High‑Concurrency Services: Practical Strategies for QPS Over 200k

This article outlines practical techniques for handling ultra‑high‑traffic online services—such as avoiding relational databases, employing multi‑level caches, leveraging multithreading, applying degradation and circuit‑breaker patterns, optimizing I/O, using retries wisely, handling edge cases, and logging efficiently—to keep response times under 300 ms.

Cachingbackend performancecircuit breaker
0 likes · 9 min read
Optimizing High‑Concurrency Services: Practical Strategies for QPS Over 200k
Big Data Technology Architecture
Big Data Technology Architecture
Jul 20, 2021 · Big Data

PB‑Level Ad‑hoc Query Practice with Flink: Threat Hunting Platform Architecture and IO‑Reducing Optimizations

This article details 360's Threat Hunting platform built on Flink, covering its evolution, architecture, block‑index design, Hilbert‑curve data ordering, like‑pushdown, join optimizations, Alluxio caching, and future plans for BI and multi‑user concurrency, all aimed at efficient PB‑scale data querying.

AlluxioBig DataBlock Index
0 likes · 18 min read
PB‑Level Ad‑hoc Query Practice with Flink: Threat Hunting Platform Architecture and IO‑Reducing Optimizations
Aikesheng Open Source Community
Aikesheng Open Source Community
Jul 2, 2021 · Databases

Observing the Effects of MySQL Group Commit through Experiments

Through a series of experiments using MySQL 8.0, this article demonstrates how group commit reduces I/O operations by consolidating multiple transactions into a single commit group, showing that doubling load increases runtime modestly while transaction count and commit groups rise significantly, highlighting performance benefits.

MySQLgroup commitio-optimization
0 likes · 4 min read
Observing the Effects of MySQL Group Commit through Experiments
vivo Internet Technology
vivo Internet Technology
Aug 19, 2020 · Operations

Linux Page Cache Optimization for Kafka: Concepts, Parameter Tuning, and Performance Evaluation

The article explains Linux page cache fundamentals, shows how to inspect and reclaim cache, and provides detailed tuning of vm.dirty_* and vm.swappiness parameters to smooth Kafka write traffic, reduce I/O spikes, and improve overall performance, illustrated with before‑and‑after benchmarks.

KafkaLinuxPerformance Tuning
0 likes · 14 min read
Linux Page Cache Optimization for Kafka: Concepts, Parameter Tuning, and Performance Evaluation
Architecture Digest
Architecture Digest
Dec 15, 2019 · Databases

MongoDB High‑Throughput Cluster Optimization: Software, Configuration, and Storage Engine Tuning

This article details how a high‑traffic MongoDB sharded cluster exceeding one million TPS was optimized through software‑level tweaks, configuration changes, WiredTiger storage‑engine tuning, and hardware upgrades, resulting in latency reductions from hundreds of milliseconds to a few milliseconds and significantly improved stability.

Database TuningMongoDBSharding
0 likes · 33 min read
MongoDB High‑Throughput Cluster Optimization: Software, Configuration, and Storage Engine Tuning
58 Tech
58 Tech
Dec 2, 2019 · Databases

Optimizing RocksDB Compaction Rate Limiting to Reduce IO Spikes in WTable

This article analyzes RocksDB's compaction rate‑limiting source code and presents practical tuning methods—both fixed and auto‑tuned—to mitigate IO spikes in the distributed KV store WTable, improving real‑time read/write latency and stability.

CompactionDistributed KV StorePerformance Tuning
0 likes · 7 min read
Optimizing RocksDB Compaction Rate Limiting to Reduce IO Spikes in WTable
Architects' Tech Alliance
Architects' Tech Alliance
Nov 25, 2017 · Fundamentals

Deep Dive into IO Optimization and System Performance Tuning

This article explains how to evaluate storage performance by analyzing host ports, storage systems, and backend disks, discusses IO aggregation penalties, business workload models, RAID and cache impacts, and provides practical guidance for accurate performance assessment and tuning.

CachingIOPSRAID
0 likes · 16 min read
Deep Dive into IO Optimization and System Performance Tuning
Qunar Tech Salon
Qunar Tech Salon
May 25, 2015 · Operations

Improving Image Conversion Performance with /dev/shm tmpfs in Qunar's Group Buying Backend

The article describes how Qunar's group‑buying backend faced IO bottlenecks during large‑scale image conversion and solved the issue by moving intermediate files to a memory‑based tmpfs filesystem (/dev/shm), achieving a dramatic load reduction without code changes or server restarts.

Linuxbackendimage processing
0 likes · 5 min read
Improving Image Conversion Performance with /dev/shm tmpfs in Qunar's Group Buying Backend