Tagged articles

throughput optimization

6 articles · Page 1 of 1

Machine Learning Algorithms & Natural Language Processing

Oct 14, 2026 · Artificial Intelligence

TokenRouter: Efficient Token-Level LLM Routing via Async Handoffs & Delayed Batching

This article analyzes TokenRouter, a NeurIPS 2026 system that enables efficient token-level LLM routing through a request-centric programming interface, asynchronous model handoffs with state preservation, and throughput-optimized delayed batching, demonstrating 2-64x throughput gains over baselines across multiple routing algorithms and workloads.

LLM servingMarkov chain modelNeurIPS 2026

0 likes · 30 min read

TokenRouter: Efficient Token-Level LLM Routing via Async Handoffs & Delayed Batching

DeepHub IMBA

Aug 3, 2026 · Artificial Intelligence

How Continuous Batching Boosts LLM Throughput by Up to 36×

The article explains continuous batching, a scheduling technique that re‑evaluates the batch at every inference iteration, eliminating the straggler problem of static batching, and shows how it, together with PagedAttention, can raise LLM service throughput dramatically while balancing latency and memory usage.

GPU utilizationLLM inferencecontinuous batching

0 likes · 9 min read

How Continuous Batching Boosts LLM Throughput by Up to 36×

Random Bulletin

Jul 17, 2026 · Backend Development

Why Moving from Real‑Time to Batch Is Essential for Scaling to Tens of Millions QPS

Scaling a service from millions to tens of millions of queries per second fails not because of data size but due to per‑request fixed costs, and the article shows how batching aggregates these costs, dramatically boosts throughput, reduces latency, and introduces new challenges such as memory pressure and partial failures.

batch processinghigh QPSlatency tradeoff

0 likes · 17 min read

Why Moving from Real‑Time to Batch Is Essential for Scaling to Tens of Millions QPS

PaperAgent

Jun 29, 2026 · Artificial Intelligence

DeepSeek Opens DSpark: A New Speculative Decoding Framework for Large Language Models

DeepSeek releases DSpark, an open‑source speculative decoding system that combines semi‑autoregressive generation with confidence‑scheduled verification, delivering 60‑85% per‑user speed gains, lower latency, and superior acceptance rates compared with Eagle3 and DFlash across multiple LLM benchmarks.

Confidence SchedulingLLM inferenceSemi-Autoregressive Generation

0 likes · 14 min read

DeepSeek Opens DSpark: A New Speculative Decoding Framework for Large Language Models

DevOps Coach

Apr 26, 2026 · Backend Development

Forget Kafka: A Lightweight Go Queue Achieves 2 Million Messages per Second

The article analyzes how replacing Kafka with a simple in‑memory Go queue reduced architectural complexity, boosted throughput from 240‑330 K to 1.8‑2.0 M messages per second, and clarified debugging, while still acknowledging scenarios where Kafka remains the better choice.

Backend PerformanceGoIn‑Memory Ring Buffer

0 likes · 8 min read

Forget Kafka: A Lightweight Go Queue Achieves 2 Million Messages per Second

Alibaba Cloud Big Data AI Platform

Sep 17, 2024 · Artificial Intelligence

Boosting LLM Inference: How NanoFlow Doubles Throughput

The article introduces NanoFlow, a novel service framework that leverages intra‑device parallelism, operation‑based pipelining, and async scheduling to significantly improve large language model serving throughput, achieving up to 1.91× higher performance while integrating with Alibaba Cloud PAI.

Alibaba Cloud PAIGPU schedulingLLM serving

0 likes · 7 min read

Boosting LLM Inference: How NanoFlow Doubles Throughput