Pipeline Architecture

DataFunSummit

Feb 17, 2025 · Artificial Intelligence

NorthStar Large‑Model Training Framework: Architecture, APIs, Pipeline and Multi‑GPU Strategies

The article introduces the NorthStar large‑model training framework developed by DeWu, detailing its background challenges, pipeline architecture, rich API support, multi‑GPU training modes, multi‑level embedding storage, hardware selection considerations, and a brief Q&A on data versus model parallelism.

AI FrameworkEmbedding StorageLarge Model Training

0 likes · 9 min read

Architect

Oct 18, 2024 · Operations

Design and Implementation of a Self‑Developed Video Transcoding Core Based on FFmpeg

This article describes the motivations, architecture, and key techniques of a custom video transcoding core built on FFmpeg, covering modular pipeline design, controllable serial/parallel execution, dynamic resolution and frame‑rate adaptation, SEI handling, and performance improvements for both live and on‑demand streaming.

FFmpegParallel ProcessingPipeline Architecture

0 likes · 11 min read

Design and Implementation of a Self‑Developed Video Transcoding Core Based on FFmpeg

NetEase Yanxuan Technology Product Team

Jan 9, 2023 · Operations

Loggie: A High-Performance Log Collection Agent System Design and Implementation

Loggie is a cloud-native, Go-based log-collection agent that replaces Filebeat and Flume by using a micro-kernel producer-consumer architecture with hot-swappable pipelines, achieving 2 GB/s read speeds, 1.6‑2.6× higher throughput while using only a quarter of the CPU, and providing built-in observability, reliability, and latency monitoring for large-scale enterprise deployments.

GoObservabilityPerformance Optimization

0 likes · 16 min read

Loggie: A High-Performance Log Collection Agent System Design and Implementation

Youku Technology

Nov 2, 2021 · Game Development

Building a High-Performance, High-Reusability, High-Reliability Audio Rendering Engine: Youku's Practice

Youku’s commercial‑grade audio rendering engine achieves high performance, reusability, and reliability by modularizing audio interfaces, post‑processing, output, caching, and focus management across multiple OSes, employing chain‑style pipelines, reactive filters, double‑linked buffer caching, latency monitoring, exception detection, and spatial‑audio filters for 5.1 surround sound.

C++ audio processingCross-Platform DevelopmentPerformance Optimization

0 likes · 11 min read

Building a High-Performance, High-Reusability, High-Reliability Audio Rendering Engine: Youku's Practice