Optimizing OpenStack Swift for Small Object Storage in Video Streaming

This article discusses how iQiyi addressed performance degradation in OpenStack Swift caused by excessive small object storage. The core issue stems from Swift's file system architecture, where each object creates multiple directories and files, leading to high I/O overhead. The solution involves redesigning the storage engine to merge small objects into larger files while leveraging RocksDB for metadata management. Key optimizations include eliminating file ID mechanisms, using prefix scans for efficient object listing, and implementing lazy deletion for immediate space recovery. Benchmark results show significant performance improvements over the original Swift implementation.

The approach maintains Swift's decentralized architecture while addressing scalability challenges. The implementation achieved high performance with minimal code changes (around 3000 lines), demonstrating the effectiveness of targeted architectural improvements. Future work includes enhancing object expiration mechanisms and integrating Go-based components for further optimization.