AI-Powered Game Recognition for League of Legends Live Streaming on Bilibili

League of Legends (LoL) is one of the most popular esports titles, with a massive player base and a thriving competitive ecosystem. Since its launch in 2009, LoL has become the benchmark for esports and has generated a large industry around it.

With the rapid development of esports, the organization and viewability of tournaments have become key success factors. Bilibili’s live streaming platform has introduced automated and intelligent features to enhance the viewing experience, among which game recognition technology is a core component. This technology spans image processing, pattern recognition, and deep learning to extract key events such as hero picks, kills, and in‑game statistics in real time.

The system was first deployed in October 2021 for the S14 World Championship and has since been applied to major events including LPL, MSI, and the Saudi Cup. It provides high‑energy timestamps that allow viewers to jump to exciting moments via the progress bar.

Game Recognition Pipeline

The pipeline consists of three main modules:

1. Text Recognition (OCR) : OCR extracts on‑screen text such as team names, game time, kill counts, and economic data. By synthesizing over ten thousand training samples that mimic the LoL UI, the OCR model’s accuracy was improved from 90% to 99% with an inference time of 33 ms on a T4 GPU.

2. Hero Recognition : A custom object‑detection model identifies hero avatars displayed during kill events. Data augmentation (resolution changes, lighting variations, overlay effects) was used to create a robust dataset, achieving 97% accuracy and 8 ms inference on the same hardware.

3. Hot‑Spot Recognition : SEI (Supplemental Enhancement Information) tags are embedded in the video stream to mark interactive regions. When a viewer hovers over these regions, a floating panel shows real‑time player statistics, team economics, and hero development data. An IoU‑based verification step reduces false activations to 0.3% and eliminates mis‑detections.

Each module outputs structured data that is merged to form a coherent narrative of the match. The combined system not only generates high‑energy points on the timeline but also enables interactive data overlays, greatly enhancing immersion and engagement.

Results and Impact

The integrated solution has been successfully used for LoL and other titles such as Honor of Kings. Viewers can now replay specific moments via the high‑energy markers and explore detailed statistics through hot‑spot interactions. This multimodal approach improves both the spectator experience and the analytical capabilities of esports organizers.

Future work includes extending the technology to a broader range of esports titles and further refining multimodal scene classification to support richer interactive features.