Build a DIY AI Bot for Honor of Kings with Transformers, scrcpy & minitouch

Implementation Principle

The system controls the mobile game Honor of Kings from a PC by mirroring the Android device with scrcpy and sending multi‑touch gestures through minitouch . The workflow is: capture the game screen via scrcpy, take a screenshot, feed the image to a Transformer model that generates a textual description of the required actions, then translate that description into touch commands executed by minitouch.

Required Tools

scrcpy – mirrors Android screen to PC and allows mouse/keyboard control without rooting; USB debugging must be enabled.

minitouch – provides a socket interface for multi‑touch events on Android, also works without root.

Python environment with pyminitouch and torch (or compatible PyTorch version) to run the Transformer model.

Algorithm Overview

The core problem is converting a game screenshot (image) into a sequence of control commands (operations). A Transformer model, similar to image‑to‑text generators, is trained to output a textual description of the current game state (e.g., "move forward, attack, cast skill"). This description is parsed into concrete touch events that are sent through minitouch.

Setup and Testing

Ensure a GPU capable of running a Transformer model.

Install the required Python libraries (e.g., pyminitouch, torch).

Install scrcpy and place the executable in the same directory as the project code.

Download the pre‑trained model files from the provided link.

Adjust the minitouch configuration if your phone’s screen resolution differs from the test device (1080×2160).

Running the Bot

Execute 启动和结束进程.py to launch scrcpy. Edit 训练数据截取_A.py to set your device’s screen dimensions and other parameters, then run it while the game is in a 5v5 AI match. The script captures frames, generates action descriptions via the Transformer, and feeds the resulting commands to minitouch.

Project Resources

Code and model are hosted at: https://github.com/FengQuanLi/ResnetGPT

Model archive (Baidu Cloud): https://pan.baidu.com/s/19kjTuX4DeTVE3B3HGw0uOw (extraction code: jack)

Limitations and Future Work

The current implementation contains messy Chinese variable names and modest performance; it can handle basic AI battles but may struggle with more complex strategies. Future improvements include code refactoring, performance optimization, and extending support to additional heroes.