How to Capture GTA V Game Frames in Python with OpenCV for AI Experiments

Original article: Reading game frames in Python with OpenCV - Python Plays GTA V (https://pythonprogramming.net/game-frames-open-cv-python-plays-gta-v/), author: Harrison.

After OpenAI's Universe was announced, many claimed that games like GTA V were ready for AI research, but GTA V was later removed without explanation. The author revisits the idea, noting that GTA V offers a richly detailed open world ideal for experiments such as autonomous driving because it allows control over time, weather, traffic, and emergencies via mods.

The goal is to build a simple self‑driving car prototype, which can be adapted to any game that provides roads and vehicles. The approach relies on reading screen images and simulating keyboard or controller input, which is sufficient for many AI training loops.

Find a reliable FPS tool to access the game screen.

Verify that keyboard commands can be sent to the game.

Test controller input, especially steering and braking.

Integrate OpenCV for image processing.

Implement autonomous driving in a simple road scenario.

Screen capture is performed with ImageGrab.grab(bbox=(0,40,800,640)) from the Pillow library, then converted to a NumPy array for OpenCV's imshow. Initial attempts were slow (2‑3 fps). After several optimisations, the loop runs in roughly 0.045‑0.060 seconds, yielding about 12‑13 fps, which is acceptable for early experiments.

loop took 0.05849909782409668 seconds
loop took 0.044053077697753906 seconds
loop took 0.04760456085205078 seconds
loop took 0.04805493354797363 seconds
loop took 0.05989837646484375 seconds

Images below illustrate the code evolution and performance measurements.

The translation of the original article also serves as a personal English practice; readers are invited to point out any mistakes.