Automating Minesweeper with Python and OpenCV: A Step‑by‑Step Tutorial

This article demonstrates how to create an automated Minesweeper player using Python and OpenCV, achieving record‑breaking speed by capturing the game window, segmenting the board into individual cells, recognizing each cell’s state, and applying a rule‑based solving algorithm.

Preparation : Install Python 3 (3.6+ recommended, Anaconda preferred) and the following libraries – numpy, Pillow (PIL), opencv‑python, pywin32 (win32gui, win32api) – plus any Python IDE. The target game must be Minesweeper Arbiter , which provides the window to control.

Window Capture : Identify the main window using its class and title, then obtain its rectangle and crop the board region.

<code>class_name = "TMain"
title_name = "Minesweeper Arbiter "
hwnd = win32gui.FindWindow(class_name, title_name)
if hwnd:
    left, top, right, bottom = win32gui.GetWindowRect(hwnd)
</code>

After adjusting for border offsets, capture the board with Pillow:

<code>left += 15
top += 101
right -= 15
bottom -= 43
rect = (left, top, right, bottom)
img = ImageGrab.grab().crop(rect)
</code>

Block Segmentation : Each cell is 16 × 16 px. Compute the number of cells horizontally and vertically, then crop each cell.

<code>block_width, block_height = 16, 16
blocks_x = int((right - left) / block_width)
blocks_y = int((bottom - top) / block_height)
</code>

<code>def crop_block(hole_img, x, y):
    x1, y1 = x * block_width, y * block_height
    x2, y2 = x1 + block_width, y1 + block_height
    return hole_img.crop((x1, y1, x2, y2))

blocks_img = [[0 for i in range(blocks_y)] for i in range(blocks_x)]
for y in range(blocks_y):
    for x in range(blocks_x):
        blocks_img[x][y] = crop_block(img, x, y)
</code>

Block Recognition : The centre pixel colour determines the cell type. The function below maps specific BGR colours to numbers, mines, flags, or unknown states.

<code>def analyze_block(self, block, location):
    block = imageProcess.pil_to_cv(block)
    block_color = block[8, 8]
    x, y = location[0], location[1]
    # -1: Not opened, -2: Opened but blank, -3: Uninitialized
    if self.equal(block_color, self.rgb_to_bgr((192, 192, 192))):
        if not self.equal(block[8, 1], self.rgb_to_bgr((255, 255, 255))):
            self.blocks_num[x][y] = -2
            self.is_started = True
        else:
            self.blocks_num[x][y] = -1
    elif self.equal(block_color, self.rgb_to_bgr((0, 0, 255))):
        self.blocks_num[x][y] = 1
    elif self.equal(block_color, self.rgb_to_bgr((0, 128, 0))):
        self.blocks_num[x][y] = 2
    elif self.equal(block_color, self.rgb_to_bgr((255, 0, 0))):
        self.blocks_num[x][y] = 3
    elif self.equal(block_color, self.rgb_to_bgr((0, 0, 128))):
        self.blocks_num[x][y] = 4
    elif self.equal(block_color, self.rgb_to_bgr((128, 0, 0))):
        self.blocks_num[x][y] = 5
    elif self.equal(block_color, self.rgb_to_bgr((0, 128, 128))):
        self.blocks_num[x][y] = 6
    elif self.equal(block_color, self.rgb_to_bgr((0, 0, 0))):
        if self.equal(block[6, 6], self.rgb_to_bgr((255, 255, 255))):
            self.blocks_num[x][y] = 9  # mine
        elif self.equal(block[5, 8], self.rgb_to_bgr((255, 0, 0))):
            self.blocks_num[x][y] = 0  # flag
        else:
            self.blocks_num[x][y] = 7
    elif self.equal(block_color, self.rgb_to_bgr((128, 128, 128))):
        self.blocks_num[x][y] = 8
    else:
        self.blocks_num[x][y] = -3
    self.is_mine_form = False
    if self.blocks_num[x][y] == -3 or not self.blocks_num[x][y] == -1:
        self.is_new_start = False
</code>

The mapping used is: 1‑8 for numbers, 9 for a mine, 0 for a flag, -1 for unopened, -2 for opened‑blank, -3 for unknown.

Sweeping Algorithm : The solver iterates over numbered cells and applies two rules:

If the count of adjacent unopened cells equals the cell’s number, all those cells are marked as mines.

If the number of already‑marked mines around a numbered cell equals its number, the remaining adjacent unopened cells are safe to click.

If neither rule yields a move, a random unopened cell is clicked (a simple fallback).

<code>def generate_kernel(k, k_width, k_height, block_location):
    ls = []
    loc_x, loc_y = block_location[0], block_location[1]
    for now_y in range(k_height):
        for now_x in range(k_width):
            if k[now_y][now_x]:
                rel_x, rel_y = now_x - 1, now_y - 1
                ls.append((loc_y + rel_y, loc_x + rel_x))
    return ls

kernel_width, kernel_height = 3, 3
kernel = [[1, 1, 1], [1, 1, 1], [1, 1, 1]]
# border adjustments omitted for brevity
</code>

<code>def count_unopen_blocks(blocks):
    count = 0
    for single_block in blocks:
        if self.blocks_num[single_block[1]][single_block[0]] == -1:
            count += 1
    return count

def mark_as_mine(blocks):
    for single_block in blocks:
        if self.blocks_num[single_block[1]][single_block[0]] == -1:
            self.blocks_is_mine[single_block[1]][single_block[0]] = 1
</code>

<code>def mark_to_click_block(blocks):
    for single_block in blocks:
        if not self.blocks_is_mine[single_block[1]][single_block[0]] == 1:
            if self.blocks_num[single_block[1]][single_block[0]] == -1:
                if not (single_block[1], single_block[0]) in self.next_steps:
                    self.next_steps.append((single_block[1], single_block[0]))
</code>

After determining the next safe cells, the script moves the mouse cursor to each cell’s screen coordinates and issues a click using win32api calls.

<code>if self.is_in_form(mouseOperation.get_mouse_point()):
    for to_click in self.next_steps:
        on_screen_location = self.rel_loc_to_real(to_click)
        mouseOperation.mouse_move(on_screen_location[0], on_screen_location[1])
        mouseOperation.mouse_click()
</code>

All major functionalities are encapsulated in separate modules (e.g., imageProcess.py for capture/segmentation and mouseOperation.py for mouse control), making the bot easy to extend or adapt to other Windows‑based games.