Implementing an Automated Minesweeper Solver with Python and OpenCV

The article demonstrates how to create an automated Minesweeper solver using Python, OpenCV, win32gui, and PIL, achieving record‑breaking speed on the classic Windows game.

Preparation : Install Python 3.6+ (Anaconda recommended), numpy, Pillow, opencv‑python, and the pywin32 packages (win32gui, win32api). You also need the Minesweeper Arbiter executable, which the script will control.

0. Window Capture : Locate the game window with its class name "TMain" and title "Minesweeper Arbiter " (note the trailing space), then retrieve its screen rectangle.

<code>hwnd = win32gui.FindWindow(class_name, title_name)
if hwnd:
    left, top, right, bottom = win32gui.GetWindowRect(hwnd)</code>

These coordinates are later adjusted to isolate the board area.

1. Board Cropping : Use ImageGrab.grab() from Pillow and apply offset “magic numbers” to obtain the exact board rectangle.

<code>left += 15
top += 101
right -= 15
bottom -= 43
rect = (left, top, right, bottom)
img = ImageGrab.grab().crop(rect)</code>

2. Cell Segmentation : Each Minesweeper cell is 16 × 16 px. Compute the number of columns and rows, then crop each cell.

<code>block_width, block_height = 16, 16
blocks_x = int((right - left) / block_width)
blocks_y = int((bottom - top) / block_height)</code>

<code>def crop_block(hole_img, x, y):
    x1, y1 = x * block_width, y * block_height
    x2, y2 = x1 + block_width, y1 + block_height
    return hole_img.crop((x1, y1, x2, y2))</code>

All cells are stored in a two‑dimensional list blocks_img .

3. Cell Recognition : Convert each cropped image to an OpenCV array and examine the centre pixel colour to decide the cell type. The function analyze_block maps colours to numbers, mines, flags, unopened, blank, or unknown states.

<code>def analyze_block(self, block, location):
    block = imageProcess.pil_to_cv(block)
    block_color = block[8, 8]
    x, y = location[0], location[1]
    # -1: Not opened, -2: Opened but blank, -3: Uninitialized
    if self.equal(block_color, self.rgb_to_bgr((192,192,192))):
        if not self.equal(block[8,1], self.rgb_to_bgr((255,255,255))):
            self.blocks_num[x][y] = -2
        else:
            self.blocks_num[x][y] = -1
    elif self.equal(block_color, self.rgb_to_bgr((0,0,255))):
        self.blocks_num[x][y] = 1
    # … (continues for colours 0,128,0 → 2, 255,0,0 → 3, etc.)
    else:
        self.blocks_num[x][y] = -3</code>

The mapping used is:

1‑8 → numbers 1‑8

9 → mine

0 → flag

-1 → unopened

-2 → opened but blank

-3 → not a Minesweeper cell

4. Solving Algorithm : For each numbered cell, generate a 3 × 3 kernel, trim it at board edges, and collect neighbour coordinates.

<code>def generate_kernel(k, k_width, k_height, block_location):
    ls = []
    loc_x, loc_y = block_location[0], block_location[1]
    for now_y in range(k_height):
        for now_x in range(k_width):
            if k[now_y][now_x]:
                rel_x, rel_y = now_x - 1, now_y - 1
                ls.append((loc_y + rel_y, loc_x + rel_x))
    return ls

kernel = [[1,1,1],[1,1,1],[1,1,1]]
kernel_width, kernel_height = 3, 3
# Adjust kernel borders when x or y is at the edge (set corresponding entries to 0)</code>

Count unopened neighbours and compare with the cell’s number:

<code>def count_unopen_blocks(blocks):
    count = 0
    for single_block in blocks:
        if self.blocks_num[single_block[1]][single_block[0]] == -1:
            count += 1
    return count

def mark_as_mine(blocks):
    for single_block in blocks:
        if self.blocks_num[single_block[1]][single_block[0]] == -1:
            self.blocks_is_mine[single_block[1]][single_block[0]] = 1

unopen_blocks = count_unopen_blocks(to_visit)
if unopen_blocks == self.blocks_num[x][y]:
    mark_as_mine(to_visit)</code>

If the numbers match, all neighbours are marked as mines; otherwise safe neighbours are added to next_steps for clicking.

<code>def mark_to_click_block(blocks):
    for single_block in blocks:
        if not self.blocks_is_mine[single_block[1]][single_block[0]] == 1:
            if self.blocks_num[single_block[1]][single_block[0]] == -1:
                if not (single_block[1], single_block[0]) in self.next_steps:
                    self.next_steps.append((single_block[1], single_block[0]))
</code>

The main loop iterates over all cells, applies the analysis, marks mines, determines safe clicks, and finally moves the mouse and issues a click using win32api calls.

<code>self.iterate_blocks_image(BoomMine.analyze_block)
self.iterate_blocks_number(BoomMine.detect_mine)
self.iterate_blocks_number(BoomMine.detect_to_click_block)
if self.is_in_form(mouseOperation.get_mouse_point()):
    for to_click in self.next_steps:
        on_screen_location = self.rel_loc_to_real(to_click)
        mouseOperation.mouse_move(on_screen_location[0], on_screen_location[1])
        mouseOperation.mouse_click()
</code>

All these steps are encapsulated in a helper module (e.g., imageProcess.py ) so the solver can be invoked with a single call to get_frame() . When the cursor is inside the board, the program automatically recognises the state and performs the appropriate clicks.