How to Build a KNN-Based CAPTCHA Solver with OpenCV in Python

Preparation

Install the required libraries:

pip3 install opencv-python
pip3 install numpy

Recognition Principle

We use a supervised learning approach with the following steps:

Image processing – denoise and binarize the image.

Segmentation – cut the image into individual character images.

Manual labeling – label each character to build a training set.

Training – train a KNN model on the labeled data.

Detection – use the trained model to recognize new CAPTCHAs.

Image Processing

Read the image and convert it to grayscale:

import cv2
im = cv2.imread(filepath)
im_gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)

Binarize the image:

ret, im_inv = cv2.threshold(im_gray, 127, 255, cv2.THRESH_BINARY_INV)

Apply Gaussian blur for noise reduction:

kernel = 1/16 * np.array([[1,2,1],[2,4,2],[1,2,1]])
im_blur = cv2.filter2D(im_inv, -1, kernel)

Binarize again after denoising:

ret, im_res = cv2.threshold(im_blur, 127, 255, cv2.THRESH_BINARY)

Image Segmentation

Find contours and draw bounding boxes:

im2, contours, hierarchy = cv2.findContours(im_res, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

Handle cases where characters are merged by splitting the bounding boxes horizontally (2‑way, 3‑way, or 4‑way splits) using the width relationships of the contours. Example for a 2‑character merge:

result = []
for contour in contours:
    x, y, w, h = cv2.boundingRect(contour)
    if w == w_max:
        box_left = np.int0([[x,y], [x+w/2,y], [x+w/2,y+h], [x,y+h]])
        box_right = np.int0([[x+w/2,y], [x+w,y], [x+w,y+h], [x+w/2,y+h]])
        result.append(box_left)
        result.append(box_right)
    else:
        box = np.int0([[x,y], [x+w,y], [x+w,y+h], [x,y+h]])
        result.append(box)

For a 3‑character merge, split into three equal parts; for a 4‑character merge, split into four equal parts. The resulting boxes are stored in result.

Manual Annotation

Iterate over each cropped character image, display it, and record the pressed key as the label:

files = os.listdir("char")
for filename in files:
    filepath = os.path.join("char", filename)
    im = cv2.imread(filepath)
    cv2.imshow("image", im)
    key = cv2.waitKey(0)
    if key == 27:
        sys.exit()
    if key == 13:
        continue
    char = chr(key)
    outfile = f"{filename.split('.')[0]}_{char}.jpg"
    outpath = os.path.join("label", outfile)
    cv2.imwrite(outpath, im)

Approximately 800 character images are labeled and saved in the label directory.

Training Data

Load the labeled images, flatten each 30×30 image to a 900‑dimensional vector, and build the training matrix and label vector:

filenames = os.listdir("label")
samples = np.empty((0, 900))
labels = []
for filename in filenames:
    filepath = os.path.join("label", filename)
    label = filename.split(".")[0].split("_")[-1]
    labels.append(label)
    im = cv2.imread(filepath, cv2.IMREAD_GRAYSCALE)
    sample = im.reshape((1, 900)).astype(np.float32)
    samples = np.append(samples, sample, 0)
unique_labels = list(set(labels))
label_id_map = {l:i for i,l in enumerate(unique_labels)}
label_ids = np.array([label_id_map[l] for l in labels], dtype=np.float32).reshape(-1,1)

Train a KNN model:

model = cv2.ml.KNearest_create()
model.train(samples, cv2.ml.ROW_SAMPLE, label_ids)

Detection Result

For a new CAPTCHA, apply the same preprocessing and segmentation steps, then recognize each character:

for box in boxes:
    roi = im_res[box[0][1]:box[2][1], box[0][0]:box[1][0]]
    roistd = cv2.resize(roi, (30,30))
    sample = roistd.reshape((1,900)).astype(np.float32)
    ret, results, neighbours, distances = model.findNearest(sample, k=3)
    label_id = int(results[0,0])
    label = id_label_map[label_id]
    print(label)

The system correctly recognized the example CAPTCHA as yy4e, achieving roughly 82% accuracy on a hundred‑image test set.