Can Palette-Based Recoloring Transform Pokémon Images Without Neural Networks?

Overview

This work presents a mathematically grounded pipeline for transferring the colour palette of one Pokémon sprite to another. The pipeline consists of (1) extracting a compact palette from a source image, (2) enumerating a dense space of colour transformations, and (3) selecting the transformation that minimises a perceptual distance between the transformed source and target images.

Palette Extraction

The palette is represented as P = {c_i, w_i}, where c_i are colour vectors and w_i their relative weights. Extraction follows the Blind Colour Separation (BCS) framework. BCS formulates palette extraction as a convex optimisation problem with a quadratic non‑negativity constraint. The optimisation is approximated by gradient descent on the following objective:

min_{C,W}\;\|X - CW\|_F^2 + \lambda\|W\|_1\quad\text{s.t.}\;C\ge 0,\;W\ge 0

where X is the image matrix, C the colour matrix, and W the weight matrix. The implementation uses a simple SGD loop with a learning rate of 1e-3 for 5000 iterations, yielding a concise palette of 5–10 colours for typical Pokémon sprites.

Colour‑Transfer Algorithm

Deep‑feature distance – Instead of raw pixel differences, the method computes a perceptual distance using a pretrained VGG‑19 network. For an image I, let Φ_l(I) denote the activation map of layer l. The distance between two images is defined as: d_{VGG}(I_a, I_b) = \sum_{l}\|Φ_l(I_a) - Φ_l(I_b)\|_2^2 Dense colour‑transform space – A set 𝒯 of admissible transformations is built by enumerating discrete shifts in HSV space:

𝒯 = {t_{h,b,s}\;|\;h\in\{-30,-20,…,30\},\;b\in\{-20,-10,…,20\},\;s\in\{-20,-10,…,20\}}

Each transformation t modifies hue, brightness and saturation of the image.

Image‑space distance – For a given transformation t, the transformed image is I∘t. Treating the VGG features of all transformed images as point sets, the Hausdorff distance is used:

d_H(𝒜,𝔅)=\max\{\sup_{a∈𝒜}\inf_{b∈𝔅}\|a-b\|,\;\sup_{b∈𝔅}\inf_{a∈𝒜}\|a-b\|\}

where 𝒜 = {Φ(I_1∘t) | t∈𝒯} and 𝔅 = {Φ(I_2∘t) | t∈𝒯}.

Optimal palette swap – The best transformation is obtained by exhaustive search over 𝒯:

t^* = \arg\min_{t∈𝒯} d_H( {Φ(I_1∘t)}, {Φ(I_2∘t)} )

The corresponding palette of I_2 is then applied to I_1 by mapping each colour c_i in the source palette to the closest colour in the target palette under the selected transformation.

Experiments

Palette extraction quality – The BCS‑based extractor consistently produced 5–8 dominant colours for a variety of Pokémon sprites, preserving the semantic regions (e.g., body, eyes, accessories).

Colour‑swap results – Swapping palettes between characters (e.g., turning Bulbasaur into a fire‑type) produced visually coherent images that retained the original line work while adopting the new colour scheme.

User Study

A blind questionnaire was conducted with 68 participants (12 artists). Each participant evaluated 25 randomly selected swapped images among 600 generated results. Three baselines were included: (1) simple brightness matching, (2) average Hue shift, and (3) Monge‑Kantorovich linearisation.

Results show that the proposed method achieved the top rank in roughly 50 % of the cases, outperforming the baselines which ranked lower on average.

Conclusion

The dense palette‑based recolouring framework provides a fully automatic, mathematically defined colour‑transfer for cartoon‑style images. By combining blind palette extraction, a dense HSV transformation space, and a perceptual Hausdorff distance in VGG feature space, the method yields aesthetically pleasing results and surpasses traditional hue‑shift techniques in both objective and subjective evaluations. The approach demonstrates that rigorous colour‑transfer can be effective in domains with limited training data, without relying on deep generative models.