High-Performance Line Art Extraction in HarmonyOS: Algorithms & Optimization

Core Algorithm Principles

Line art extraction is essentially edge detection in computer vision. The basic pipeline consists of three steps:

Grayscale conversion : eliminate color interference, retain only luminance information.

Gradient calculation : detect the magnitude of pixel brightness changes. Larger changes indicate edges.

Thresholding : decide whether a pixel becomes black (line) or white (background) based on gradient strength.

1. Grayscale Formula

The human eye is most sensitive to green, least to blue. The standard grayscale conversion formula is:

2. Edge Detection Operator

The project uses a simplified differential algorithm to compute gradients, which has lower computational cost than Sobel or Canny operators and is better suited for direct execution on mobile frontends.

For a pixel at (x, y), horizontal gradient Gx and vertical gradient Gy are calculated as:

Gx = |Gray(x+1, y) - Gray(x-1, y)| (right - left)

Gy = |Gray(y+1, x) - Gray(y-1, x)| (bottom - top)

The final gradient magnitude G is:

Key Technical Implementation

1. Efficient Pixel Access: ArrayBuffer

In ArkTS, never use pixelMap.getPixel(x, y) for per-pixel access because the cross-language communication (JS <-> Native) overhead is huge.

Correct approach : read the entire image pixel data into an ArrayBuffer at once, then operate with a Uint8ClampedArray typed array.

// 1. Get image dimensions
const imageInfo = await this.originalPixelMap.getImageInfo();
const width = imageInfo.size.width;
const height = imageInfo.size.height;

// 2. Allocate memory buffer
const pixelBuffer = new ArrayBuffer(width * height * 4); // RGBA 4 channels

// 3. Bulk read pixel data (extremely fast)
await this.originalPixelMap.readPixelsToBuffer(pixelBuffer);
const pixels = new Uint8ClampedArray(pixelBuffer);

2. Core Algorithm Code Walkthrough

The following core logic processes a single pixel chunk. The processChunk function implements the edge detection algorithm described above.

// Iterate each pixel (exclude a one-pixel border to avoid out-of-bounds)
for (let y = startY; y < endY; y++) {
  for (let x = 1; x < width - 1; x++) {
    const idx = (y * width + x) * 4;

    // --- Step 1: Get grayscale of current pixel and its four neighbors ---
    // Compute neighbor indices
    const topIdx = ((y - 1) * width + x) * 4;
    const bottomIdx = ((y + 1) * width + x) * 4;
    const leftIdx = (y * width + (x - 1)) * 4;
    const rightIdx = (y * width + (x + 1)) * 4;

    // Helper: compute grayscale at an index (inlined for performance)
    // Gray = 0.299*R + 0.587*G + 0.114*B
    const topGray = Math.round(0.299 * pixels[topIdx] + 0.587 * pixels[topIdx + 1] + 0.114 * pixels[topIdx + 2]);
    const bottomGray = Math.round(0.299 * pixels[bottomIdx] + 0.587 * pixels[bottomIdx + 1] + 0.114 * pixels[bottomIdx + 2]);
    const leftGray = Math.round(0.299 * pixels[leftIdx] + 0.587 * pixels[leftIdx + 1] + 0.114 * pixels[leftIdx + 2]);
    const rightGray = Math.round(0.299 * pixels[rightIdx] + 0.587 * pixels[rightIdx + 1] + 0.114 * pixels[rightIdx + 2]);

    // --- Step 2: Compute gradients ---
    const gradientX = Math.abs(rightGray - leftGray);
    const gradientY = Math.abs(bottomGray - topGray);
    let gradient = Math.sqrt(gradientX * gradientX + gradientY * gradientY);

    // --- Step 3: Threshold decision and coloring ---
    // If gradient exceeds threshold, it's an edge and needs coloring
    if (gradient > threshold) {
      // Write target color (supports custom color)
      const rgb = this.hexToRgb(this.lineColor);

      // Write to result array (note: ArkUI PixelMap may use BGRA order depending on format; adjust as needed)
      lineArtPixels[idx] = rgb.b;     // B
      lineArtPixels[idx + 1] = rgb.g; // G
      lineArtPixels[idx + 2] = rgb.r; // R
      lineArtPixels[idx + 3] = 255;   // Alpha opaque
    } else {
      // Non-edge areas fill white background
      lineArtPixels[idx] = 255;
      lineArtPixels[idx + 1] = 255;
      lineArtPixels[idx + 2] = 255;
      lineArtPixels[idx + 3] = 255;
    }
  }
}

3. Style Processing (Sketch & Comic)

To add variety, different styles are achieved by adjusting gradient computation:

Sketch style : Simulates pencil roughness.

Principle : Add random noise and vary line opacity (alpha) based on gradient strength, creating depth variation.

Implementation : Randomly boost weak edges; map gradient to opacity (stronger gradient → more opaque).

if (style === "素描") {
  // Enhance contrast
  gradient = gradient * 1.3;
  // Random noise: randomly strengthen weak edges
  if (Math.random() < 0.1 && gradient > threshold * 0.3) {
    gradient = threshold + 1;
  }
  // Opacity mapping: larger gradient → more opaque
  lineOpacity = Math.min(255, Math.max(100, gradient * 1.2));
}

Comic style : Aims for high contrast, clean lines, removing clutter.

Principle : Only retain very obvious edges, uniform pure black.

Implementation : Raise filtering threshold; zero out weak edges; set opacity to 255.

if (style === "漫画") {
  // Zero out weak edges for denoising
  if (gradient < threshold * 1.5) {
    gradient = 0;
  }
  // Uniform opacity
  lineOpacity = 255;
}

Performance Optimization Strategies (Anti-Freeze)

Processing a 4K image (e.g., 4000 × 3000 pixels) involves over 12 million loop iterations. Running this on the main thread in one go would freeze the UI for 2-5 seconds, potentially triggering an ANR.

The solution adopts a Time Slicing strategy.

1. Async Chunked Processing

The image is split into row chunks (e.g., 50 rows per chunk). After each chunk, setTimeout or Promise yields control back to the main thread, allowing UI updates (like a progress bar) before the next chunk.

private async processImageInChunks(pixels: Uint8Array, width: number, height: number) {
  const chunkSize = 50; // Process 50 rows at a time

  for (let startY = 1; startY < height - 1; startY += chunkSize) {
    const endY = Math.min(startY + chunkSize, height - 1);

    // 1. Process current chunk
    await this.processChunk(pixels, ..., startY, endY);

    // 2. Update progress bar (UI responsiveness)
    this.processProgress = Math.round((startY / height) * 100);

    // 3. Critical: delay 1ms to yield main thread
    await this.delay(1);
  }
}

private delay(ms: number): Promise<void> {
  return new Promise(resolve => setTimeout(resolve, ms));
}

2. Large Image Auto-Downsampling

For images exceeding 2048px in width or height, downsampling is performed before line extraction. This multiplies processing speed and reduces high-frequency noise that causes messy lines, yielding cleaner results.

const maxSize = 2048;
if (width > maxSize || height > maxSize) {
  const scale = Math.min(maxSize / width, maxSize / height);
  // Use ArkTS Image API to scale
  await scaledPixelMap.scale(scale, scale);
}

Summary

This article demonstrates building a line art extraction feature from scratch in a HarmonyOS app. Without relying on heavy third-party libraries like OpenCV, it directly manipulates PixelMap binary buffers with a custom edge detection algorithm, delivering a lightweight, efficient, and customizable solution.

Key highlights:

Direct memory access : Uses ArrayBuffer to avoid expensive cross-language calls.

Algorithm customization : Tweaks gradient logic for sketch and comic styles.

User experience optimization : Async time-slicing keeps the app responsive during heavy image computation.