Zero‑Copy Texture Sharing in Flutter: Using CVPixelBuffer for Native‑GPU Integration

Background and Problem

Flutter’s external texture API originally transfers native textures to Flutter via CVPixelBuffer, which forces a GPU‑>CPU‑>GPU copy and creates a performance bottleneck for real‑time video or graphics workloads.

Engine Modification Approach

One solution modifies the Flutter engine to connect the native OpenGL context with Flutter’s GL context through a shared ShareGroup, eliminating the extra memory copy but exposing the engine internals and increasing debugging complexity.

Principle of Flutter External Textures

The process consists of two parts: texture registration and texture rendering.

Texture Registration

Create an object that implements the FlutterTexture protocol to manage texture data.

Register this object with FlutterTextureRegistry to obtain a Flutter texture ID.

Pass the ID to the Dart side via a platform channel; the Dart side uses the Texture widget with this ID.

Texture Rendering

The Dart side declares a Texture widget referencing the native texture ID.

The engine receives a TextureLayer node in the layer tree responsible for rendering the external texture.

Using the ID, the engine locates the registered FlutterTexture implementation, which provides a copyPixelBuffer method.

The engine calls copyPixelBuffer to retrieve the pixel data and hands it to the GPU for rendering.

Analyzing CVPixelBuffer

The key to eliminating the copy lies in the fact that CVPixelBuffer can be directly mapped to an OpenGL texture via CVOpenGLESTextureCacheCreateTextureFromImage. This creates a live binding between the image buffer and the texture object, allowing both CPU and GPU to access the same memory region under specific synchronization rules.

When the GPU accesses the texture, the CVPixelBuffer must not be locked.

When the CPU accesses the buffer, the GPU must have finished rendering, typically ensured by calling glFlush() before CPU reads or writes.

Shared‑Memory Solution

By creating a CVPixelBuffer with the kCVPixelBufferIOSurfacePropertiesKey property, the same memory can be shared between the native OpenGL context and the Flutter engine. Both contexts create their own texture objects that reference the same underlying buffer, enabling zero‑copy rendering.

Demo Implementation

The following demo renders a rotating triangle at 60 FPS into a texture backed by a shared CVPixelBuffer, then notifies Flutter to read the buffer each frame. The demo confirms smooth performance and can be adapted to other CPU‑GPU shared‑memory scenarios.

void IOSExternalTextureGL::CreateTextureFromPixelBuffer() {
  CVOpenGLESTextureRef texture;
  CVReturn err = CVOpenGLESTextureCacheCreateTextureFromImage(
      kCFAllocatorDefault, cache_ref_, buffer_ref_, nullptr,
      GL_TEXTURE_2D, GL_RGBA,
      (int)CVPixelBufferGetWidth(buffer_ref_),
      (int)CVPixelBufferGetHeight(buffer_ref_),
      GL_BGRA, GL_UNSIGNED_BYTE, 0, &texture);
  if (err != noErr) {
    FML_LOG(WARNING) << "Could not create texture from pixel buffer: " << err;
  } else {
    texture_ref_.Reset(texture);
  }
}

- (void)createCVBufferWith:(CVPixelBufferRef *)target withOutTexture:(CVOpenGLESTextureRef *)texture {
    CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, _context, NULL, &_textureCache);
    // Set shared‑memory flag
    CFDictionarySetValue(attrs, kCVPixelBufferIOSurfacePropertiesKey, empty);
    // Allocate pixel buffer (BGRA format required by Flutter)
    CVPixelBufferCreate(kCFAllocatorDefault, _size.width, _size.height,
        kCVPixelFormatType_32BGRA, attrs, target);
    // Map pixel buffer to an OpenGL texture
    CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, _textureCache,
        *target, NULL, GL_TEXTURE_2D, GL_RGBA, _size.width, _size.height,
        GL_BGRA, GL_UNSIGNED_BYTE, 0, texture);
    CFRelease(empty);
    CFRelease(attrs);
}

- (CVPixelBufferRef)copyPixelBuffer {
    CVBufferRetain(_target);
    return _target; // FlutterTexture protocol implementation
}

- (void)initGL {
    _context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
    [EAGLContext setCurrentContext:_context];
    [self createCVBufferWith:&_target withOutTexture:&_texture];
    glGenFramebuffers(1, &_frameBuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, _frameBuffer);
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D,
        CVOpenGLESTextureGetName(_texture), 0);
    // ... rendering loop omitted for brevity ...
}

Conclusion

Using CVPixelBuffer as a shared memory bridge allows native iOS OpenGL rendering and Flutter texture consumption to operate on the same memory without extra copies, achieving high‑performance, low‑latency rendering suitable for video, AR, and other real‑time graphics applications.

References

Original article: https://juejin.im/post/5b7b9051e51d45388b6aeceb

Full demo source: https://github.com/luoyibu/flutter_texture

Author’s blog post: http://www.luoyibu.cn/posts/9703/