Pixeltable: One Table to Power Multimodal AI with Declarative Python

01 Concept Innovation

Why "everything is a table" matters.

Multimodal AI development suffers from fragmented tools: separate vector databases, SQL stores, object storage, and custom glue code. Pixeltable solves this by providing a single abstraction where images, text blocks, model outputs, and vector indexes are all columns in a table, and data processing is expressed as declarative computed columns.

Image is a column, each row holds one image.

Text block is a column, each row holds a document fragment.

Model output is a column, automatically computed and stored.

Vector index is built into the table, no external database needed.

The declarative paradigm lets you describe what to compute rather than how , with the system handling dependencies, incremental recomputation, and caching.

import pixeltable as pxt
# Create a basic table: declare schema only
movies = pxt.create_table(
    'films',
    {'name': pxt.String, 'revenue': pxt.Float, 'budget': pxt.Float},
    if_exists='replace'
)
movies.insert([
    {'name': 'Inside Out', 'revenue': 800.5, 'budget': 200.0},
    {'name': 'Toy Story', 'revenue': 1073.4, 'budget': 200.0}
])
# Declare a computed column
movies.add_computed_column(
    profit=(movies.revenue - movies.budget),
    if_exists='replace'
)
results = movies.select(movies.name, movies.profit).collect()
print(results)

Pixeltable separates computation logic from execution, automatically optimizing the execution plan and recomputing only the necessary parts.

02 Multimodal Data Processing

Image, text, and vector search unified.

Handling different data types is a core challenge. Pixeltable provides a uniform table interface for both image and text processing.

Image pipeline example:

# Create an image table
images = pxt.create_table('my_images', {'img': pxt.Image}, if_exists='replace')
# Insert images from URL, local path, or PIL object
images.insert([
    {'img': 'https://example.com/image1.jpg'},
    {'img': '/local/path/to/image2.png'},
    {'img': image_pil_object}
])
# Object detection using a HuggingFace model
from pixeltable.functions import huggingface
images.add_computed_column(
    objects=huggingface.detr_for_object_detection(
        images.img,
        model_id='facebook/detr-resnet-50'
    )
)
# Image caption generation with OpenAI Vision
from pixeltable.functions import openai
images.add_computed_column(
    description=openai.vision(
        model='gpt-4o-mini',
        prompt='详细描述图像内容',
        image=images.img
    )
)

Vector search is integrated: the index lives inside the table, eliminating the need for a separate vector database.

# Create an embedding index for the image column
from pixeltable.functions.huggingface import clip
images.add_embedding_index(
    'img',
    embedding=clip.using(model_id='openai/clip-vit-base-patch32')
)
# Text‑to‑image search
query_text = '一只在公园玩耍的狗'
similarity_score = images.img.similarity(query_text)
results = images.order_by(similarity_score, asc=False).limit(5).collect()
# Image‑to‑image search
query_image_url = 'https://example.com/query_dog.jpg'
image_similarity = images.img.similarity(query_image_url)
image_results = images.order_by(image_similarity, asc=False).limit(3).collect()

The integrated design automatically updates indexes when new images are added or modified.

03 Complete Workflow

30 lines of code implement a Retrieval‑Augmented Generation (RAG) system.

Traditional RAG requires stitching together document stores, chunkers, embedding models, and LLM calls. Pixeltable expresses the entire pipeline declaratively.

# Document storage
docs = pxt.create_table('my_docs.docs', {'doc': pxt.Document})
docs.insert([
    {'doc': 'https://example.com/ai_report.pdf'},
    {'doc': 'https://example.com/tech_whitepaper.docx'}
])
# Split documents into chunks
chunks = pxt.create_view(
    'doc_chunks',
    docs,
    iterator=pxt.functions.DocumentSplitter.create(
        document=docs.doc,
        separators=['。', '!', '?', '

'],
        chunk_size=300,
        overlap=50
    )
)
# Create a semantic index
from pixeltable.functions import huggingface
embed_model = huggingface.sentence_transformer.using(model_id='all-MiniLM-L6-v2')
chunks.add_embedding_index('text', string_embed=embed_model)
# Retrieval function
@pxt.query
def retrieve_relevant_chunks(query: str, top_k: int = 3):
    """Retrieve most relevant text chunks"""
    similarity = chunks.text.similarity(query)
    return chunks.order_by(similarity, asc=False).limit(top_k).select(chunks.text)
# QA system table
qa_system = pxt.create_table('my_docs.qa', {'question': pxt.String})
qa_system.add_computed_column(context=retrieve_relevant_chunks(qa_system.question))
qa_system.add_computed_column(
    prompt=pxtf.string.format(
        "参考信息：
{0}

问题：{1}
请根据参考信息回答问题：",
        qa_system.context,
        qa_system.question
    )
)
qa_system.add_computed_column(
    answer=openai.chat_completions(
        model='Qwen2.5-32B',
        messages=[{'role': 'user', 'content': qa_system.prompt}],
        temperature=0.2
    ).choices[0].message.content
)
# Use the system
qa_system.insert([{'question': '人工智能的主要发展趋势是什么？'}])
answers = qa_system.select(qa_system.question, qa_system.answer).collect()

All components are declarative and composable; changing the splitter or embedding model requires only a single line change.

04 Advanced Features

Version control and incremental computation.

Production AI needs reproducibility. Pixeltable includes a built‑in versioned table mode, allowing experiments to be checkpointed, compared, and rolled back.

# Experiment tracking with version control
experiments = pxt.create_table(
    'model_experiments',
    {'config': pxt.String, 'data_version': pxt.String, 'result': pxt.Float},
    mode='versioned'
)
# Baseline experiment
experiments.insert([{'config': 'v1', 'data_version': '2024-01', 'result': 0.78}])
checkpoint_v1 = experiments.checkpoint('基线模型')
# Improved experiment
experiments.update(
    {'result': 0.85},
    where=experiments.config == 'v1'
)
checkpoint_v2 = experiments.checkpoint('增加数据增强')
# Compare versions
historical_results = experiments.at(checkpoint_v1).collect()
print(f'版本{checkpoint_v1}的结果：{historical_results}')
changes = experiments.diff(checkpoint_v1, checkpoint_v2)
print(f'版本间变更：{changes}')
# Incremental recomputation: only affected rows are recomputed
experiments.add_computed_column(
    improved_score=experiments.result * 1.1,
    if_exists='replace'
)

The system analyses the dependency graph and recomputes only the columns impacted by data changes, greatly improving efficiency on large datasets.

05 Extensions and Integration

Connecting to the existing Python ecosystem.

Pixeltable is not a closed system; custom UDFs and external data sources can be integrated.

# Custom image captioning UDF
import torch
from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
@pxt.udf(batch_size=4, return_type=pxt.ArrayType(dtype=pxt.String))
def custom_image_caption(images: list) -> list:
    """Batch image caption generation"""
    processor = BlipProcessor.from_pretrained('Salesforce/blip-image-captioning-base')
    model = BlipForConditionalGeneration.from_pretrained('Salesforce/blip-image-captioning-base')
    captions = []
    for img_batch in [images[i:i+4] for i in range(0, len(images), 4)]:
        inputs = processor(images=img_batch, return_tensors='pt', padding=True)
        outputs = model.generate(**inputs)
        batch_captions = processor.batch_decode(outputs, skip_special_tokens=True)
        captions.extend(batch_captions)
    return captions

images_table = pxt.create_table('custom_caption_images', {'img': pxt.Image})
images_table.insert([{'img': 'https://example.com/scene.jpg'}])
images_table.add_computed_column(custom_caption=custom_image_caption(images_table.img))

# External API integration
@pxt.udf
def fetch_external_data(query: str) -> str:
    """Fetch data from an external API"""
    import requests
    response = requests.get(f'https://api.example.com/search?q={query}')
    return response.text

enhanced_qa = pxt.create_view(
    'enhanced_qa',
    qa_system,
    computed_columns={'external_context': fetch_external_data(qa_system.question)}
)

These mechanisms provide out‑of‑the‑box convenience while retaining flexibility for complex scenarios.

06 Practical Applications: From Prototype to Production

Pixeltable supports the full AI lifecycle.

# Rapid prototyping
exploration = pxt.create_table('explore_data', {
    'image': pxt.Image,
    'text': pxt.String,
    'metadata': pxt.Json
})
exploration.add_computed_column(objects=pxt.functions.huggingface.detect_objects(exploration.image))
exploration.add_computed_column(sentiment=pxt.functions.openai.analyze_sentiment(exploration.text))
exploration.add_computed_column(combined_score=0.6 * exploration.objects.confidence + 0.4 * exploration.sentiment.score)

# Production pipeline with versioned table
production_table = pxt.create_table('production_images', {
    'raw_image': pxt.Image,
    'timestamp': pxt.Timestamp
}, mode='versioned')
@pxt.udf(error_handling='null')
def robust_image_processing(image):
    """Image processing with error handling"""
    try:
        if image.mode != 'RGB':
            image = image.convert('RGB')
        result = process_image(image)
        return result
    except Exception as e:
        log_error(f'图像处理失败: {e}')
        return None
production_table.add_computed_column(processed_result=robust_image_processing(production_table.raw_image))
# Batch and streaming processing using the same interface
historical_data = production_table.where(production_table.timestamp < '2024-01-01').select(production_table.processed_result).collect()
new_data = production_table.where(production_table.timestamp >= '2024-01-01').select(production_table.processed_result).collect()

These examples demonstrate that the same declarative abstraction scales from a few hundred rows in a prototype to millions of rows in a production system.

Pixeltable rethinks the fundamental unit of multimodal AI development: data, transformation, and query are all expressed as tables and columns, offering a potential new standard for AI infrastructure.