Automaker's Large-Scale Multimodal Labeling with EMR Serverless Spark AI Functions

Engineering Challenges of Large-Scale Model Invocation

When processing shifts from a few images to continuously accumulating business data, model invocation must coordinate with data reading, parallel execution, result parsing, and data writing. Building custom wrappers forces developers to handle connection pooling, service throttling, and retry logic, intertwining data-processing logic with model-request logic and increasing maintenance burden. Different business scenarios often duplicate similar invocation code, reducing reusability.

Scaling also requires reconciling two execution rhythms: Spark parallelizes data processing, while model services have their own throughput and quota limits. Simply increasing Spark parallelism does not guarantee proportional speedup. Enterprises therefore need to combine model invocation with data-job execution, management, and maintenance so that AI capabilities flow smoothly into existing data pipelines.

EMR Serverless Spark AI Function addresses this by encapsulating model capabilities as built-in functions, allowing data engineers to call models via SQL, PySpark, and Ray and chain them with filtering, joining, parsing, and table writing steps.

Customer Practice: From Scene Understanding to Batch Vectorization

A leading automotive customer has processed hundreds of millions of images on Alibaba Cloud EMR Serverless Spark. Image understanding, information extraction, and vector generation are integrated into Spark jobs through AI Functions, with results used for downstream scene retrieval, sample screening, and training dataset construction.

Lane Splitting/Merging and Traffic Lights: Extracting Scene Information

In lane splitting/merging scenarios, the model identifies road forks and merges; in traffic-light scenarios, it extracts signal status. Extracted results are written to data tables for subsequent scene-based retrieval and sample screening. Image reading, model invocation, result parsing, and writing run consecutively within a single Spark job. Data engineers reuse familiar SQL, PySpark, and Ray development patterns, adding image understanding to existing data-processing steps and avoiding separate model-request wrapping and script maintenance.

Although the two scenarios require different extracted information, they share the same model-invocation pattern. For common batch-job issues such as throttling and retries, the platform provides generic handling mechanisms, letting developers focus on extraction content and result quality.

Batch Vectorization: Foundation for Similar Scene Retrieval

Beyond scene extraction, images are vectorized to enable similarity search. Embeddings convert image content into numerical features so that inter-image similarity can be computed, providing a basis for later retrieval.

The customer uses Spark AI Function to generate image vectors in bulk, incorporating this step into the data pipeline. In the actual chain, data lands in DLF Paimon tables; Spark jobs perform the processing, and the resulting embeddings are written to downstream databases, supporting scene retrieval, sample screening, and training dataset construction.

Scene information facilitates conditional data organization, while vectors provide the basis for finding similar images. Through Spark AI Function, the customer integrates both processing tasks into existing Spark workflows, making road images more accessible for subsequent retrieval and use.

Four Values of Built-in AI Functions

EMR Serverless Spark packages common model capabilities as AI Functions. Developers use ai_query for image understanding and information extraction, ai_embedding_multimodal for vector generation, and then parse, filter, join, and write outputs via SQL, PySpark, and Ray. AI invocation thus becomes a standard step within existing data jobs. Spark handles distributed reading, processing, and writing of batch data, while AI Functions connect model capabilities; together they enable image understanding and vector generation to run alongside existing data jobs.

Simpler Integration: Reducing Duplicate Development

Spark AI Function offers SQL, PySpark, and Ray invocation modes covering content understanding, information extraction, classification, translation, embedding, and other common tasks. EMR Serverless Ray also supports AI Functions via the ray.ai package. Data engineers continue using familiar data-development practices, combining AI processing with existing jobs. Generic work such as model-request wrapping is offloaded to the platform, reducing repeated coding effort across data tasks.

More Efficient Batch Processing: Coordinating Data Parallelism and Model Requests

Large-scale AI data processing demands both distributed computing and well-organized model requests. AI Functions incorporate concurrency control, connection reuse, and request batching (where applicable) to reduce redundant request overhead and align Spark job pacing with model service throughput. Actual throughput depends on model service quotas, input data characteristics, and job configuration, allowing users to plan batch tasks accordingly.

More Robust Operation: Lowering Exception-Handling Burden

Production jobs must handle throttling, timeouts, and model service anomalies. AI Functions provide built-in throttling and retry capabilities, reducing the need for users to maintain generic exception-handling code. Developers can further organize failed-data handling and result verification per task requirements, integrating AI calls more reliably into continuously running data pipelines.

Unified Management: Easier Model Service Maintenance

Centralized configuration of model service endpoints and credentials on the platform avoids scattered configurations across scripts, simplifying maintenance. Data jobs can reuse configured model services and select models on demand, eliminating repetitive configuration in multiple job scripts.

Additional Use Cases

Beyond automotive scenarios, AI Functions apply to:

Product image processing: Extract product image information and generate vectors for product search and content management.

Text data processing: Perform classification, summarization, and information extraction to make unstructured text easier to analyze.

Knowledge base preparation: Clean and vectorize text for retrieval-augmented generation (RAG) applications.

Embodied intelligence data processing: Understand robot operation images, label actions such as grasping and placing, and generate vectors for similar scene retrieval and sample screening.

Getting Started

Teams with image understanding, information extraction, or batch vectorization needs can start by validating on a sample dataset. After enabling AI Center, confirming the engine version, and activating AI Functions, developers can refer to the examples below to call the relevant functions in Spark SQL, PySpark, and Ray jobs.

AI Function Introduction: https://help.aliyun.com/zh/emr/emr-serverless-spark/user-guide/introduction-to-ai-function

Multimodal Function Best Practices: https://help.aliyun.com/zh/emr/emr-serverless-spark/use-cases/multimodal-function-best-practices

PySpark AI Function Invocation: https://help.aliyun.com/zh/emr/emr-serverless-spark/user-guide/call-ai-function-with-pyspark