Advanced Guide to LLM API: Multimodal Input and Structured Output

When LLMs move from pure text to real‑world applications, two capabilities become critical: multimodal input (images, audio, video) and structured output that can be directly consumed by downstream systems. Together they form a complete data pipeline, turning model predictions into stable system components.

Why Multimodal and Structured Output Matter

Without multimodal support, users must pre‑process images, screenshots, or audio into text, losing information and adding latency. Without structured output, even a correct answer may be unusable because the JSON is malformed, missing fields, or contains unexpected data types.

Three Core Capabilities

Multimodal Input : The model can understand images, audio, or video.

Text‑to‑Image Generation : The model can produce image results.

Structured Output : The model returns data in a format that downstream code can parse reliably.

Multimodal Input Details

Different providers accept different formats. OpenAI’s GPT‑4o family accepts an image_url object where the url can be an HTTP URL or a data:image/jpeg;base64,… string. The detail field ("auto", "low", "high") controls resolution and token cost. Anthropic Claude requires the image to be sent as a base64 source block; Google Gemini uses an upload step that returns a file ID. Kimi follows the OpenAI style, while many other models (e.g., DeepSeek) only support text.

client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": [
            {"type": "text", "text": "Describe this picture"},
            {"type": "image_url", "image_url": {"url": "https://example.com/img.jpg", "detail": "auto"}}
        ]}
    ]
)

Image Generation (Text‑to‑Image) Parameters

OpenAI’s DALL·E 3 uses the size, quality, and style parameters. Typical values are size="1024x1024" or size="1792x1024", quality="standard" or quality="hd", and style="vivid" or style="natural". These directly affect cost and visual fidelity.

client.images.generate(
    model="dall-e-3",
    prompt="A cyberpunk city skyline",
    size="1792x1024",
    quality="hd",
    style="vivid",
    n=1
)

Structured Output Challenges

Many teams start with response_format={"type": "json_object"}. This often yields “JSON‑like” text that may miss fields, add extra commentary, or change data types, causing parsing failures. The article proposes a three‑layer approach:

Prompt Layer : Explicitly ask for pure JSON (e.g., “Output only a valid JSON object, no explanations”).

Parsing Layer : In code, catch json.JSONDecodeError and fall back to retry or manual review.

Schema Layer : Use OpenAI’s json_schema (or Gemini’s response_schema) to enforce field names, types, required properties, and disallow additional properties.

Example of a strict schema for invoice extraction:

response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[...],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "invoice",
            "strict": true,
            "schema": {
                "type": "object",
                "properties": {
                    "invoice_number": {"type": "string"},
                    "date": {"type": "string"},
                    "total_amount": {"type": "number"},
                    "seller_name": {"type": "string"}
                },
                "required": ["invoice_number", "date", "total_amount", "seller_name"],
                "additionalProperties": false
            }
        }
    }
)

Practical Scenarios

Invoice Recognition : Upload an image (or base64) and request fields via a strict JSON schema. The key is not just image understanding but guaranteeing the JSON can be stored directly.

Content Production : Generate an image with DALL·E 3 and immediately request a textual design explanation from a chat model, keeping both results in a single workflow.

Best‑Practice Checklist

Know whether the provider expects a URL, base64, or both for image_url.

Set detail (OpenAI) or equivalent to match required resolution and token budget.

When cost matters, prefer lower‑resolution or lower‑detail settings for OCR‑type tasks.

Use json_schema (or equivalent) for mission‑critical data; fall back to json_object only for low‑risk cases.

Always include explicit “output JSON only” instructions in the system prompt.

Wrap JSON parsing in try/except and define a retry or manual‑review path.

If a platform lacks native schema support, employ tool/function calling to force a structured response.

Parameter Quick Reference

Multimodal Input : image_url (URL or base64), detail (auto/low/high) – OpenAI, Kimi, Gemini.

Image Generation : model (dall‑e‑3 / dall‑e‑2), size, quality, style – OpenAI.

Structured Output : response_format with json_object (most platforms) or json_schema (OpenAI, Gemini).

Conclusion

Multimodal input and structured output together reshape the LLM‑to‑system interface: models can now see real‑world data and return results that behave like stable API contracts. Selecting the right format, cost‑aware resolution, and enforcing schemas are essential for production‑grade reliability.