Unlocking GPT‑4V: A Concise Guide to Multimodal Capabilities and Prompt Techniques

Input Modalities

GPT‑4V accepts visual data in two formats:

Single image‑text pair : one image alone or an image accompanied by a brief textual description or task instruction.

Interleaved image‑text sequence : a series of images each followed by a question, command, or contextual note (e.g., “Calculate tax from the receipt image”).

Images can be supplied as uploaded files or via direct URLs.

Prompt Engineering Patterns

Simple text commands : Direct instructions such as Describe the picture in ten words. Output format constraints : Request JSON, tables, or specific markup (e.g., extract driving‑license fields as JSON).

Few‑shot contextual examples : Provide one or more image‑text exemplars before the target query so the model can infer the desired format and reasoning steps.

Visual markers : Annotate the image with arrows, circles, or hand‑drawn notes and refer to those markers in the prompt.

Mixed multimodal prompts : Combine several images, sub‑images, textual descriptions, and visual cues to simulate a step‑by‑step learning process.

Chain‑of‑thought prompting : Use a “think → act → observe” loop to guide stepwise visual reasoning.

Example of a well‑crafted prompt for counting objects:

You are an expert at counting objects in images. Count the apples in the picture below, listing each step of your reasoning to ensure accuracy.

Core Multimodal Capabilities

Visual understanding & language output : image description, landmark/celebrity recognition, medical‑image interpretation, logo detection, scene analysis, counter‑factual description.

Object localization, counting & annotation : bounding‑box generation, counting items (e.g., apples), highlighting specific regions.

OCR & chart comprehension : read handwritten/printed text, solve math problems from scanned worksheets, extract tables, generate code from diagrams.

Video (multi‑frame) reasoning : understand sequences of frames, predict future actions, locate a described moment within a video.

Abstract visual reasoning : solve IQ‑style puzzles, pattern inference, odd‑one‑out detection.

Humor & cultural context : interpret memes, multilingual captions, cultural references.

Emotional intelligence : infer emotions from facial expressions, assess aesthetic quality.

Representative Application Domains

Defect detection : compare a reference image with a product photo to locate dents, scratches, or other anomalies.

Safety inspection : identify missing personal protective equipment (helmets, safety belts) on construction‑site images.

Retail checkout automation : recognize items on a shelf, read price tags, and compute totals.

Medical assistance : generate draft radiology reports from sequences of scans; extract key findings.

Auto‑insurance assessment : evaluate vehicle damage from crash photos and produce structured claim reports.

Photo organization & search : tag family members, pets, or objects and enable natural‑language queries such as “photos with Linda, Sam, and a dog”.

Image annotation & segmentation : produce detailed captions, bounding boxes, or masks for objects of interest.

Image‑text similarity evaluation : score how well a generated image matches a textual description (e.g., 1‑10 scale).

AI‑assisted image editing : generate optimized textual prompts for downstream editors (MidJourney, Stable Diffusion).

Embodied agents / robotics : enable a robot to recognize appliance buttons, plan navigation steps, and execute tasks like “go to the kitchen and fetch an item from the fridge”.

GUI navigation : interpret screen captures, decide the next mouse click or keyboard action to achieve a goal (e.g., open a news page).

Enhancing GPT‑4V with Classic LLM Techniques

Plugins : integrate search‑engine or tool plugins to provide real‑time information.

Multimodal chain‑of‑thought (ReAct) : combine reasoning and action loops to decompose complex visual tasks.

Self‑reflection & double‑check : ask the model to verify its answer before finalizing output.

Retrieval‑augmented generation (RAG) : embed product images, price tables, or domain documents in the prompt to reduce hallucinations and improve factual accuracy.

Limitations & Future Directions

The paper notes that GPT‑4V’s OCR performance on Chinese characters remains error‑prone, and that the model currently cannot generate mixed image‑text outputs. Future research aims to close these gaps, improve multilingual OCR, and enable fully bidirectional multimodal generation.

Key Visual Illustrations