AI Model Types, Quantization & File Formats: A Complete Guide

01 Introduction

When people think of AI, they often picture chat applications like ChatGPT and large language models (LLMs) such as DeepSeek or Qwen. However, a knowledge base, voice assistant, or image generation app relies on more than just an LLM. Retrieval may use an embedding model, reranking a reranker model, while speech recognition and image generation each require specialized models. On model download pages, labels like 8B, Instruct, VL, FP16, Q4_K_M, GGUF, and Safetensors appear together, but they describe different aspects: some indicate what the model does, others the modalities it handles, and still others the weight precision or file format. Mixing these concepts leads to confusion. This article separates them into four clear dimensions.

02 Four Dimensions of Model Description

A single model can carry multiple labels simultaneously. To understand them, distinguish four orthogonal dimensions:

Purpose & Task – What the model is used for (e.g., chat, image recognition, speech transcription, content retrieval).

Information Modality – What form of information the model processes (text, image, audio, video).

Numerical Precision & Quantization – How weights are represented numerically (FP16, INT8, Q4, etc.).

Storage & Deployment Format – How the model is saved and which software reads it (Safetensors, GGUF, ONNX).

For example, a model may be used for visual question answering, accept both images and text, use 4-bit quantization, and be saved in GGUF format. All four descriptions are true at the same time. The article notes that dense vs. MoE architecture is a separate structural dimension covered in a previous post.

03 Model Classification by Task

Models are categorized by the most intuitive question: what can they help us do?

1. Large Language Models (LLMs)

LLMs primarily process language—Q&A, translation, summarization, writing, code generation. “Language” includes programming languages because code can be tokenized. However, not every text-processing model is an LLM; a small sentiment classifier handles text but cannot hold open-ended conversations or generate articles.

2. Vision Models

Vision models process images. Common tasks:

Image classification – assign a category to the whole image.

Object detection – locate objects like vehicles or pedestrians.

Image segmentation – label each region or pixel with an object class.

A GIS example: a satellite image can be used to identify crop types, detect roads and buildings, and delineate water boundaries—different outputs from the same input modality.

3. Image and Video Generation Models

Some vision models recognize and analyze existing visuals; others generate new images or videos from text or reference material. Image recognition answers “what is in the picture?”, generation answers “draw what I describe”, and editing combines both: understand the original then modify per text instructions.

4. Speech and Audio Models

Speech applications typically chain multiple models:

Task            | Input → Output       | Common Use Cases
----------------|----------------------|------------------
Speech Recognition | Audio → Text        | Meeting transcription, voice input
Speech Synthesis   | Text → Audio        | Voice broadcasting, audio content
Audio Understanding| Audio → Labels/Text | Emotion, event, or music recognition
Audio Generation   | Text/Audio → Audio  | Music, sound effects, voice generation

A voice assistant that “listens and speaks” may use separate recognition and synthesis models.

5. Embedding Models (Vector Models)

Embedding models convert text, images, etc., into vectors. Semantically similar content ends up close in vector space, enabling semantic search (e.g., a query for “move Ollama model to another computer” can match a document titled “Model Migration”). They power knowledge-base retrieval, semantic search, and recommendations; the vectors are consumed programmatically and rarely seen by users.

6. Reranker Models

Vector retrieval returns a candidate set, but the top results may not best answer the query. A reranker re-scores candidates using the query and reorders them. In a knowledge-base QA pipeline:

User Question
     ↓
Embedding Model Finds Candidates
     ↓
Reranker Re-ranks Candidates
     ↓
LLM Generates Answer Using Top Candidates

A simple chat box may thus invoke three distinct models. Other common types include recommendation, time-series forecasting, and anomaly detection models.

04 Multimodal Models

1. What Is a Modality?

Modality refers to the form of information:

Text is one modality.

Image is another.

Audio and video are each distinct modalities.

Modality is not file extension: TXT, Markdown, DOCX all carry text; PNG and JPEG both carry images.

2. Multimodal Does Not Mean It Does Everything

A model that handles two or more modalities and learns cross-modal relationships is multimodal. It may contain multiple encoders trained jointly. A Vision-Language Model (VLM) accepts an image and a text question, then answers in text:

Image + “How many cars in the picture?”
         ↓
    Vision-Language Model
         ↓
    “There are three.”

Even though output is text-only, it is multimodal because input fuses image and text. Text-to-image generation is also cross-modal. Multimodal, image generation, and speech recognition are not mutually exclusive categories; “multimodal LLM” usually means a general-purpose model that accepts multiple inputs and performs open-ended understanding and generation. Supporting multiple modalities does not imply universal capability: a VLM may answer questions about images but not generate them; a speech recognition model may transcribe but not synthesize.

3. Multimodal Applications vs. Multimodal Models

A multimodal application can accept speech, images, and text, yet those capabilities may come from multiple specialized models. The central LLM might still only process text. Example voice assistant pipeline:

User Voice
     ↓
Speech Recognition Model
     ↓
Text
     ↓
Large Language Model
     ↓
Response Text
     ↓
Speech Synthesis Model
     ↓
Playback Voice

The application is multimodal, but the LLM remains text-only. This differs from a native multimodal model that directly ingests speech. Judge a model by its own supported inputs, outputs, and processing flow—not by the application’s UI. Deployment must also verify inference framework support.

05 Quantized Models

1. Quantization Changes Weight Representation

A trained model contains millions of weights stored as FP32, FP16, or BF16. Quantization converts these to lower-bit representations (e.g., INT8, 4-bit), reducing file size, memory footprint, and weight-read bandwidth, enabling large models to run on limited hardware. Quantization does not change the model’s purpose, architecture, or knowledge—only how weights are stored and computed.

2. What FP16, Q8, Q4 Roughly Mean

Name  | Rough Meaning
------|--------------
FP16  | Weights mainly 16-bit floating point
INT8  | Weights or computation use 8-bit integers
Q8    | An 8-bit quantization scheme
Q4    | A 4-bit quantization scheme

Suffixes like K, M, XL (e.g., Q4_K_M) come from specific quantization tools and indicate grouping strategies, per-tensor precision choices, or quality trade-offs. No universal standard exists; consult the model repo and inference tool documentation.

3. Why Quantization Reduces Memory Usage

Ideal weight size ≈ parameter count × bytes per parameter. For an 8B (8 billion parameter) model:

Weight Representation | Bytes per Parameter | 8B Model Estimate
----------------------|---------------------|------------------
FP16                  | 2 bytes             | ~16 GB
8-bit                 | 1 byte              | ~8 GB
4-bit                 | 0.5 bytes           | ~4 GB

Real-world overhead includes metadata, scaling factors, KV cache, temporary tensors, and framework overhead. A file size just below VRAM capacity does not guarantee successful loading.

4. Differences Before and After Quantization

Aspect              | Before Quantization (FP16/BF16) | After Quantization
--------------------|----------------------------------|-------------------
Model File Size     | Usually larger                   | Lower bits → smaller file
Memory Usage        | Needs more VRAM/RAM              | Weight memory significantly reduced
Weight Read Volume  | More data moved per step         | Data movement typically lower
Inference Speed     | Depends on FP compute & bandwidth  | May be faster with good kernel/hardware support
Output Quality      | Closest to original capability     | Possible precision loss, varies by task
Software Compatibility | Widely supported in frameworks   | Depends on quantization method & inference engine

Quantization’s main benefit is fitting models into memory and easing bandwidth pressure. The cost is potential quality degradation, especially in math, code, and logic tasks. Lower bits are not always better; choose based on what fits your hardware, what your inference engine supports, and empirical quality/speed trade-offs on your tasks.

06 Model File Formats

1. Model Files Are Not Single Files

Downloading a model often yields a folder containing:

One or more weight files (often sharded).

Model architecture and runtime configuration.

Tokenizer and its configuration.

Image/audio preprocessing configs for multimodal models.

Shards must be combined; a single shard cannot run alone. Different frameworks require different file sets; renaming extensions or copying one file rarely yields a runnable model.

2. Safetensors: Safe Weight Storage

Safetensors securely and efficiently stores tensors (multi-dimensional arrays). Older PyTorch pickle-based formats could execute arbitrary Python code on load, posing security risks. Safetensors restricts storage to data only, enables fast partial loading, and is widely used with Transformers for loading, fine-tuning, or continued training. However, Safetensors only holds weights; it cannot describe how to run the model. It typically appears alongside config, tokenizer, and preprocessing files. .safetensors only indicates the weight storage format; it reveals nothing about model purpose, size, precision, or quantization.

3. GGUF: Easy Distribution for Local Inference

GGUF is the native format of the llama.cpp ecosystem. It bundles weights, model architecture, tokenizer, and other metadata into a single file, so downloading one GGUF often suffices for compatible tools (llama.cpp, Ollama, LM Studio). It commonly distributes multiple quantization variants:

model-Q4_K_M.gguf
model-Q8_0.gguf

.gguf

denotes the file format; Q4_K_M and Q8_0 denote quantization schemes. GGUF can store both quantized and higher-precision weights, so “GGUF equals 4-bit” is incorrect. It supports CPU and hybrid CPU/GPU inference, but not all training frameworks or server inference engines read GGUF directly. Some multimodal models still require extra projection files or resources.

4. ONNX: Compute Graph Exchange Format

ONNX bridges training frameworks and deployment runtimes. It stores weights and describes the computation graph—operator sequence and data flow. Example: export a PyTorch image classifier to ONNX, then run it via ONNX Runtime on CPU, GPU, or other hardware without installing PyTorch. Common scenarios:

Deploy trained models to a different runtime.

Unified inference interface across servers, desktops, edge devices.

Deploy models with relatively fixed graphs (image classification, object detection, speech processing).

Existing tooling built around ONNX Runtime.

LLMs can be exported to ONNX, but whether it’s worthwhile depends on architecture, inference flow, and runtime support. For local quantized LLM inference via Ollama/LM Studio, GGUF is more direct. Consider ONNX when cross-framework deployment is needed and the target environment already uses ONNX Runtime. Exporting to ONNX does not guarantee universal device compatibility; the runtime must support all operators, dynamic shapes may need adaptation, and performance/accuracy must be re-validated.

5. .pt, .pth, .bin, .ckpt

These extensions are common but their contents depend on the saving software and project conventions. They may hold only weights, full objects, or training checkpoints. The extension alone cannot determine how to load them; rely on model documentation, config files, and the originating framework.

6. Common Misconceptions About Formats and Quantization

Common Claim                     | Accurate? | Reason
---------------------------------|-----------|-------------------------------------------
GGUF is 4-bit quantization       | No        | GGUF is a format; can store various precisions
Safetensors are always high-precision | No     | It’s a tensor container; can hold quantized weights
Q4/Q8 are file formats           | No        | They describe quantization schemes
Renaming extension converts format | No      | Conversion requires reorganizing weights & metadata
Format conversion adds capabilities | No      | Format doesn’t teach new tasks

07 Putting Labels Together in a Filename

Example filename:

示例模型-8B-Instruct-Q4_K_M.gguf

Label      | Information Conveyed
-----------|---------------------
8B         | ~8 billion parameters
Instruct   | Instruction-tuned variant
Q4_K_M     | Specific 4-bit quantization scheme
.gguf      | Saved in GGUF format

The filename still does not answer:

Whether it accepts images or audio.

Supported context length.

Current software compatibility with its architecture.

Actual runtime memory requirements.

Performance on specific tasks.

Keywords like VL, Audio, Coder in the name offer hints, but naming conventions vary. Always verify capabilities and compatibility via model documentation and inference framework docs.

08 Checklist Before Downloading a Model

Follow this practical order:

Model Task Matches Need
         ↓
Input/Output Modalities Supported
         ↓
Precision/Quantization Fits Available Memory
         ↓
File Format & Architecture Loadable by Inference Tool

This sequence is more useful than comparing parameter counts or file sizes first. Whether a model runs depends on matching capability, resource needs, and software support.

Conclusion

Model names are long but each segment answers a distinct question. LLMs, vision, audio, embedding, and reranker models describe tasks; unimodal vs. multimodal describes supported information forms; FP16, Q8, Q4 describe numerical precision and quantization; GGUF, Safetensors, ONNX describe storage and deployment formats. Understanding these dimensions separately clarifies the otherwise confusing soup of labels.