Deploy Jev-Omni Locally: Multimodal Decision Model Tutorial with Benchmarks

Multimodal Decision Model: Jev-Omni

Jev-Omni is a discriminative decision model for multimodal inputs. Unlike the text-only Jev, it extends input dimensions to directly accept images, audio, and video while outputting classification judgments. Built on the Gemma 4 12B IT base model, it underwent ~30k fine-tuning examples. Official benchmark results show near 90% accuracy with very low per-call cost, positioning it for cost-sensitive, high-frequency scenarios where acceptable probability judgments replace token-by-token generation.

Calibration on DecisionBench Medium yields an ECE of 0.04, with the calibration curve closely tracking the diagonal, meaning a predicted 0.9 confidence corresponds to ~0.9 empirical accuracy. This allows business systems to use output probabilities directly as gating thresholds without recalibration.

Comparison with Laya

A table contrasts Jev-Omni with Laya (covered in the previous article). Laya targets ultra-low latency and high concurrency for pure text classification; Jev-Omni handles multimodal inputs at hundreds of milliseconds to seconds latency. Choose Laya for pure text speed; choose Jev-Omni when multimodal input is required.

Step-by-Step Deployment

1. Environment Preparation

Requires a CUDA GPU; the official loader does not support CPU-only execution. FP32 decision backbone weights (~48 GB) plus the Gemma 4 base (~24 GB BF16) peak at ~60 GB memory during loading, so ≥60 GB unified memory is recommended. System must have ffmpeg installed for audio/video preprocessing. Python ≥3.10 (tested on 3.12). Create an isolated virtual environment:

# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

Download requirements.txt from the experiment repo and install: pip install -r requirements.txt Two version constraints are critical: transformers==5.17.0 (lower versions cannot import Gemma 4 architecture) and torch / torchvision matched to the local CUDA driver. For Chinese network environments, set Hugging Face mirror and disable Xet protocol:

export HF_ENDPOINT=https://hf-mirror.com
export HF_HUB_DISABLE_XET=1

2. Download Weights

Total ~72 GB: 48 GB FP32 decision backbone (13 shards) + 24 GB Gemma 4 base (BF16). Use the provided script with --mirror for domestic acceleration: python scripts/download_models.py --mirror Optionally specify --target-dir; later pass matching --models-dir to verification scripts. Weights are saved to models/ (or HF default cache).

3. Prepare Test Assets

Download three free, commercially usable assets via script: python scripts/download_assets.py This populates assets/ with: cat.jpg (tabby cat photo) conversation.mp3 (~12s English dialogue) person-reading.mp4 (~19s close-up video)

Each comes from independent sources (Flickr, Freesound, Pexels) with hash verification. For full benchmark evaluation, add --full to fetch 10 image classes and 3 audio clips under assets/eval/.

4. Minimal Verification

Run the quickstart script (ensure ffmpeg is in PATH): python scripts/quickstart.py First load takes ~8 minutes (423–449 seconds) to read 48 GB FP32 shards from disk. After loading, the script runs five test cases across all four modalities.

Understanding the Output

Example call for Chinese ticket classification:

clf.predict(
    state="我这个月信用卡被扣了两次费，请尽快把多扣的钱退给我",
    question="这个请求应该由哪个部门处理？",
    options=["账务", "技术", "销售", "其他"]
)

Output is a structured dict (no free-text generation):

{
  "prediction": "账务",
  "prediction_index": 0,
  "confidence": 0.997093677520752,
  "probabilities": {
    "账务": 0.997093677520752,
    "技术": 0.000299832783639431,
    "销售": 0.0003560602490324527,
    "其他": 0.002250370336696055
  }
}

Key fields: prediction (top option), prediction_index, confidence (probability of top option), probabilities (full distribution). The structured dict eliminates regex parsing and JSON syntax errors. In production, confidence serves as a routing threshold: above threshold → auto-route; below → escalate to larger model or human review.

Measured on GB10: text/image/audio inference <1s; video (16 frames) ~4s. First inference includes warm-up overhead; a dry run is recommended before production traffic. Full timing logs are in the repo.

How One Model Supports Four Modalities

Jev-Omni leverages the Gemma 4 backbone. Each modality passes through a lightweight projector into a unified token space, then concatenated with text tokens into a single sequence fed to the same language model. A classification head atop produces option probabilities. This avoids heavy separate vision/audio towers, keeping end-to-end latency low, at the cost of limited deep feature extraction for complex images or long audio.

Latency table (GB10 measured percentiles vs. official H200 benchmarks) shows modality-specific times. Official H200 numbers use optimized backend, warmed-up median only, excluding preprocessing/network. Hardware differences account for the gap.

Experiment Repository & Scripts

All scripts, configs, and raw terminal logs are in the GitHub repo:

git clone https://github.com/li-xiu-qi/XiaokeAILabs.git
cd XiaokeAILabs/experiments/test_jev_open_source/jevomni

Core scripts include environment setup, weight/assets download, quickstart, latency benchmark, and batch evaluation. Execution logs are stored under experiments/test_jev_open_source/docs/执行日志/. Running the scripts auto-creates models/, assets/, and results/ directories locally.

Conclusion

Multimodal decision is a practical need: detect presence of a person in frame, abnormal sound, camera motion — all with clear options and a confidence score, not generative descriptions. Jev-Omni unifies text, image, audio, video under one decision interface and releases weights for low-cost local validation. It is not universal; best suited for scenarios with fixed option sets requiring confident classification. Recommendation: set up the environment, run the minimal script, then test with your own real data to decide its role in your pipeline.