VideoJev: 100x Faster Video QA by Scoring Candidates Instead of Generating Answers

Problem: Autoregressive Generation Slows Video QA

Video understanding models encode video quickly, but answering a question is slow because most models use autoregressive generation to produce the answer token by token — even when the answer is a single letter like "A" or "B".

Method: Unified Candidate Scoring

VideoJev reframes both temporal grounding and multiple-choice QA as a candidate-scoring task. Given a video and a set of candidates (time intervals or answer options), the model outputs a matching score for each candidate in a single forward pass. The video is encoded once; multiple candidates are scored in parallel.

Results: Speed and Accuracy

On Charades, ActivityNet, and QVHighlights temporal grounding benchmarks, VideoJev-4B and VideoJev-35B-A3B improve average mIoU by 6.7 and 10.8 percentage points over their respective base models.

On four video QA benchmarks, performance matches the thinking-mode base models, but inference speed is 100-129x faster.

Measured latency on H200: VideoJev-4B Joint mode 0.276 s/query vs. Qwen3.5-4B Instruct 0.763 s; VideoJev-35B-A3B Joint mode 0.336 s vs. base 1.054 s.

For 10 temporal grounding queries, VideoJev takes 0.39 s total, while a generative baseline requires 5.98 s.

Inference Modes

Joint : a batch of questions processed in one forward pass.

Cached : video embeddings are cached; each new question needs only one additional forward pass.

Limitations

The paper acknowledges confidence calibration issues: average calibration error of 8-9 points, with the model being overconfident. When the query content is absent from the video (e.g., searching for a dog in a cat-only video), the decoder still forces an interval prediction, lacking a "rejection" mechanism for material-search scenarios.

External Validation

A developer of the Remotion video QA tool reported that a similar multimodal decision model replaced per-frame LLM review, reducing per-video QA cost from $127 to $0.16 and cutting time by three quarters.

Open-Source Resources

HuggingFace model collection: https://huggingface.co/collections/MCG-NJU/videojev GitHub repository: https://github.com/MCG-NJU/VideoJev Paper:

https://arxiv.org/abs/2610.18615