Does Gemini Pro Really Outperform GPT‑4? A Deep Comparative Review

Google recently launched Gemini Pro, followed by plans for a more powerful Gemini Ultra, sparking widespread interest in the AI community.

Comparisons with other advanced models, such as Claude 2 and Google’s Bard, raise the question of whether the Gemini series truly surpasses GPT‑4 in performance.

Bloggers cite benchmark tables claiming Gemini outperforms GPT‑4, but benchmarks can be biased if models are trained on the test data, so results should be interpreted cautiously.

Below is a summarized comparison of Gemini Pro and GPT‑4:

Reasoning ability: GPT‑4 is efficient and accurate; Gemini Pro often gives less precise answers.

Vision capability: GPT‑4 accurately understands images; Gemini Pro lags behind.

Token limit: GPT‑4 handles up to ~17,408 tokens; Gemini Pro has a lower limit, reaching its ceiling around 5,300 tokens.

Long‑text summarization: GPT‑4 produces clear, structured summaries; Gemini Pro can summarize YouTube videos but its text summaries are weaker.

Knowledge update: GPT‑4 is updated to April 2023; Gemini Pro’s last update is unclear.

Mathematical skills: GPT‑4 consistently solves math problems; Gemini Pro shows variability.

Web search: Gemini Pro leverages Google’s search for stronger results; GPT‑4 has limited built‑in search.

Logical problem example: "If student Jack moves from class A to B, does the average IQ of both classes increase?" Explanation shows how reasoning is evaluated.

GPT‑4 provides a concise, accurate answer, while Gemini Pro’s response is more complex and sometimes incorrect.

Common‑sense question: "How to measure 6 L using a 12 L and a 6 L jug?" The correct answer is simply to use the 6 L jug.

GPT‑4 answers correctly; Gemini Pro gives a wrong answer, as shown in the screenshots.

In image‑based tests, Gemini Pro misidentifies objects (e.g., calling a rhinoceros a turtle) and fails to capture humor, whereas GPT‑4 accurately interprets the content.

Token capacity is crucial for large language models. GPT‑4 can process up to ~17,408 tokens, enabling comprehensive summaries of long texts, while Gemini Pro reaches its limit around 5,300 tokens, limiting its ability to handle lengthy inputs.

When summarizing YouTube video transcripts, Gemini Pro produces a direct, less structured summary, whereas GPT‑4’s output is shorter but well‑organized with clear sections.

Gemini Pro can directly summarize YouTube videos—a feature unsurprising given Google’s ownership of YouTube—yet GPT‑4, especially with the VoxScript plugin, often yields more coherent summaries.

For recent topics like “how to create your own GPT model in ChatGPT,” Gemini Pro gives a decent answer but appears vague about its data sources, while GPT‑4’s training data lacks this information, highlighting GPT‑4’s advantage with web‑enabled capabilities.

Mathematical problem tests from UC Berkeley show GPT‑4 consistently delivering correct answers, whereas Gemini Pro sometimes omits correct options or provides incorrect ones.

In complex web‑search tasks, Gemini Pro excels, efficiently identifying sustainable packaging suppliers and presenting data in downloadable tables, while GPT‑4’s web‑search (even with plugins) falls short in generating comprehensive tables and price information.

Overall, Gemini Pro marks a significant leap from its predecessor Bard, offering more features and better performance in several areas, yet it still trails GPT‑4 in reasoning, vision, and handling extensive text.

The upcoming Gemini Ultra is expected to improve further, though it may still be slightly behind GPT‑4.

Free tools like Gemini, alongside accessible models such as ChatGPT and Claude 2, represent a major win for AI users, expanding diversity and practicality in the field.