Kimi K3 vs GPT‑5.6 Sol: A Full‑Scale Comparative Evaluation

Overview

Moonshot’s newly released open‑source model Kimi K3 (2.8 trillion parameters) is benchmarked against the top‑tier closed‑source model GPT‑5.6 Sol. The authors design a two‑part experiment: (1) reproducing five classic childhood games to evaluate 3D game‑development capability, and (2) assigning Kimi K3 a non‑game full‑stack project (the Paper‑Graph‑Manager system) to test end‑to‑end backend and frontend automation.

Evaluation Criteria for Games

The models are judged on five dimensions: (a) can the code run, (b) visual completeness, (c) control feel, (d) rule completeness, and (e) perceived intelligence.

Game‑by‑Game Findings

1. Plants vs. Zombies

Kimi K3 produces a single‑file version with a green lawn, zombie waves and potato‑mine explosions but lacks level selection, pause, star rating, and win‑loss statistics. GPT‑5.6 Sol adds a level‑select screen, star evaluation and win‑loss tracking. Code‑wise K3 inlines all logic in one file; Sol splits the logic into modules and adds default save‑game handling.

2. Metal Slug

K3’s build shows custom character rendering but only two enemy types and ends the game immediately on death. Sol implements checkpoints that allow continuation after death and uses rectangular colliders, aligning more closely with standard game architecture. Sol’s enemy set is richer and the experience feels closer to the arcade original.

3. King of Fighters ’97

K3 offers character selection and difficulty settings with dense special‑move feedback, yet it lacks stage selection, health/time/damage multipliers and persistent settings. Sol provides the missing options, persisting rules and delivering a more repeatable fighting‑game loop.

4. Contra

K3 launches directly and the shooting/movement works, but there is no title screen, level selection, pause or resume. Sol adds a full arcade‑style flow with platform collision and a richer enemy set.

5. Tank Battle

K3 includes core mechanics (base protection, upgrades, mines) but omits menus, pause, level return and validation. Sol supplies a complete menu system, pause menu, settlement screen and level‑config validation, making it feel like a long‑term playable title.

Overall Model Comparison

K3’s strengths are immediate launchability, solid core mechanics, strong visual highlights in some games, and occasional single‑point advantages (e.g., character rendering). Its weaknesses are generally incomplete rule sets, missing UI flows, high death penalties and uneven UI polish. Sol delivers a stronger sense of a finished product by investing in menus, rule persistence, boundary checks and screen effects. In efficiency, Sol finishes the tasks about 25 % faster and consumes fewer tokens, but its API pricing and cross‑border access costs are higher for Chinese developers, whereas K3 is directly accessible despite higher token usage per run.

Full‑Stack Automation Test (Kimi K3)

The authors give K3 a non‑game task: clone https://github.com/li-xiu-qi/paper-graph-manager, install dependencies, start the FastAPI backend, add a database table, create graph‑generation endpoints, and verify the frontend rendering without any manual debugging.

Steps performed by K3:

Run uvicorn main:app --host 0.0.0.0 --port 8001. K3 identifies three missing modules (dotenv, pyvis, kimi_agent_sdk), installs them, and the service becomes healthy at http://localhost:8001.

Modify database.py to add a paper_references table with composite primary key, foreign key and index.

Update graph.py to implement build_reference_graph() and export_reference_html().

Extend main.py with four new reference endpoints and two graph endpoints.

Execute an 8‑step end‑to‑end test suite on port 8002; all 134 tests pass except two legacy failures unrelated to the new functionality.

Start the Vite frontend; encounter a path‑alias error (

Failed to resolve import "@/lib/utils" from "src/components/ui/dialog.tsx"

), a typical engineering issue.

Use Kimi Code’s WebBridge to inject JavaScript that measures rootLen. The counter grows from 0 to 30081, confirming successful page rendering.

Final dashboard displays total papers, annotated count, coverage, recent papers (e.g., “Mistral 7B”, “Verifiable Fully Homomorphic Encryption”) and entry points for paper management, knowledge graph, chat and notes.

The test demonstrates that K3 can distinguish environment problems from code bugs, design a database schema, generate REST APIs, and verify the front‑end rendering automatically.

Conclusion

Kimi K3 achieves impressive prototype quality in both game generation and full‑stack engineering, reaching roughly 90 % of the performance of leading closed‑source models in the game cases. It excels at rapid, playable prototypes but still lags behind GPT‑5.6 Sol in product‑level polish, rule completeness and engineering conventions. For developers seeking an open‑source model that can independently deliver end‑to‑end functional code, K3 is a strong candidate; for those needing a ready‑to‑ship experience, GPT‑5.6 Sol remains superior.