How a New AI Probe Can Reverse‑Engineer LLM Parameter Counts

Background

In the AI community, the size of large language models (LLMs) is often kept secret. OpenAI does not disclose GPT‑4’s parameter count, Google is silent on Gemini, and many domestic providers only highlight successes.

Uncompressible Knowledge Probe

Researcher Li Bojie introduced the “Uncompressible Knowledge Probe” framework. The core idea is to measure how much “hard knowledge” a model can retain rather than estimating training data volume. Larger models can store more uncompressible knowledge fragments.

How the Probe Works

Li designed a set of highly random questions that are unlikely to appear in training data. If the model answers correctly, it demonstrates the presence of uncompressible memory. By aggregating pass rates across probes of varying difficulty, one can infer an approximate range of the model’s effective parameter count.

Black‑Box API Only

The method requires only access to a model’s public API; no weights, architecture details, or training information are needed. Users submit the “knowledge exam” to the model and read the score to estimate its “brain capacity”.

Community Reaction and Implications

The approach gives developers an independent way to verify vendor claims and sparks debate about model evaluation standards. Traditional benchmarks can be gamed, whereas this probe targets the intrinsic memory‑vs‑understanding distinction.

Analysis of Memory vs Understanding

According to the author, a 10‑billion‑parameter model will inevitably perform worse on completely novel, random knowledge than a 20‑billion‑parameter model, regardless of data volume. The probe therefore measures a “soft‑skill” boundary under a hard‑parameter constraint.

Limitations

The estimate reflects “effective parameters” rather than the actual count. Under‑trained large models may not fully utilize their capacity, while highly compressed small models can exhibit unusually high knowledge density in specific domains.

Industry Impact

If users can infer model size from API calls, vendors may feel pressured to disclose parameters to maintain trust. Over time, more providers might publish sizes voluntarily.

Conclusion

The “Uncompressible Knowledge Probe” is still in early discussion, but it raises broader questions about how to define, measure, and trust AI models beyond raw performance.