Why Large Language Models Aren’t Magic: Understanding Compression and Prompt Engineering

Large Models as Compression Systems

Large language models (LLMs) are fundamentally very large neural networks that perform a massive compression of raw text data. A simple lossless compression example (AAABBBB → 3A4B) illustrates the principle: repeated patterns are replaced by shorter codes, reducing storage from 1 GB to 500 MB. In the same way, training corpora for modern LLMs can be on the order of tens of terabytes (e.g., Llama 3 uses ~15 TB of text). After training, the model parameters occupy only a few gigabytes (≈10 GB), representing a compression ratio of roughly 1 000 : 1 while preserving the statistical structure of the data.

Learning Abstract Patterns

During training the model adjusts weights and biases to capture not only literal token sequences but also higher‑level relationships such as logical syllogisms. For instance, from sentences like “Socrates is human, humans die → Socrates dies” the model internalizes the rule “if X is human and humans die, then X dies”. After training, the parameters are fixed and can be used for inference: given a novel prompt “Merton is human, humans die”, the model can infer “Merton will die” even though the name “Merton” never appeared in the training data.

Deterministic Inference and Sampling Strategies

The matrix‑multiplication that produces the logits is deterministic; randomness only appears when the logits are sampled to generate a token sequence. Common sampling methods are:

Greedy decoding – always pick the highest‑probability token (deterministic but low diversity).

Beam search – keep multiple high‑probability sequences and select the best final output.

Temperature scaling – adjusts the sharpness of the probability distribution; lower temperature makes the distribution peakier, higher temperature yields more diverse outputs.

Top‑k sampling – restricts the choice to the k most likely tokens before sampling.

These techniques trade off coherence against diversity.

Prompt Engineering Hierarchy

Prompt engineering is often the most cost‑effective way to extract value from LLMs, especially when only a few hundred words of domain knowledge are needed. Fine‑tuning is reserved for large, vertical domains because it can cause catastrophic forgetting of previously learned capabilities.

L1 – Simple Question‑Answering

One‑shot prompts that ask a single question and expect a direct answer. Typical use cases include chat, translation, code generation, and summarisation.

L2 – Compositional Prompts

Complex tasks are decomposed into a sequence of prompts that interact with external tools. Example workflow:

Prompt the model to retrieve the latest three‑year loan rates for four banks using a search‑engine tool.

Prompt the model to format the retrieved data for a chart‑generation tool.

Prompt the model to render the chart and return the image URL.

This pattern follows the ReAct loop (Think → Act → Observe) and is useful for retrieval‑augmented generation (RAG), data visualisation, and multi‑step reasoning.

L3 – Agent Architectures

For highly complex pipelines (e.g., end‑to‑end code development, testing, and deployment), the overall task is split among several specialised LLM agents, each with its own context and toolset. A central dispatcher routes sub‑tasks to the appropriate agent. Benchmarks such as SWE‑bench show that multi‑agent approaches can dramatically improve performance on real‑world software engineering problems.

Practical Considerations and Challenges

Rapid prototyping with LLM APIs is straightforward; many prototypes can be built within a half‑day.

Token‑budget and latency become limiting factors when many prompts are chained; global optimisation of prompt flow is required.

Performance tuning includes model quantisation, caching, GPU acceleration, and streaming responses.

Prompt optimisation relies heavily on human expertise and extensive case studies; automated tools can assist but do not replace expert iteration.

Understanding the compression analogy, deterministic inference, sampling mechanisms, and hierarchical prompt design equips practitioners to build effective, scalable AI‑driven solutions.