How to Quickly Validate LLM Capabilities Without Standard Benchmarks

Run Scores Can't Be Trusted

Where scores exist, models are tuned to the leaderboard; the benchmark ceiling often caps scores more than the model’s true ability.

Problems with Standard Test Sets

Problem 1: Test items are in the training data

Many public test sets have been widely circulated; models likely have seen the questions or similar solutions during pre‑training. Models perform worse on questions dated after the knowledge cutoff, indicating reliance on memorization rather than generalization.

Problem 2: Test scenarios differ from real development

HumanEval’s 164 tasks focus on writing a function from a docstring, which is rare in practice. Real work involves reading existing code, debugging, refactoring, handling third‑party compatibility, or turning a textual requirement into a runnable module.

Problem 3: Scoring metrics ignore engineering quality

Pass@1 only tells whether code passes the given test cases, not whether it is robust, memory‑safe, or production‑ready. For example, two solutions may both pass tests, yet one OOMs on 100 k records while the other does not—Pass@1 cannot reveal this.

Self‑Verification

Basic Principles for Question Design

Questions must be unpublished; do not copy directly from LeetCode or similar platforms. You may reuse the problem type but change I/O formats and constraints.

Provide an executable verification method; the model’s answer must be runnable and automatically judged correct.

Cover multiple dimensions, not just “code runs”.

Include a difficulty gradient; medium‑difficulty tasks give the most discrimination.

Record first‑round results; do not keep prompting until the model produces a correct answer, as that measures prompt‑engineering skill rather than baseline capability.

Potential Pitfalls

Environment dependency completeness: the model should supply a requirements.txt or at least comment required package versions (e.g., pandas>=2.0).

Proactive side‑effect warnings: the model should flag operations that modify production data and suggest backups.

Performance boundary warnings: indicate possible OOM or recursion depth issues that may not appear on small test data.

Test Types

Type 1: Custom Algorithm Implementation

Derive an algorithm problem from your own business scenario, modify I/O formats, and ask the model to implement it. Because the problem is unpublished, the model must reason rather than recall.

Does it handle empty input or malformed formats?

Is the code logic clear and variable naming expressive?

If multiple solutions exist, does it explain the chosen approach?

Example: "Given an order list (possible duplicate order IDs), find all order pairs whose amount difference does not exceed N yuan, sorted by order ID lexicographically."

Type 2: Bug Fix with Real Context

Provide a real code snippet that contains a known bug together with the full stack trace. The model must locate and fix the bug.

Is the root cause identified?

Does the fix introduce new issues?

If several fixes are possible, does the model justify its choice?

Typical bugs include race conditions, floating‑point precision errors, or behavior changes due to third‑party library versions.

Type 3: Ambiguous Requirement to Code

Give a deliberately vague requirement and observe whether the model asks clarifying questions before coding.

Does it seek clarification first?

If it proceeds, are its assumptions documented in comments?

How does the solution change after follow‑up questions?

Example: "I need a service that handles user‑uploaded files."

Type 4: Code Refactoring and Optimization

Supply a 400–600‑line real code base with hidden structural problems and ask the model to refactor without explicit hints.

Is the resulting code size reduced (beyond mere formatting)?

Are core structural issues (e.g., a 200‑line function doing five tasks) identified?

Is behavior preserved and are regression‑test suggestions provided?

Type 5: Cross‑Language Migration

Ask the model to translate a business‑logic snippet from language A to language B using idiomatic constructs of the target language.

Are error‑handling conventions respected?

Does it leverage the target language’s standard library instead of re‑implementing functionality?

Are type system, generics, and concurrency semantics used correctly?

Example: converting Python async code to Go goroutine style, or mapping Python decorators to Go middleware.

Type 6: Self‑Written Tests

After the model writes code, ask it to design test cases covering edge conditions (empty input, huge input, special characters, concurrency, type limits). Then probe the model: "Which boundary conditions do you think your code might fail on?"

Does it give concrete, justified weak points?

Does it give vague statements ("it should be robust")?

Does it deny any issues?

Accurate self‑assessment indicates the model understands execution semantics beyond syntax.

Scoring Dimensions and Method

Correctness : code runs and passes your own tests.

First‑Round Hit Rate : proportion of tasks solved correctly without follow‑up.

Robustness : handling of edge cases, no crashes, accurate self‑prediction.

Engineering Quality : completeness of environment dependencies, proactive side‑effect warnings, reasonable code structure.

Efficiency : output redundancy and response latency.

Anonymous Blind‑Testing Procedure

Normalize comment styles across model outputs.

Strip all explanatory text outside code blocks.

Assign anonymous letters (A, B, C…) to each output.

Time and Cost Considerations

Waiting time and API fees are often ignored. For efficiency‑focused developers, a model that takes 20 seconds longer and adds 30 % redundant comments may be sub‑optimal despite similar code quality.

TTFT (time to first token) over 5 seconds feels sluggish; over 10 seconds breaks workflow. Inference models usually have higher TTFT because they perform internal reasoning before output.

Output redundancy: Model A produces 80 clean lines, Model B 120 lines with 40 lines of explanations and boilerplate. Identical code quality means extra reading and cleaning time for Model B, which accumulates over many daily interactions.

Rough cost estimate: compare total character count of outputs for the same prompt, multiply by model price per token to gauge suitability for high‑frequency use.

Simple SOP

Combine the above into a step‑by‑step workflow.

Step 1: Prepare Question Bank (1–2 h)

Craft 6–8 questions covering the test types you need.

Step 2: Execute Tests (2–3 h)

Prompt each model with identical prompts for each question.

Record first‑round output, need for follow‑up, TTFT, and total output length.

Apply the anonymous processing described earlier.

Step 3: Independent Scoring

Score each answer on the five dimensions, then compare distributions rather than only total scores.

Step 4: Apply Fallback Rules

If a model fails to produce runnable code after three follow‑ups, mark the question as unusable for that model.

Step 5: Aggregate and Conclude

Summarize per‑model scores; models with similar total scores may differ markedly on engineering quality or latency, guiding selection for specific project constraints.

Conclusion

Benchmarks are useful for coarse filtering but should not be the sole basis for model selection. Testing with problems that reflect your actual workflow and scoring with clear engineering criteria yields more reliable decisions.