Debugging LLM Adapter Parameter Passing: Config vs. Generate Call Mismatch

Troubleshooting Starts at the Call Site, Not at Config Echo

When a model repeats itself, developers often first adjust decoding configuration by adding repetition_penalty and no_repeat_ngram_size. However, if logs already show these parameters but the output still repeats, the next step is to verify whether the parameters actually reached the generate call.

Local HF Provider: The Adapter Layer Filters Parameters

The LocalHFProvider receives conversation, sampling config, and tool specifications, prepares model inputs, and calls the underlying generate. The wrapper explicitly extracts only four keys from the config: max_new_tokens, do_sample, temperature, and top_p. It passes these as individual arguments and does not spread the entire config dictionary.

Therefore, adding repetition_penalty and no_repeat_ngram_size to the config dictionary does not cause the wrapper to forward them. The parameter names may be correct, but they can stop at the boundary between caller and adapter.

To verify this, the author replaces the model and tokenizer with in-memory stubs that record the actual sampling arguments passed to generate. With a test config containing the four standard keys plus the two repetition-control keys, the recorded sampling arguments contain only the first four, while the returned sampling_config metadata still shows all six keys. This discrepancy shows that logs can reflect the caller's intent rather than the adapter's actual behavior.

Further tracing reveals that ModelOutput.metadata copies the returned object's config into metadata, and the run loop writes this metadata into step-by-step output logs. The same keys appearing in multiple log entries may simply be copies of a single echo, not independent confirmations. Field comments in the return structure describe the config as "parameters used for this run," but the actual assignment comes from the original config dictionary. When comments diverge from the execution path, the execution path must be trusted.

The conclusion: the repetition-control keys were not explicitly forwarded by this wrapper. The underlying model may have its own generation_config with default values for these keys, but that was not verified because no real model was loaded.

Empty Config Counterexample: Defaults Still Apply

When the config is an empty dictionary, the wrapper still constructs the four sampling arguments with defaults: max_new_tokens=512, do_sample=true, temperature=0.7, top_p=0.9. However, the returned metadata shows an empty config. Relying solely on that log would hide the four defaults and make it impossible to tell whether two runs used the same settings. An empty config expresses the caller's lack of explicit values, not the absence of sampling settings.

Numeric types also differ: the test config writes numbers as strings, the wrapper converts them to integers or floats before passing downstream, but the metadata echo retains the original strings. Even with identical key names, the two records may not describe the same value representation.

A proper audit trail should capture both the caller's input (intent) and the adapter's constructed arguments (behavior), along with the source and version of any defaults. Not every internal object needs to be logged, but the parameters relevant to the decision should be recorded with their capture point noted.

Temperature Zero Does Not Automatically Disable Sampling

The same wrapper reads temperature and do_sample independently and converts their types separately. In a test config containing only temperature=0, the captured temperature is 0.0 while do_sample remains the default true. The temperature knob turns, but the sampling switch stays unchanged. How the real generation library handles this combination depends on the model, library version, and execution path — the stub does not validate legality or simulate sampling.

Thus, setting temperature to zero cannot be assumed to switch decoding modes. If the goal is to change the decoding strategy, the relevant switches and their adaptation logic must be verified together, and the downstream acceptance must be checked. Comparing output changes before confirming the parameter-passing contract risks comparing two unrecorded configurations.

API Request Construction: Server-Side Acceptance Still Unverified

The APIProvider merges the sampling config into the request body. With the same test config, the two repetition-control keys do appear in the serialized JSON, unlike the local path. However, the verification uses a placeholder endpoint with no credentials; the stub captures the request bytes and immediately raises a manual exception. No HTTP request is sent, and no server response is received.

This records only request construction and call attempt. It does not prove the server received, supported, accepted, or used those keys. Even a successful response later would require service contracts and evidence that the parameters entered the actual generation algorithm.

Merging dictionaries introduces a concrete issue: sampling config keys can overwrite the initial model name and message fields. When both are placed in the config, the final request body adopts the config's values. Treating the config as merely a set of sampling knobs ignores its role in constructing the entire request.

Tool field handling also differs by branch: explicit non-empty tool lists are processed first and then overwrite the config's tools field; without an explicit list, the config's original tools are retained. Tool choice and parallel tool flags use default-filling logic that depends on the presence of tools. Field names do not automatically gain cross-backend semantics: max_new_tokens remains max_new_tokens in the JSON and is not renamed to max_tokens. String numeric values stay strings in the request body, whereas the local HF path converts them. Serialization completion does not guarantee the server's type expectations.

These observations do not declare one path more correct; they show that a unified interface still hides different adaptation rules. Switching backends requires re-verifying request structure, defaults, and override precedence — not assuming a single config file guarantees identical experimental conditions.

The Maintenance Cost of Strict Parameter Checking

One improvement is to have adapters explicitly declare supported keys and error on unknown keys at call time; another is to log ignored keys alongside constructed values. This prevents developers from unknowingly comparing effects with parameters that were never forwarded.

Strict checking has costs: backends support different key sets, versions add fields, and services have proprietary extensions. Allow-lists and transformation rules must track actual contracts; bluntly rejecting all unknown keys may block legitimate new capabilities.

Therefore, checkpoints should sit at the adaptation boundary and document which backend and version the rules apply to. Preserve the original config before the call, record the adaptation result at the call site, and separately record what the service confirms. Multiple pieces of evidence have distinct locations and must not be merged into a single "already effective" echo.

These checks work best with small, fixed regression inputs. When adaptation rules change, the same test config can re-verify which keys are selected, how defaults are filled, and whether override order changed. This validates the parameter-passing contract; actual generation quality is evaluated separately with task samples — two checks answering different questions.

Only after confirming parameter delivery can effect experiments begin. With fixed input, model, and other conditions, comparing output before and after repetition-control changes can show whether the technique helps for the task. Successful delivery is a prerequisite for comparison; it does not itself prove reduced repetition or improved fine-tuning quality.

When logs show parameters but behavior does not change as expected, trace the layer where keys are selected, defaulted, overridden, or dropped. Clarifying these actions determines whether the next step is to fix the adapter, check service rules, or adjust generation settings.