Why a Single -100 Line Determines Who the Multi‑Round SFT Learns to Speak

The author begins with a conclusion: the most hidden mistake in multi‑round supervised fine‑tuning (SFT) may not trigger a training error. Data can be read, tensor dimensions are correct, and loss is computable, yet the model might be asked to predict a segment other than the intended assistant output.

In PyTorch’s cross‑entropy loss, a label value of -100 is ignored. Tokens at those positions do not generate a direct loss term and are not treated as prediction targets, but they remain in the input sequence, providing context for subsequent predictions.

A full dialogue containing system , user , assistant and tool messages is usually not required to have every token reproduced by the model. The system defines behavior, the user asks a question, the tool returns factual data, and only the assistant’s response is the generation target. Including the user in labels would force the model to predict how the user asks, turning informal language, typos, and random phrasing into supervision signals.

Masking does not mean exclusion from computation. For example, the token currency: USD in a tool response should not be generated, yet it must influence whether the assistant later mentions USD. The -100 mask removes the direct scoring for that position but does not delete it from the computation graph.

Printing input_ids and labels side by side makes the masking logic clear: both have the same length, assistant‑position labels retain the original token IDs, while all other roles are replaced with -100. Thus, user and tool tokens, although not answers, can still affect parameter updates through the assistant’s loss.

How to Locate Assistant Spans

After applying a chat template, the dialogue becomes a flat token sequence with no explicit “assistant starts here” marker. A common method is prefix‑difference: render the first i messages, then the first i+1 messages; the length difference indicates the span of the i+1 ‑th message.

for i, msg in enumerate(messages):
    prefix_ids = apply_template(messages[: i + 1])
    cur_len = len(prefix_ids)
    if msg.get('role') == 'assistant':
        assistant_spans.append((prev_len, cur_len))
    prev_len = cur_len

Once all assistant spans are identified, two training scopes are common: only_last_assistant=True – supervise only the final reply.

Supervise every assistant segment, including tool‑call outputs.

The prefix‑difference also captures role markers, newlines, and end‑of‑sequence tokens. Whether the start token participates in loss must match the inference template; the end token must never be truncated. If the start falls inside a user message, user text may unintentionally become a target, so boundaries must be verified against the actual template.

Choosing between the two scopes depends on data reliability. If intermediate rounds are exploratory outputs of an older model, supervising only the final, human‑verified answer is safer. If every tool selection, parameter fill‑in, and intermediate reply has been vetted, supervising all assistant segments can provide richer signals. The key is that every loss interval must represent a trustworthy demonstration.

Masking must also be preserved during batch assembly: pad tokens in input_ids receive the pad token, while the corresponding positions in labels must remain -100. Otherwise short samples would learn to output pad tokens. When truncating overly long dialogues, the truncation strategy (e.g., keeping the last answer) can drop early conditions, and overly long assistant spans may be cut off, removing part of the supervised region.

Character‑Level Demonstration of Mask Shifts

A character‑level demo treats each Unicode character as a token and uses a fixed Qwen‑style template, avoiding a real tokenizer while clearly showing how spans move.

Note that the numbers 360, 68, and 179 are character counts, not real BPE token counts, and should not be used to estimate training cost.

The sample contains five parts: system prompt, user request for a refund, assistant invoking oms.get_order, tool returning “60 USD”, and the final assistant answer.

Training only the last assistant segment yields:

seq_len=360  loss_tokens=68  masked_out=292
 loss segment [292:360] ->
 '<|im_start|>assistant
已核实订单，按实付 60 USD 为你办理退款……<|im_end|>
'

Here only the final 68 characters are direct supervision targets; the preceding four parts serve as context. Training all assistant segments changes loss_tokens to 179, adding 111 characters from the first assistant turn (including tool call text and template markers).

More supervised characters do not automatically mean better performance. If the intermediate tool call is unreliable, including it in loss can reinforce incorrect behavior. Conversely, if tool usage is the core capability to learn, omitting it loses essential supervision. The numbers merely indicate how the supervised region changes; quality assessment still requires examining data generation and verification processes.

The boundary of tool data is clarified: <tool_call> is generated by the assistant and can be a direct supervision target; the tool’s execution result belongs to the tool message, receives -100, yet still influences whether the assistant later says “60 USD”.

Longer multi‑turn samples contain more tool returns, widening the gap between total sequence length and directly supervised tokens. When reporting dataset statistics, it is advisable to log loss_tokens separately, as it reflects the true amount of answer content the model is trained to produce.

Why a 22‑Character Shift Can Slip By Silently

The prefix‑difference assumes that rendering each prefix independently yields exactly the same token sequence as the corresponding prefix of the full sequence. Any discrepancy—such as adding an extra generation prompt <|im_start|>assistant\n to each prefix—shifts the start of the final assistant span by 22 characters and pushes the end beyond the sequence, reducing loss_tokens from 68 to 46.

Python slicing silently drops out‑of‑range indices, so the assistant’s beginning is omitted from supervision without raising an error. This experiment shows that a misaligned mask can silently affect training, though its impact on downstream metrics still needs verification with a real tokenizer and controlled experiments.

To catch such misalignments early, compare each prefix_ids with the first len(prefix_ids) tokens of the full sequence. If they differ, stop generating labels. Equality of length alone is insufficient; the token IDs must match.

In practice, offsets can arise from changes to the chat template, special token rules, rendering of the “think” segment, or the introduction of unfamiliar roles. Even if the training script is unchanged, labels may no longer align.

Four Pre‑Training Assertions

Non‑zero check: loss_tokens must be greater than 0; a fully -100 label set indicates missing assistant content or unrecognized roles.

Boundary check: decoded label spans must lie entirely within an assistant message, respecting start and end token conventions.

Scope declaration: explicitly decide whether only the last assistant turn or all assistant turns are supervised; do not let default parameters dictate data acceptance rules.

Template consistency: after any change to the template, tokenizer, max length, or role set, re‑sample and re‑run the checks.

Embedding these four checks into data‑preprocessing tests—using a small, fixed set of samples covering single‑turn replies, multi‑turn dialogues, tool calls, and over‑length truncation—ensures that any shift in loss intervals, start/end positions, or decoded content is detected before training begins. The cost is low, yet it prevents the costly illusion of training progress while the model learns the wrong target.

Finally, a successful training launch only proves that tensor shapes are compatible; a decreasing loss does not guarantee that the supervised region is correct. Decoding the labels to verify who the model is being asked to predict should precede any discussion of epochs, learning rates, or final performance.