Understanding EOS, stop_token_ids, and stop_sequences in vLLM

Core conclusion

In vLLM, EOS stops generation at the token level inside the Engine (process A), while stop strings are matched at the text level in the API server (process B). EOS uses a token ID, stop_token_ids also use token IDs, and stop/stop_sequences use decoded strings.

API entry point

The relevant fields in protocol.py are:

stop: str | list[str] | None = []
stop_token_ids: list[int] | None = []
include_stop_str_in_output: bool = False
ignore_eos: bool = False

These are packed into a SamplingParams object by to_sampling_params(). Anthropic maps stop_sequences to the same stop field.

EOS implementation

The EOS token ID is injected into SamplingParams by InputProcessor before the Engine runs:

sampling_params.update_from_generation_config(
    self.generation_config_fields,
    self.renderer.get_eos_token_id(),
)

During request processing, update_from_generation_config() stores the EOS ID in _eos_token_id and adds it to _all_stop_token_ids (used by MinTokensLogitsProcessor to mask the EOS token when min_tokens is not yet satisfied). The Scheduler calls check_stop() after each new token:

def check_stop(request, max_model_len) -> bool:
    if request.num_output_tokens < sampling_params.min_tokens:
        return False
    last_token_id = request.output_token_ids[-1]
    if last_token_id == sampling_params.eos_token_id:
        request.status = RequestStatus.FINISHED_STOPPED
        return True
    if last_token_id in (sampling_params.stop_token_ids or ()): 
        request.status = RequestStatus.FINISHED_STOPPED
        request.stop_reason = last_token_id
        return True
    if request.num_tokens >= max_model_len or request.num_output_tokens >= request.max_tokens:
        request.status = RequestStatus.FINISHED_LENGTH_CAPPED
        return True
    return False

If ignore_eos is true, the EOS check is skipped, allowing the request to continue until max_tokens is reached.

Stop‑string implementation

The Scheduler never reads stop strings. Instead, the Detokenizer in process B loads the stop list, computes a buffer length ( max(len(s) for s in stop) - 1), and after each token batch calls check_stop_strings() on the decoded text:

def check_stop_strings(output_text, new_char_count, stop, include_in_output):
    for stop_str in stop:
        stop_index = output_text.find(stop_str, 1 - new_char_count - stop_string_len)
        if stop_index == -1:
            continue
        if include_in_output:
            stop_index += stop_string_len
        return stop_str, stop_index  # truncate output_text

If a stop string matches, the Detokenizer returns finish_reason = FinishReason.STOP and stop_reason = stop_string. Because the match happens after decoding, the API server may need to abort the Engine if it has already generated extra tokens:

if not engine_core_output.finished:
    reqs_to_abort.append(req_id)
await engine_core.abort_requests_async(reqs_to_abort)

Buffering for streaming

When a stop string is not included in the output ( include_stop_str_in_output=False), the Detokenizer withholds the last len(longest_stop)-1 characters to avoid sending a partial stop token to the client.

Examples

EOS example : The model emits token 151645 (EOS). Scheduler marks the request FINISHED_STOPPED. Detokenizer streams the text and returns finish_reason="stop" with stop_reason=null.

stop_token_ids example : Request includes "stop_token_ids": [151643]. When token 151643 appears, check_stop() stops the request, sets stop_reason=151643, and the API returns finish_reason="stop".

stop string example : Request includes "stop": ["User:"]. The Scheduler never stops; after decoding, the Detokenizer finds the substring "User:" and truncates the output, setting finish_reason="stop" and stop_reason="User:". The Engine may be aborted to discard any extra tokens generated after the match.

ignore_eos example : With "ignore_eos": true, the EOS token is ignored; generation continues until max_tokens is hit, yielding finish_reason="length". This mode is useful for fixed‑length benchmarking.

Comparison table (textual)

OpenAI vs Anthropic parameters : OpenAI uses stop_token_ids and stop; Anthropic uses only stop_sequences. Both configure the stop behavior via the API request, but the Engine checks token‑level stops, while the API server checks string‑level stops.

Where the stop is evaluated : EOS and stop_token_ids are evaluated by the Scheduler (process A) after each token. stop/stop_sequences are evaluated by the Detokenizer (process B) after decoding.

Need for abort : EOS and stop_token_ids never require aborting the Engine. String stops often require abort because the Engine may have already generated extra tokens before the Detokenizer detects the match.

Final summary

The five concepts—EOS, stop_token_ids, stop/stop_sequences, finish_reason, and stop_reason —live on two different layers. During generation, EOS and token‑level stops are decided instantly by the model or Scheduler. After generation, the API reports finish_reason (e.g., "stop", "length") and stop_reason (None, token ID, or string) to tell the caller why the stream ended. Understanding this split explains why finish_reason="stop" can mask two distinct underlying mechanisms.