How to Quickly Deploy an Enterprise LLM API Using SGLang

1. Define deployment scope

A production inference service consists of five layers: model files, GPU driver & container runtime, the SGLang process, an OpenAI‑compatible API, and an upstream gateway. SGLang handles inference and request scheduling; authentication, TLS, quota, and multi‑node traffic management must be provided by Nginx, an API gateway, or a service mesh.

Before launch, fix the following parameters: model type and precision, maximum context length, GPU IDs, tensor‑parallel degree (usually equal to the number of GPUs per instance), instance port, allowed subnets, expected concurrency, and input/output limits.

2. Pre‑deployment host and GPU checks

Run a read‑only script to verify kernel, Docker, disk space, and GPU status. The script prints uname -a, Docker version, nvidia‑smi output, and model directory size.

#!/usr/bin/env bash
set -euo pipefail

MODEL_DIR="<model_dir>"

uname -a
docker version --format 'Server={{.Server.Version}}' 2>/dev/null || true
nvidia-smi --query-gpu=index,name,memory.total,memory.used,utilization.gpu \
  --format=csv,noheader
df -h "${MODEL_DIR}"
du -sh "${MODEL_DIR}"

Key checks: nvidia‑smi must return correctly, no unknown processes should occupy the target GPUs, and the model filesystem must have enough free space. GPU access inside the container also depends on the NVIDIA Container Toolkit; host nvidia‑smi success does not guarantee container access.

Validate that the model directory contains at least config.json, tokenizer.json, and weight files (e.g., *.safetensors or *.safetensors.index.json).

3. Verify image version and supported flags

Never use the latest tag in production. Inspect the image digest and run the built‑in help to ensure the required flags are present.

IMAGE="<sglang_image>:<tag>"

docker image inspect "$IMAGE" --format 'Id={{.Id}} Digests={{json .RepoDigests}}'

docker run --rm "$IMAGE" python3 -m sglang.launch_server --help | less

Confirm the presence of --model-path, --host, --port, --tp, --mem-fraction-static, --context-length, and --served-model-name. Adjust the Compose file if the help output differs.

4. Minimal foreground launch

Run the first instance in the foreground to surface initialization errors directly. Example for two GPUs (IDs 0 and 1) with tensor parallelism 2, listening on port 8000:

docker run --rm --gpus "device=0,1" \
  --ipc=host \
  --shm-size=32g \
  -p 8000:8000 \
  -v "<model_dir>:/models/model:ro" \
  <sglang_image>:<tag> \
  python3 -m sglang.launch_server \
    --model-path /models/model \
    --served-model-name <external_model_name> \
    --host 0.0.0.0 \
    --port 8000 \
    --tp 2 \
    --context-length 8192 \
    --mem-fraction-static 0.85

--ipc=host

and a large shared memory size avoid the default /dev/shm limitation. --mem-fraction-static 0.85 is not an absolute cap; higher values increase KV‑Cache space but raise OOM risk, so start conservatively and adjust after load testing.

5. Solidify the setup with Docker Compose

After a successful foreground run, encode all parameters in a Compose file. The example below uses explicit GPU device mapping, read‑only model mount, health check, and log rotation.

services:
  sglang-api:
    image: <sglang_image>:<tag>
    container_name: sglang_api_8000
    restart: unless-stopped
    ipc: host
    shm_size: 32g
    ports:
      - "8000:8000"
    environment:
      CUDA_VISIBLE_DEVICES: "0,1"
    volumes:
      - <model_dir>:/models/model:ro
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0", "1"]
              capabilities: [gpu]
    command:
      - python3
      - -m
      - sglang.launch_server
      - --model-path
      - /models/model
      - --served-model-name
      - <external_model_name>
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --tp
      - "2"
      - --context-length
      - "8192"
      - --mem-fraction-static
      - "0.85"
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/health', timeout=3)\""]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 600s
    logging:
      driver: json-file
      options:
        max-size: 100m
        max-file: "5"

Set start_period long enough to cover model loading time; otherwise the container may be marked unhealthy during initialization.

6. Layered verification from port to inference

Check that the process is listening and that the health endpoint returns successfully:

ss -lntp | grep ':8000 '
curl --fail --silent --show-error --connect-timeout 3 --max-time 10 http://127.0.0.1:8000/health

Verify the OpenAI‑compatible model list matches the served name:

curl --fail --silent --show-error http://127.0.0.1:8000/v1/models | jq .

Perform a non‑streaming chat test to confirm token limits and response structure:

curl --fail --silent --show-error -X POST http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "<external_model_name>",
    "messages": [{"role": "system", "content": "你是企业内部运维助手。"}, {"role": "user", "content": "只回复：ready"}],
    "temperature": 0,
    "max_tokens": 16,
    "stream": false
  }' | jq .

For streaming, disable client buffering with curl -N to observe SSE chunks.

7. Baseline load testing

Run a simple concurrency script that sends short requests and records HTTP status and latency. This helps locate the stable concurrency boundary without exhausting GPU resources.

#!/usr/bin/env bash
set -euo pipefail

URL="${URL:-http://127.0.0.1:8000/v1/chat/completions}"
MODEL="${MODEL:-<external_model_name>}"
CONCURRENCY="${CONCURRENCY:-8}"
TOTAL="${TOTAL:-32}"
TMP_DIR="$(mktemp -d)"
trap 'rm -rf "${TMP_DIR}"' EXIT

request_one() {
  local id="$1"
  curl -sS -o "${TMP_DIR}/${id}.json" \
    -w "id=${id} code=%{http_code} total=%{time_total}
" \
    -H 'Content-Type: application/json' \
    -d "{\"model\":\"${MODEL}\",\"messages\":[{\"role\":\"user\",\"content\":\"返回编号 ${id}\"}],\"max_tokens\":32,\"temperature\":0}" \
    "${URL}"
}
export -f request_one
export URL MODEL TMP_DIR
seq 1 "${TOTAL}" | xargs -P "${CONCURRENCY}" -I{} bash -c 'request_one "$@"' _ {}

Collect GPU utilization with nvidia‑smi dmon -s pucvmet -d 1 -o DT. If throughput stalls while latency grows, the instance has passed its economical concurrency point and should be throttled or scaled.

8. Failure evidence chain

8.1 Container exits immediately

CONTAINER="sglang_api_8000"

docker inspect "$CONTAINER" --format 'Status={{.State.Status}} Exit={{.State.ExitCode}} Error={{.State.Error}} OOM={{.State.OOMKilled}}'

docker logs --timestamps --tail=300 "$CONTAINER"

docker inspect "$CONTAINER" --format '{{json .Config.Cmd}}' | jq .

Exit code 137 may indicate OOM kill or external termination; verify .State.OOMKilled, kernel logs, and GPU logs before concluding.

8.2 CUDA OOM or static memory pool failure

nvidia-smi --query-gpu=index,uuid,memory.total,memory.used,memory.free --format=csv
nvidia-smi --query-compute-apps=gpu_uuid,pid,process_name,used_memory --format=csv
docker inspect sglang_api_8000 --format '{{json .HostConfig.DeviceRequests}}' | jq .

If static memory pool allocation fails, reduce --mem-fraction-static, shorten --context-length, or add GPUs. Changing parallelism requires re‑testing.

8.3 Multi‑GPU initialization hangs

nvidia-smi topo -m
ps -eo pid,ppid,stat,etime,cmd | grep -E '[s]glang|[t]orch'
ss -lntp | grep -E ':8000|torch|python' || true

Confirm that all GPUs belong to the same instance and that NVLink/PCIe topology matches expectations. High GPU utilization alone does not guarantee effective inference.

8.4 Service reachable but requests timeout

curl -sS -o /dev/null \
  -w 'dns=%{time_namelookup} connect=%{time_connect} start=%{time_starttransfer} total=%{time_total} code=%{http_code}
' \
  --max-time 120 \
  http://127.0.0.1:8000/v1/models

Separate DNS lookup, TCP connect, first‑byte, and total time to locate bottlenecks. Compare direct localhost access versus gateway‑routed access.

9. Secure the exposure surface

Bind the service to 127.0.0.1 when only accessed by a local Nginx, or restrict to internal network addresses, security groups, or firewalls. Never expose the SGLang port directly to the internet. Place authentication, TLS, request‑size limits, rate limiting, and audit logging at the gateway layer.

Run containers with read‑only model mounts and verify the service account lacks write permissions:

docker inspect sglang_api_8000 --format '{{range .Mounts}}{{println .Source "->" .Destination "RW=" .RW}}{{end}}'

docker exec sglang_api_8000 sh -c 'id; test -r /models/model/config.json; test ! -w /models/model/config.json'

10. Upgrade, canary, and rollback

Before upgrading, record the old image digest, full Compose file, model config checksum, and baseline request metrics. Backup the Compose file and rendered configuration:

#!/usr/bin/env bash
set -euo pipefail

COMPOSE_FILE="<compose_path>"
BACKUP_DIR="<backup_dir>/sglang-$(date +%Y%m%d-%H%M%S)"
mkdir -p "${BACKUP_DIR}"

cp -a "${COMPOSE_FILE}" "${BACKUP_DIR}/"

docker compose -f "${COMPOSE_FILE}" config > "${BACKUP_DIR}/compose.rendered.yaml"

docker image inspect <sglang_image>:<old_tag> > "${BACKUP_DIR}/image.inspect.json"
sha256sum <model_dir>/config.json > "${BACKUP_DIR}/model-config.sha256"
printf 'Backup: %s
' "${BACKUP_DIR}"

Deploy the new version on a different port (e.g., 8001), run health checks, short and long‑context requests, and a low‑traffic canary before switching the gateway. If errors appear, switch the gateway back to the old port and stop the new instance only after it has been removed from the load balancer.

11. Routine health‑check script

#!/usr/bin/env bash
set -euo pipefail

CONTAINER="${CONTAINER:-sglang_api_8000}"
BASE_URL="${BASE_URL:-http://127.0.0.1:8000}"
EXPECTED_MODEL="${EXPECTED_MODEL:-<external_model_name>}"

status=$(docker inspect -f '{{.State.Status}}' "$CONTAINER")
[[ "$status" == "running" ]] || { echo "CRITICAL: container status=$status" >&2; exit 2; }

curl -fsS --max-time 5 "$BASE_URL/health" >/dev/null
models=$(curl -fsS --max-time 10 "$BASE_URL/v1/models")
jq -e --arg model "$EXPECTED_MODEL" '.data[] | select(.id == $model)' <<< "$models" >/dev/null

nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu,temperature.gpu --format=csv,noheader

echo "OK: $CONTAINER model=$EXPECTED_MODEL"

12. Acceptance checklist

Fixed image digest and version tag.

Model directory mounted read‑only.

GPU IDs match the tensor‑parallel degree.

All startup flags match the help output.

Health check, model list, non‑streaming and streaming requests all pass.

Long‑context and over‑limit request behavior matches expectations.

Load test covers the target concurrency.

No logs leak prompts or model outputs.

SGLang port is not exposed to the public internet.

Gateway provides authentication, TLS, rate limiting, and timeout.

Upgrade uses a new‑port canary with a documented rollback path.

Old images, configurations, and rollback steps remain usable.

Only when the entire stack—from infrastructure to business entry point—passes these checks is the service truly production‑ready.