Reversible PII Masking Pipeline for Secure LLM Gateway Implementation

Scenario: Customer Service Drafting Without Exposing Raw Data

A customer service agent selects a ticket record and asks the model to draft an SMS. The record contains a phone number, email, and order ID. The model must understand that the phone and email refer to the same customer, but must not see the actual values. Additionally, tickets occasionally contain pasted access tokens, database connection strings, or private key fragments — such credentials must be blocked entirely, not merely masked, and require manual rotation.

Gateway Design: Narrowed-Capability Text Security Gateway

The gateway accepts only non-streaming, plain-text chat requests. It first blocks credentials, then replaces phone numbers and emails with random placeholders unique to each request. Finally, based on the caller's verified permissions, it decides whether to restore the original values in the response. The gateway does not protect images, attachments, OCR, RAG snippets, tool calls, or streaming output — those must be explicitly rejected, not silently passed through.

Data Visibility Matrix

Phone/email in ticket body → mapped to same random token per request. Reason: model retains entity reference and repetition relationships without seeing raw values.

Private keys, Bearer tokens, API keys, connection strings → 403 reject, log rule ID only. Reason: credentials must not enter inference context; must be rotated.

Caller's Authorization header → used only for gateway auth, never forwarded upstream. Reason: it is an access-control credential, not subject to "detect-and-block".

Caller without pii:reveal scope → receives tokenized response. Reason: prevents API response from becoming an unauthorized disclosure channel.

Controlled backend with pii:reveal scope → restores known tokens within the same request. Reason: only for display paths already authorized to view raw ticket content.

Model-generated phone, email, payment instructions → treated as text only, cannot drive real actions. Reason: tool execution must use business IDs, server-side authorization, and independent parameter validation.

Key Concepts: Caller Authentication vs. Business Content Detection

Caller authentication answers "who can call the gateway and see raw values". The client's Authorization header is validated but never forwarded to the model provider.

Business content detection answers "does the ticket text contain data that must not leave". Only messages[].content is scanned; any unparsed fields are outside the protection guarantee.

Placeholder mappings live only in the single HTTP request's local memory: same phone number becomes same token within a request, different requests always get different tokens. Mappings must not enter logs, trace attributes, metrics labels, databases, or queues. Production must restrict debugger access, disable unnecessary core dumps, and audit APM sampling.

Data Flow: Drafting, Not Auto-Sending

客服页面 → 工单后端（已鉴权） → 安全网关 → 云端模型
                 │                     │
                 │                     ├─ 凭据命中：拒绝，零次上游调用
                 │                     ├─ 手机/邮箱：替换为随机 token
                 │                     └─ 上游仅收到 token 化文本
                 │
                 └─ 有 pii:reveal：显示还原后的“短信草稿”
                      无 pii:reveal：显示 token 化草稿

短信发送：只能由工单后端使用订单/客户业务 ID 再次授权后执行

Even if the model says "sent to that number", it is only natural language. The model has no SMS tool and no customer number; the business system must not extract phone numbers from model output to call SMS services.

Minimal Runnable Implementation (Python 3.11+)

Dependencies: fastapi>=0.110, uvicorn[standard]>=0.27, httpx>=0.27, pydantic>=2, pytest.

The gateway ( app.py) implements a /v1/chat/completions compatible non-streaming plain-text upstream. Model name, upstream URL, and upstream service key come only from deployment config. Callers can only choose from allowed models; they cannot supply upstream URL, API key, or arbitrary headers.

Core Components

Credential rules (regex patterns for private keys, GitHub tokens, Stripe live keys, assignment secrets, authorization headers, URL passwords) — checked first; any match returns 403 with rule ID, no upstream call.

PII patterns for Chinese mobile phones ( 1[3-9]\d{9}) and emails — detected after credential check.

Token format : [[GW_PHONE_<32-hex>]] or [[GW_EMAIL_<32-hex>]] generated via secrets.token_hex(16).

Transform : builds forward/reverse maps per request, replaces detected spans from end to start to preserve indices.

Restore : only restores tokens generated in the current request; unknown or malformed tokens cause 502 fail-closed.

Safe response : validates upstream response structure, rejects tool_calls or non-string content, returns only allowed fields, restores tokens only if caller has can_reveal.

Configuration Example

export UPSTREAM_URL='https://api.openai.com/v1/chat/completions'
export UPSTREAM_API_KEY='injected-by-deployment-system'
export ALLOWED_MODELS='approved-model-name'
export GATEWAY_MASKED_CALLER_TOKEN='token-for-masked-draft-only'
export GATEWAY_REVEAL_CALLER_TOKEN='token-for-controlled-backend'
uvicorn app:app --host 127.0.0.1 --port 8080

Example Call (No Reveal Permission)

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Authorization: Bearer token-for-masked-draft-only' \
  -H 'Content-Type: application/json' \
  -d '{"model":"approved-model-name","messages":[{"role":"user","content":"为订单 A-20260928-17 的客户 13800138000 起草回访短信，抄送 [email protected]。"}]}'

The model receives text like "customer [[GW_PHONE_…]], CC [[GW_EMAIL_…]] ". Without pii:reveal, the response keeps tokens; the authorized backend gets the restored draft in the same request. Neither path gives the model ability to send SMS or email.

Required Tests

Saved as test_app.py. Tests run without calling a real model; end-to-end tests should capture gateway-to-upstream traffic in an isolated network to confirm raw values never appear. test_same_request_reuses_token_but_requests_are_isolated: same phone twice in one request yields one token; different requests yield different tokens; cross-request restore fails. test_credential_blocks_before_any_redaction: credential detection returns 403 with rule ID before any masking occurs. test_client_cannot_inject_gateway_token_namespace: input containing [[GW_PHONE_...]] is rejected (422). test_unknown_or_damaged_upstream_token_fails_closed: restore fails on unknown or malformed tokens. test_response_is_masked_without_reveal_scope: without can_reveal, response contains tokens; with it, tokens are restored. test_tool_output_is_not_silently_accepted: upstream response with tool_calls raises ValueError.

Run with pytest -q.

Production Checklist

Integrate permission decisions with existing identity system. Demo tokens only show control points; real systems must embed tenant, user, ticket ownership, and

pii:reveal</scope in signed identity claims, verified at gateway. Browser must not submit <code>can_reveal=true

directly.

Build asset inventory for fields, rules, and paths. Phone/email are just the first batch. ID cards, bank cards, addresses, names, internal project names, attachment content, RAG snippets must each be explicitly classified as "detect-and-replace", "block", or "not onboarded" — never assume protected by default.

Manage detection rules, not just regexes. Assign each rule an ID, version, owner, regression samples, and false-positive/negative metrics. Evaluate coverage with authorized labeled data. Credential detection should be more conservative than ordinary entities. Unknown secret formats may still leak; high-sensitivity businesses should combine source-field tagging, classification labels, and business DLP.

Restrict network and protocol. UPSTREAM_URL fixed in deployment config; network egress limited to approved domains/IPs; disable redirects and restrict DNS resolution. Gateway constructs only whitelisted request headers; never forwards client headers. Do not allow request to specify URL, model provider, or proxy.

Fail closed. Detection unavailable, rule config load failure, upstream error, unknown token in response, response format change — all return errors; never fall back to sending raw text.

Design streaming and multimodal separately. SSE tokens can cross event boundaries; secure implementation needs event-level state buffering, final-event validation, cancellation handling, disconnect strategy. Until then, keep extra="forbid" to reject stream. OpenAI streaming delivers output via server-sent events; each delta is not a complete restorable text.

Observe without storing raw data. Log only request ID, tenant ID, rule ID, rule version, masking count, permission decision, latency, error category. Forbid raw values or token mappings in access logs, exception stacks, APM attributes, trace baggage, dead-letter queues, or support tickets.

Isolate automated actions from the model. SMS, email, refunds, queries executed by backend using business IDs to re-read controlled data, verify operator permissions, amount/frequency/recipient policies, and require appropriate human confirmation. Never extract numbers from free-form model text to execute actions.

Acceptance Criteria for Go-Live

Capture real test requests from gateway to upstream; confirm phone, email, and caller Authorization never appear.

Every credential test sample is rejected, and audit shows zero upstream calls for those requests.

Callers without pii:reveal cannot obtain raw values from any response field, error message, or log; authorized callers can only restore tokens from the same request.

Same phone in different requests produces different tokens; one request's mapping cannot restore another request's token. stream, tools, attachments/images, unsupported roles, and extra fields are all rejected.

Upstream returning unknown, truncated, or malformed tokens fails closed; new upstream fields are not reflected to client.

Load test records P50/P95/P99 for detection, upstream, and total latency using realistic mixed text; do not claim "gateway P95 under X ms" without measured data.

Only when all above paths pass and every business entry point is inventoried should the gateway be connected to real customer service traffic.

References

Python secrets: secure random value generation.

OpenAI API: streaming response events.