Deploy Laya Locally: Open-Source Jev Alternative for Agent Classification & Routing

What Problem Laya Solves

When building agents or customer-service systems, certain tasks require language understanding but not long-form generation: classifying user intent, routing tickets, checking safety boundaries, or deciding which tool to call. Handing these to autoregressive LLMs forces token-by-token generation of a single label or JSON, introducing latency, formatting issues, and hallucinated labels.

Laya takes a discriminative approach. Built on a bidirectional encoder like ModernBERT, it ingests a structured State (e.g., email metadata) and a set of Questions with predefined candidate options, then outputs a probability distribution over those candidates. It does not generate prose or continue text. This makes it a front-line classifier that can decide “billing vs. technical vs. sales” before a generative model is invoked, or route directly to a human.

Public benchmarks on a GB10 (aarch64, 20-core CPU, 121 GB unified memory, PyTorch 2.14.0+cu130) show single-query p50 latency of 8.31–16.41 ms with three model weights resident consuming 4–6 GB VRAM. These figures are environment-specific and should be used only for order-of-magnitude estimation.

Environment Preparation

Recommended baseline: ≥6 GB VRAM, ≥16 GB system RAM, Python 3.10+ (tested on 3.12), Linux or macOS, x86_64 or aarch64. Create an isolated virtual environment to avoid dependency conflicts:

python -m venv ~/laya-env
source ~/laya-env/bin/activate  # Windows: ~/laya-env/Scripts/activate
pip install laya

Verify version with pip show laya; at least 0.3.5 is required (check current PyPI). For faster weight downloads from China, set Hugging Face mirrors:

export HF_ENDPOINT=https://hf-mirror.com
export HF_HUB_DISABLE_XET=1

(Windows PowerShell uses $env:HF_ENDPOINT=...). Mirrors only change download paths, not model content.

Router Automatically Selects Model Branch

Laya currently offers three model branches: English, multilingual, and typed-decisions (fine-tuned for workflow decisions). The Router detects input language and dispatches accordingly—Chinese-heavy input goes to laya-multilingual, English to the English branch. This mirrors the “classify-then-generate” pattern: the router performs a lightweight language classification so business code doesn’t need to scatter model-selection logic.

Minimal Working Script

from laya import Router

router = Router(device="cuda", preload=True, max_loaded=3)

state = {
    "from": "[email protected]",
    "subject": "Duplicate charge on invoice #4411",
    "body": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel your plan."
}

questions = {
    "department": {
        "type": "choice",
        "instructions": "Which department should handle this request?",
        "criteria": {
            "billing": "invoices, payments, refunds",
            "technical": "bugs, outages, system errors",
            "sales": "pricing, new contracts",
            "other": "everything else"
        }
    }
}

result = router.predict(state, questions)
print("Routed to:", result["routing"]["model"])
print("Decision:", result["answers"]["department"]["choice"],
      "Confidence:", round(result["answers"]["department"]["confidence"], 4))

First run downloads weights into local cache; ignore its latency. Three design points: (1) state accepts structured fields directly—no need to flatten into a prompt. (2) questions defines the candidate set explicitly; the model only chooses among them. (3) Output is a plain Python dict, ready for downstream logic without regex parsing.

Reading the Return Structure

Example output from the GB10 test run (values are illustrative only):

{
  "model": "laya-rl-agent",
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {
        "billing": 0.9582,
        "technical": 0.017,
        "sales": 0.0134,
        "other": 0.0114
      },
      "confidence": 0.8419,
      "action": {"act_probability": 1.0}
    }
  },
  "usage": {"input_tokens": 96, "output_tokens": 0},
  "routing": {
    "model": "english",
    "repo": "convaiinnovations/laya",
    "reason": "English Latin text",
    "detection": {
      "script": "latin",
      "script_profile": {"latin": 1.0},
      "language": "en",
      "is_english": true,
      "language_undecided": false,
      "diacritic_rate": 0.0,
      "non_latin_fraction": 0.0
    },
    "workflow": null
  }
}

Three layers to inspect:

1. choice is the selected branch

It answers “which candidate is most likely” but cannot validate whether the candidate set itself is correct. If real-world cases include “security incident” or “insufficient info,” those must be added to criteria upfront; otherwise the model is forced to pick the closest wrong label.

2. probabilities and confidence enable gating

probabilities

gives the full distribution; confidence is a composite score. Low-confidence requests should escalate to humans or a stronger generative model. However, action.act_probability is nearly always 1.0 in current versions and does not reflect accuracy. Moreover, confidence ≠ accuracy: Chinese samples can show >0.90 confidence yet be wrong. Thresholds must be calibrated on your own business data, not copied from logs.

3. routing.model reveals the actual branch

The top-level model field may stay fixed at laya-rl-agent. To know which weights were used, read routing.model together with routing.reason and routing.detection. For Chinese input, reason may show non-Latin script (han, 95% of letters) and switch to multilingual; the exact ratio varies with input composition, so don’t hard-code it.

Laya also supports score (ordinal) and boolean (binary) question types; multiple questions can be bundled in one predict call.

Three Invocation Patterns

Method 1: Agent locked to a single branch

from laya import Agent
agent = Agent("convaiinnovations/laya", device="cuda", subfolder="multilingual")
result = agent.predict(state, questions)

Pros: predictable VRAM (~1.25 GB on GB10), simpler audit. Cons: you own model selection—if Chinese traffic hits the English branch, no auto-correction.

Method 2: Router with global default fallback

router = Router(device="cuda", default="multilingual")

Language detection still runs; default only applies when detection is uncertain. Useful for mixed-language or short-field inputs where detection may waver.

Method 3: Per-call override

result = router.predict(state, questions, model="multilingual")

Highest priority; skips language detection. Suited for canary releases, A/B tests, or batches with known language. Avoid scattering hard-coded model names in everyday business code.

Benchmarks on the same Chinese input: Router adds ~0.2 ms overhead for language detection but keeps multiple weights resident (4.39 GB VRAM). Agent locked to one branch uses ~1.25 GB. Numbers are GB10-specific.

Model Cards & Experiment Repository

Three Hugging Face model cards detail each branch: convaiinnovations/laya (base), convaiinnovations/laya-multilingual, convaiinnovations/laya-typed-decisions. Check parameter counts, language coverage, and task alignment there.

Accompanying experiment repo:

https://github.com/li-xiu-qi/XiaokeAILabs/tree/main/experiments/test_jev_open_source/laya

. Clone locally:

git clone https://github.com/li-xiu-qi/XiaokeAILabs.git
cd XiaokeAILabs/experiments/test_jev_open_source/laya

Key scripts: test_laya.py – minimal invocation laya-verify.py – exports full protocol for four question types laya-latency.py – cold-start, multi-question batching, VRAM logging laya-direct.py, laya-pin.py – direct vs. pinned-branch comparisons laya-zh-en.py – bilingual parallel samples laya-scale.py – scaling from 1 to 256 questions laya-common.py – local re-implementation of architecture & confidence algorithm

Logs and scripts evolve; re-read README, model cards, and current scripts before production deployment.

Four Chinese-Scenario Pitfalls

1. Don’t extrapolate small-sample accuracy

Public 20-sample bilingual test shows 95% Chinese accuracy for the multilingual branch, matching English. Sample size, task scope, and hardware are limited—do not assume “all Chinese tickets hit 95%.” Build a calibration set from your historical tickets covering normal, ambiguous, unknown, and adversarial inputs; measure real error rates per confidence bucket.

2. Chinese confidence tends to run high

In the same 20-sample test, Chinese average confidence ≈0.866 vs. English ≈0.506; some misclassified Chinese samples still scored >0.90. These are observations, not universal thresholds. Raising auto-accept threshold to 0.95 reduces coverage; lowering it increases false passes. Set thresholds per business cost: misrouting a routine inquiry vs. missing a security incident demand different trade-offs.

3. Prefer choice over boolean for binary decisions

Current boolean type shows probability compression toward 0.5 on Chinese input. If “yes/no” can be expressed as explicit candidates, use choice instead:

questions = {
  "needs_escalation": {
    "type": "choice",
    "instructions": "Does this request need escalation?",
    "criteria": {
      "yes": "security incident, payment dispute, or severe outage",
      "no": "ordinary request that the current team can handle"
    }
  }
}

This makes candidate definitions explicit and probability distributions easier to map to business branches. Still validate with your Chinese data.

4. Warm up before serving production traffic

First forward pass triggers CUDA initialization and can take ~1.9 s vs. subsequent milliseconds. On service start, run a no-op test input to warm the model, then open real traffic. Warm-up only addresses initialization latency, not branch selection errors, distribution shift, or confidence calibration.

Closing Thoughts

Laya fits at the front of an agent pipeline, handling intent classification, parameter routing, and compliance checks—tasks with a well-defined answer space. It doesn’t write prose or perform open-ended reasoning.

Deployment shortcut: create venv → install & verify version → configure HF mirror → run Router minimal script → inspect routing.model, choice, probabilities, confidence. After confirming dominant language and VRAM headroom, decide whether to keep auto-routing or lock to a single Agent.

The real value of discriminative models like Laya isn’t “smarter than LLMs” but pulling deterministic judgments out of the generation chain. Success hinges on three mundane questions: Is the candidate set complete? Has confidence been calibrated on your data? Is there a reliable human/LLM fallback for low-confidence cases?