Needle2: 45M‑Parameter Open‑Source Edge Agent – 14 MB Model, 28 MB Runtime for Offline Tool Calls

Problem Statement

Developers targeting embedded devices such as smart watches, IoT hubs, robots, or microcontrollers need AI that runs offline and fits within severe memory limits. Cloud APIs become unavailable when offline and large local models (hundreds of MB to GB) cannot be deployed on tens‑of‑MB hardware.

Needle2 Overview

45 M parameters, packaged as a single .cact binary of 14 MB.

Constant runtime memory ≈28 MB, independent of conversation length.

CQ2‑bit (Cactus Quants) quantization.

256‑token sliding context window.

Optimized for three tasks: tool calling, device control, and unstructured‑text‑to‑JSON extraction.

Runs on Raspberry Pi 5, VR headsets, and ESP32‑S3 microcontrollers.

Core Capabilities

Byte‑level syntax‑constrained decoding : Compiles a JSON Schema into byte‑level constraints so each generated token stays within a valid JSON structure, dramatically reducing malformed output.

Confidence‑gating mechanism : Each output includes a calibrated confidence score; high‑score results can be executed directly, while low‑score results should be deferred to a fallback LLM or human review.

Large‑toolset retrieval : An internal retrieval head selects the top‑5 most relevant tools per turn, limiting context overflow and applying syntax constraints only to the chosen subset.

KV‑Sink constant memory : A 256‑token sliding window evicts old dialogue, while tool definitions are stored permanently in a KV cache, keeping total RAM usage at ~28 MB regardless of conversation length.

Single‑file offline deployment + LoRA‑friendly : The model is distributed as a single .cact file; LoRA adapters can be trained on top of the frozen base and merged back into a single file without recompiling the inference engine.

Architecture (Simple Attention Network)

Hadamard MLP : Uses a fixed Walsh‑Hadamard orthogonal matrix, eliminating trainable weights and achieving n·log(n) complexity.

Grouped Query Attention (GQA) : Compresses attention parameters while preserving speed and accuracy.

Engram Hash n‑gram memory : Aggregates KV pairs in a hash table to extend short‑term memory.

Multi‑Channel Hyper‑Connection (mHC) : Parallel residual streams with dynamic gating based on input.

Performance Benchmarks

Token‑throughput: desktop ≈850 tokens/s, Raspberry Pi 5 ≈500 tokens/s, VR headsets 400‑1500 tokens/s, smartphones 300‑700 tokens/s.

Compute per token: Needle2 ≈70 MFLOPs, FunctionGemma ≈540 MFLOPs, Apple FM ≈6000 MFLOPs. Model size is 5‑70× smaller than the baselines. Tool‑calling benchmarks show mixed results—Needle2 wins on some tasks but does not dominate across the board.

Deployment Notes

On first run the inference engine downloads a cache from the internet; after caching the model runs fully offline. Air‑gapped devices must pre‑populate the cache; direct pip installation on such machines is not supported. Windows lacks a complete installation guide and is only suitable for quick experiments, not production.

Hands‑On Demos

# CPU‑only installation
pip install cactus-needle
# GPU acceleration
pip install "cactus-needle[gpu]"
# Apple Silicon (Metal) acceleration
pip install "cactus-needle[metal]"

Demo 1 – Tool registration via decorator

import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
resp = agent.run("what's it like in Lagos right now?")
print(resp["results"])

Demo 2 – Structured extraction with Pydantic

from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

invoice = needle.extract(
    "Invoice from Acme Corp, $1,200.00, due 2026-09-01",
    Invoice,
)
print(invoice.vendor, invoice.total)

Demo 3 – Playground UI

needle playground
# Use a custom LoRA‑fine‑tuned model
# needle playground --weights my_needle.cact

Access the UI at http://127.0.0.1:7860.

LoRA Fine‑Tuning Workflow

Prepare a JSONL dataset (one sample per line).

Optionally generate data automatically (requires OpenRouter API key).

Run fine‑tuning:

needle finetune data.jsonl --epochs 10 --lora-rank 16

Merge LoRA into a single .cact model (must specify --bits 2 to keep the 14 MB size):

needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl \
    --out my_needle.cact --bits 2

Load the custom model:

agent = needle.Needle(weights="my_needle.cact", tools=[...])

Known Issue: After LoRA fine‑tuning, confidence scores become unreliable; production code must validate outputs beyond the confidence value.

Community‑Raised Concerns & Practical Pitfalls

Not a general‑purpose chat model; 45 M parameters limit chat and long‑text generation.

Windows installation is incomplete; not recommended for production.

Offline deployment requires pre‑downloaded inference cache.

LoRA training invalidates confidence scores.

Context window limited to 256 tokens; tool definitions consume part of this budget.

Automatic dataset generation depends on external APIs; offline environments must craft JSONL manually.

Overall, Needle2 is a specialized model, not a universal solution.

When to Choose Needle2

IoT, smart‑home, wearable watches, robots where privacy forbids cloud usage.

Devices with extreme memory limits (tens of MB).

VR/AR edge agents needing low latency without network reliance.

Air‑gapped equipment that cannot access the internet.

Local logs, sensor data, or tickets that require structured extraction.

When Not to Use Needle2

Open‑domain QA, multi‑step reasoning, or long‑form generation.

Requirements for context windows far beyond 256 tokens.

Production deployments on Windows machines.

Simple text parsing tasks that can be solved with regular expressions.

Hybrid Recommendation

Run Needle2 locally for tool routing and structured extraction; if confidence falls below a threshold, forward the request to a cloud LLM for fallback handling.

Conclusion

Needle2’s design philosophy is to bring AI to the smallest hardware rather than moving devices to the cloud. By stripping away unnecessary capabilities and focusing on tool calling and JSON extraction, it achieves a 14 MB model file and a steady 28 MB runtime using a Simple Attention Network backbone and CQ2‑bit quantization. The model has clear boundaries—small context window, specialized abilities, and known confidence‑score issues—but provides an open‑source foundation for massive edge‑IoT, wearables, and robot deployments.

Project Links

https://github.com/cactus‑compute/needle

https://huggingface.co/Cactus-Compute/needle2

https://arxiv.org/abs/2607.18363