How nndeploy Simplifies the Last Mile of On-Device AI Deployment

At the end of 2022 a senior AI algorithm deployment engineer at a large tech company realized that every AI project repeats the same cumbersome steps, the so‑called "last mile" of AI deployment.

To solve this, he conceived a generic framework based on a drag‑and‑drop workflow, a directed acyclic graph, and multi‑endpoint inference, and even took a 40% salary cut to join a 955‑type company and develop it full‑time.

1. AI Deployment: The Overlooked “Last Mile”

Deploying AI on edge devices (phones, cars, cameras) cannot rely on remote APIs due to latency, privacy, and cost constraints. The challenge is to fit smart models into resource‑limited hardware while maintaining performance.

2. nndeploy – A Toolbox for Every Stage of an AI Engineer

nndeploy becomes essential because it offers three key capabilities:

Beginner‑friendly : an ultra‑low‑threshold visual tool that lets newcomers start quickly.

Professional advancement : deep optimization for high‑performance solutions.

Efficient practice : a rich algorithm library that works out‑of‑the‑box.

2.1 Visual “Wooden Sword”, Easy to Use

For beginners, nndeploy provides:

Visual workflow : drag nodes, adjust parameters, and see results instantly.

Custom nodes : integrate Python preprocessing or C++/CUDA high‑performance nodes seamlessly.

One‑click multi‑endpoint deployment : export the workflow as JSON and run it on Linux, Windows, macOS, Android, iOS, etc.

2.2 High‑Performance “Dragon‑Slayer Sword”

For senior engineers, nndeploy includes:

Parallel optimization : serial, pipeline, and task parallelism.

Memory optimization : zero‑copy, memory pool, reuse.

High‑performance kernels : built‑in C++/CUDA/Ascend SIMD nodes.

Multi‑endpoint inference : supports 13+ mainstream inference backends (TensorRT, ONNXRuntime, OpenVINO, etc.) for cloud, edge, and device scenarios.

Example: YOLOv11s end‑to‑end workflow shows serial vs pipeline parallel execution time.

2.3 Ready‑to‑Use Algorithm Library

nndeploy ships with over 100 common nodes covering image classification, object detection, segmentation, and large‑language‑model inference, allowing developers to use them without reinventing the wheel.

3. Get Your First AI App in Five Minutes

Installation and launch are straightforward:

1. One‑click install

pip install --upgrade nndeploy

2. Start the visual editor

nndeploy-app --port 8000

Open http://localhost:8000 to access the editor, drag nodes, adjust parameters, and preview results instantly.

3. Save and execute: prototype to production

After building and debugging, click Save to export the workflow as a JSON file. The JSON can be run via command line:

Method 1: One‑click CLI

# Python
nndeploy-run-json --json_file path/to/workflow.json
# C++
nndeploy_demo_run_json --json_file path/to/workflow.json

Method 2: Load in C++/Python code

Python example:

graph = nndeploy.dag.Graph("")
graph.remove_in_out_node()
# Load exported JSON
graph.load_file("path/to/llm_workflow.json")
graph.init()
# Prepare input
input = graph.get_input(0)
text = nndeploy.tokenizer.TokenizerText()
text.texts_ = ["<|im_start|>user
Please introduce NBA superstar Michael Jordan<|im_end|>
<|im_start|>assistant
"]
input.set(text)
# Run and get result
status = graph.run()
output = graph.get_output(0)
result = output.get_graph_output()
graph.deinit()

C++ example:

std::shared_ptr<Graph> graph = std::make_shared<Graph>("");
base::Status status = graph->loadFile("path/to/llm_workflow.json");
graph->removeInOutNode();
status = graph->init();
// Prepare input
Edge* input = graph->getInput(0);
tokenizer::TokenizerText* text = new tokenizer::TokenizerText();
text->texts_ = {"<|im_start|>user
Please introduce NBA superstar Michael Jordan<|im_end|>
<|im_start|>assistant
"};
input->set(text, false);
// Run and get result
status = graph->run();
Edge* output = graph->getOutput(0);
tokenizer::TokenizerText* result = output->getGraphOutput<tokenizer::TokenizerText>();
status = graph->deinit();

4. Technical Deep Dive: Simplicity Meets Performance

nndeploy decomposes AI algorithms into workflow nodes and adopts a three‑layer separation architecture.

The three layers are:

Graph (workflow container) : manages nodes, topological sorting, and scheduling.

Node (computation unit) : independent processing block.

Edge (data channel) : connects nodes.

Execution modes include serial, pipeline parallel, task parallel, and combined parallel, giving users LEGO‑like freedom to compose complex AI pipelines.

Serial execution : follows topological order.

Pipeline parallel : parallelizes preprocessing, inference, post‑processing.

Task parallel : runs independent nodes concurrently.

Combined parallel : nests and mixes parallel strategies.

4.2 Multi‑Endpoint Inference

nndeploy provides a unified interface for over ten inference backends, abstracting differences such as TensorRT’s io_binding, OpenVINO’s ov::Tensor, and TNN’s TNN::Blob via common Tensor and Buffer containers.

An heterogeneous device abstraction layer supports CPU (x86, ARM), GPU (CUDA), and NPU (AscendCL), enabling "write once, deploy everywhere".

The framework also includes a default inference sub‑module for environments without external backends, supporting models like ResNet‑50, YOLOv11, and RMBG1.4, with ongoing work on large‑language‑model inference.

5. From an Idea to a Community Effort

nndeploy was open‑sourced on 2023‑08‑28, quickly gaining over 100 stars thanks to early endorsement from CGraph author Chunel. Contributors such as csrdxka, youxiudeshouyeren, and zhangzhaosen joined, driving rapid growth.

Real‑world deployments include OCR pipelines for document processing, NLP models in automotive cabins, and edge AI boxes for various industries.

Document processing company: full OCR pipeline.

Smart car firm: NLP in cockpit projects.

AI Box company: edge algorithm deployment.

6. Closing Thoughts

The author reflects on the collaboration with the founder, emphasizing the importance of long‑term vision and solving personal pain points, which led to the creation of a visual front‑end that dramatically lowered the barrier for AI deployment.

GitHub address: github.com/nndeploy/nndeploy