How Nacos Enables Dynamic Prompt and Parameter Management for AI Apps

Background and Challenges

In AI application development, storing sensitive information (e.g., API keys), adjusting model parameters, and iterating prompt templates often require code changes and service restarts, leading to inefficiency, security risks, and slow response to business needs.

Solution Overview

Nacos, an open‑source dynamic service discovery and configuration center, can centrally manage prompts, model parameters, and secrets, offering dynamic updates, versioning, gray‑release, and encryption.

Initial Environment Setup

Clone the Nacos Docker repository and start Nacos with Docker Compose.

git clone https://github.com/nacos-group/nacos-docker.git
cd nacos-docker
docker-compose -f example/standalone-derby.yaml up

Verify the service is running at http://localhost:8848.

docker logs -f $container_id

Managing Prompt Templates with Nacos

Create a configuration file Prompt.template in Nacos. Example content:

请为{location}编写一份面向自由行游客的旅行指南。指南应包括以下部分：
1. 简介：简要介绍该目的地的历史和文化背景。
2. 主要景点：列出并简要描述几个主要景点。
3. 推荐活动：推荐几个有趣的活动或体验。

In the application, use the Nacos Python SDK (or other language SDK) to fetch the template and listen for changes, updating the PromptTemplate at runtime.

import asyncio
from fastapi import FastAPI
from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI
from v2.nacos import ClientConfigBuilder, NacosConfigService, ConfigParam

class AIAssistant:
    async def init(self):
        client_config = (ClientConfigBuilder()
                         .server_address("localhost:8848")
                         .build())
        config_client = await NacosConfigService.create_config_service(client_config)
        template = await config_client.get_config(ConfigParam(data_id="Prompt.template", group="DEFAULT_GROUP"))
        self.prompt = PromptTemplate.from_template(template)
        await config_client.add_listener("Prompt.template", "DEFAULT_GROUP", self.prompt_listener)

    async def prompt_listener(self, tenant_id, group_id, data_id, content):
        self.prompt = PromptTemplate.from_template(content)

Dynamic Model Parameter Management

Store model parameters (temperature, model name, API key, etc.) in a JSON file LLMParam.json in Nacos:

{
    "temperature": 0,
    "model_name": "qwen-turbo",
    "streaming": false,
    "openai_api_key": "sk-xxxxxxxx",
    "openai_api_base": "https://dashscope.aliyuncs.com/compatible-mode/v1"
}

Fetch and listen to this configuration similarly, allowing on‑the‑fly adjustments without restarting the service.

LLMParamContent = await config_client.get_config(ConfigParam(data_id="LLMParam.json", group="DEFAULT_GROUP"))
LLMParam = json.loads(LLMParamContent)
self.llm = ChatOpenAI(**LLMParam)
await config_client.add_listener("LLMParam.json", "DEFAULT_GROUP", self.llm_param_listener)

async def llm_param_listener(self, tenant_id, group_id, data_id, content):
    params = json.loads(content)
    self.llm = ChatOpenAI(**params)

Gray‑Release of Prompt Updates

Nacos supports tag‑based gray release. Create two application instances with different app labels (e.g., demo1 and demo2) and assign the new prompt only to demo1. Verify the change on the gray node before full rollout.

# Instance 1 (gray node)
client_config = (ClientConfigBuilder()
                 .server_address("localhost:8848")
                 .app_conn_labels({"app":"demo1"})
                 .build())
config_client = await NacosConfigService.create_config_service(client_config)

# Instance 2 (stable node)
client_config = (ClientConfigBuilder()
                 .server_address("localhost:8848")
                 .app_conn_labels({"app":"demo2"})
                 .build())
config_client = await NacosConfigService.create_config_service(client_config)

In Nacos console, create a gray configuration for Prompt.template with version name gray1 targeting the label app=demo1. After verification, publish the version to all nodes.

Configuration Encryption for Sensitive Data

Nacos can encrypt configuration values, protecting API keys and other secrets. When using MSE Nacos, encryption can be integrated with Alibaba Cloud KMS for asymmetric encryption, meeting high security standards and complying with data protection regulations.

Running the Demo

Start the FastAPI service (default port 80) after Nacos is up.

Call the /call endpoint with JSON payload {"text":"上海"} to obtain a travel guide generated by the model.

Modify the prompt or parameters in Nacos; the changes take effect immediately without redeploy.

Conclusion

By externalizing prompts, model parameters, and sensitive credentials to Nacos, AI applications achieve rapid iteration, safe secret management, and dynamic tuning. Nacos also provides version history, gray‑release, and encryption, making it a robust solution for cloud‑native AI configuration management.