Create a Custom Enterprise Conversational Search with Alibaba Cloud OpenSearch Vector & LLM

Introduction

This article explains how to use Alibaba Cloud OpenSearch Vector Search edition and Intelligent Q&A edition to build a flexible, custom enterprise conversational search system.

Environment Preparation

When you first create an Alibaba Cloud account and log into the console, you need to create an AccessKey to continue using the services. Both the main account and RAM sub‑accounts require non‑empty AccessKey parameters.

Configuration Overview

The overall operation flow is illustrated in the diagram below.

Vector Search Edition Responsibilities

Custom data structures and related filtering/sorting.

Storage of original document data and vector data.

Vector data retrieval.

Intelligent Q&A Edition Responsibilities

Document slicing and vectorization.

Query vectorization.

LLM provides answer generation.

Creating and Configuring the Vector Search Instance

Purchase Vector Search Instance

Refer to the official purchase guide at the provided URL.

Configure Vector Search Instance

After purchase, the instance status is "Pending Configuration". You must configure the data source, then the index, and finally rebuild the index before it can be used for search.

1. Table Basic Information

Click "Add Table", set the table name, data shard count, data update resource count, and choose a scene template.

Table Name : customizable.

Data Shard Count : must be consistent across tables or at least one table with 1 shard.

Data Update Resource Count : each index provides two free 4‑core 8 GB resources; extra usage incurs fees.

Scene Template : three built‑in templates – General, Vector‑Image Search, Vector‑Text Semantic.

2. Data Synchronization

Configure a data source (MaxCompute or API push). The example uses API push.

Reference documentation:

MaxCompute + API data source: https://help.aliyun.com/zh/open-search/vector-search-edition/maxcompute-api-data-source

API data source: https://help.aliyun.com/zh/open-search/vector-search-edition/api-dataseouce

3. Field Configuration

OpenSearch pre‑populates fields based on the selected template and imports any existing fields from the source data.

At least two fields are required: a primary key field (int or string) and a vector field (multi‑value float). Example required fields:

Document primary key (e.g., doc_id).

Split content (e.g., split_content).

Split content embedding (e.g., split_content_embedding), stored as a comma‑separated vector.

4. Index Structure

Non‑vector fields automatically get single‑field indexes. Configure the vector index as follows:

Vector dimension: set to 1536.

Other parameters: keep defaults.

Primary key and vector fields are mandatory; label field is optional.

Only the three fixed fields can be selected; no additional fields are supported.

5. Confirm Creation

After configuring the index, click "Confirm Creation".

Creating and Configuring the Intelligent Q&A Instance

Purchase and Configure

Switch to the LLM Intelligent Q&A edition in the OpenSearch console (e.g., region "East China 2 (Shanghai)").

In "Instance Management", click "Create Instance", select the LLM Intelligent Q&A edition, set instance name and storage capacity, then click "Buy Now".

Confirm the order and click "Activate Immediately".

After purchase, the instance is ready without further configuration.

Data Ingestion

Document Slicing and Vectorization

The vectorization model supports up to 300 tokens, so large documents must be sliced before vectorization.

Push Sliced & Vectorized Documents to Vector Search Instance

Create a new primary key by concatenating the original document ID with the slice ID (e.g., doc_id = doc_raw_id_chunk_id).

Map fields: doc_id, doc_raw_id, split_content_embedding, split_content, time.

Push documents using the vector search instance API.

Refer to the data push demo: https://help.aliyun.com/zh/open-search/vector-search-edition/data-push-demo-2

Query and Answer

1. Query Vectorization

Convert the user's query text into a vector.

Documentation: https://help.aliyun.com/zh/open-search/llm-intelligent-q-a-version/single-document-slicing-and-vectorization

2. Retrieve Top‑5 Matching Vectors

Search the vector search instance with the query vector and retrieve the top 5 results.

Documentation: https://help.aliyun.com/zh/open-search/vector-search-edition/document-search-demo-2

3. LLM Answer Generation

Pass the retrieved document snippets to the LLM Intelligent Q&A API to generate a natural‑language answer.

Documentation: https://help.aliyun.com/zh/open-search/llm-intelligent-q-a-version/knowledge-llm-large-model-dialogue-interface

Code Example (Java SDK)

Maven Dependencies

<dependency>
  <groupId>com.aliyun</groupId>
  <artifactId>aliyun-sdk-ha3engine</artifactId>
  <version>1.3.4</version>
</dependency>
<dependency>
    <groupId>com.aliyun.opensearch</groupId>
    <artifactId>aliyun-sdk-opensearch</artifactId>
    <version>4.0.0</version>
</dependency>

Java Demo

import com.aliyun.ha3engine.Client;
import com.aliyun.ha3engine.models.*;
import com.aliyun.opensearch.OpenSearchClient;
import com.aliyun.opensearch.sdk.dependencies.org.json.JSONObject;
import com.aliyun.opensearch.sdk.generated.OpenSearch;
import com.aliyun.opensearch.sdk.generated.commons.OpenSearchClientException;
import com.aliyun.opensearch.sdk.generated.commons.OpenSearchException;
import com.aliyun.opensearch.sdk.generated.commons.OpenSearchResult;
import com.google.gson.JsonArray;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
/**
 * Demo integrating Vector Search and Intelligent Q&A
 */
public class LLMDemo {
    private static String llmAppName = "xxx";
    private static String llmHost = "http://opensearch-cn-shanghai.aliyuncs.com";
    private static String llmAccessKey = "xxx";
    private static String llmAccessSecret = "xxx";
    private static String embeddingEndpoint = "ha-cn-xxx.public.ha.aliyuncs.com";
    private static String embeddingInstanceId = "ha-cn-xxx";
    private static String embeddingTableName = "ha-cn-xxx_llm";
    private static String embeddingIndexName = "llm";
    private static String embeddingPkField = "doc_id";
    private static String embeddingUserName = "xxx";
    private static String embeddingPassword = "xxx";
    public static void main(String[] args) throws Exception {
        // 1. Create access objects for LLM and Vector Search instances
        OpenSearch openSearch = new OpenSearch(llmAccessKey, llmAccessSecret, llmHost);
        OpenSearchClient llmClient = new OpenSearchClient(openSearch);
        Config config = new Config();
        config.setEndpoint(embeddingEndpoint);
        config.setInstanceId(embeddingInstanceId);
        config.setAccessUserName(embeddingUserName);
        config.setAccessPassWord(embeddingPassword);
        Client embeddingClient = new Client(config);
        // 2.1 Slice and vectorize original documents via LLM
        Map<String, String> splitParams = new HashMap<String, String>() {{
            put("format", "full_json");
            put("_POST_BODY", "{\"content\":\"OpenSearch is a large‑scale distributed search engine...\",\"use_embedding\":true}");
        }};
        String splitPath = String.format("/apps/%s/actions/knowledge-split", llmAppName);
        OpenSearchResult splitResult = llmClient.callAndDecodeResult(splitPath, splitParams, "POST");
        System.out.println("split result：" + splitResult.getResult());
        JsonArray array = JsonParser.parseString(splitResult.getResult()).getAsJsonArray();
        // Build documents for push
        List<Map<String, ?>> documents = new ArrayList<>();
        String doc_raw_id = "001";
        for (JsonElement element : array) {
            JsonObject object = element.getAsJsonObject();
            Map<String, Object> addDoc = new HashMap<>();
            Map<String, Object> fields = new HashMap<>();
            fields.put("doc_id", doc_raw_id + "_" + object.get("chunk_id").getAsString());
            fields.put("doc_raw_id", doc_raw_id);
            fields.put("split_content_embedding", object.get("embedding").getAsString());
            fields.put("split_content", object.get("chunk"));
            fields.put("time", System.currentTimeMillis());
            addDoc.put("fields", fields);
            addDoc.put("cmd", "add");
            documents.add(addDoc);
        }
        System.out.println("push docs：" + documents.toString());
        // 2.2 Push to Vector Search
        PushDocumentsRequestModel requestModel = new PushDocumentsRequestModel();
        requestModel.setBody(documents);
        PushDocumentsResponseModel responseModel = embeddingClient.pushDocuments(embeddingTableName, embeddingPkField, requestModel);
        System.out.println("push result：" + responseModel.getBody());
        // 3.1 Vectorize query
        Map<String, String> embeddingParams = new HashMap<String, String>() {{
            put("format", "full_json");
            put("_POST_BODY", "{\"content\":\"What is OpenSearch?\",\"query\":true}");
        }};
        String embeddingPath = String.format("/apps/%s/actions/knowledge-embedding", llmAppName);
        OpenSearchResult embedResult = llmClient.callAndDecodeResult(embeddingPath, embeddingParams, "POST");
        String embedding = embedResult.getResult();
        // 3.2 Retrieve top‑5 matches
        SearchRequestModel searchReq = new SearchRequestModel();
        SearchQuery rawQuery = new SearchQuery();
        rawQuery.setQuery("query=split_content_embedding:'" + embedding + "'&&config=start:0,hit:5,format:json&&cluster=general");
        searchReq.setQuery(rawQuery);
        SearchResponseModel searchResp = embeddingClient.Search(searchReq);
        System.out.println("search result：" + searchResp.getBody());
        JsonObject recallResult = JsonParser.parseString(searchResp.getBody()).getAsJsonObject();
        long hits = recallResult.get("result").getAsJsonObject().get("numHits").getAsLong();
        List<String> snippets = new ArrayList<>();
        if (hits > 0) {
            JsonArray items = recallResult.get("result").getAsJsonObject().get("items").getAsJsonArray();
            for (JsonElement el : items) {
                JsonObject obj = el.getAsJsonObject();
                String content = obj.get("fields").getAsJsonObject().get("split_content").getAsString();
                snippets.add(content);
            }
        }
        // 3.3 Generate answer via LLM
        StringBuilder sb = new StringBuilder();
        sb.append("{ \"question\" : \"What is OpenSearch?\", \"type\" : \"text\", \"content\" : [");
        for (String s : snippets) {
            sb.append('"').append(s).append('"').append(',');
        }
        if (!snippets.isEmpty()) sb.deleteCharAt(sb.length() - 1);
        sb.append("]}");
        Map<String, String> llmParams = new HashMap<String, String>() {{
            put("format", "full_json");
            put("_POST_BODY", sb.toString());
        }};
        String llmPath = String.format("/apps/%s/actions/knowledge-llm", llmAppName);
        OpenSearchResult llmResult = llmClient.callAndDecodeResult(llmPath, llmParams, "POST");
        System.out.println("llm result:" + llmResult.getResult());
    }
}