10 Practical LangChain Performance Hacks to Speed Up and Cut Costs

Production LangChain applications typically spend 2–5 seconds per LLM call, leading to token costs of thousands of dollars per month. The main cost drivers are:

Repeated LLM calls (same query recomputed each time)

Serial execution of independent tasks

Overly long prompts that inflate token usage

Using large‑capacity models for simple tasks

01 Caching – fastest wins

Technique 1: InMemoryCache (process‑level)

Best for development or short‑lived scripts. The cache lives only while the process runs.

import { ChatOpenAI } from "@langchain/openai";
import { InMemoryCache } from "@langchain/core/caches";

const cache = new InMemoryCache();
const llm = new ChatOpenAI({ model: "gpt-4o-mini", cache });
// First call – real request (~2 s)
const res1 = await llm.invoke("用一句话解释什么是向量数据库");
console.log("第一次:", res1.content);
// Second call – cache hit (<1 ms)
const res2 = await llm.invoke("用一句话解释什么是向量数据库");
console.log("第二次（缓存）:", res2.content);

Technique 2: RedisCache (persistent, cross‑process)

Suitable for production; cache survives process restarts and is shared across instances.

import { RedisCache } from "@langchain/community/caches/ioredis";
import { Redis } from "ioredis";

const client = new Redis({ host: "localhost", port: 6379 });
const llm = new ChatOpenAI({
  model: "gpt-4o-mini",
  cache: new RedisCache(client, { ttl: 3600 }) // 1 hour TTL
});
const result = await llm.invoke("解释 RAG 的工作原理");

Measured effect: In repeat‑query workloads API calls drop by 60‑80 %.

Technique 3: Semantic Cache (similar‑question hits)

Extends caching to semantically similar inputs using embeddings.

import { RedisSemanticCache } from "@langchain/community/caches/ioredis";
import { OpenAIEmbeddings } from "@langchain/openai";

const semanticCache = new RedisSemanticCache({
  redisUrl: "redis://localhost:6379",
  embeddings: new OpenAIEmbeddings(),
  similarityThreshold: 0.9 // >90 % similarity triggers a hit
});
const llm = new ChatOpenAI({ model: "gpt-4o", cache: semanticCache });
await llm.invoke("什么是向量数据库？");
await llm.invoke("向量数据库是什么？"); // ✅ cache hit

02 Concurrency – turn serial into parallel

Technique 4: RunnableParallel

Runs multiple independent LLM tasks concurrently.

import { RunnableParallel } from "@langchain/core/runnables";
import { ChatOpenAI } from "@langchain/openai";
import { PromptTemplate } from "@langchain/core/prompts";

const llm = new ChatOpenAI({ model: "gpt-4o-mini" });
const parallel = RunnableParallel.from({
  summary: PromptTemplate.fromTemplate("用50字总结：{text}").pipe(llm),
  keywords: PromptTemplate.fromTemplate("提取5个关键词：{text}").pipe(llm),
  sentiment: PromptTemplate.fromTemplate("判断情感倾向（正/负/中性）：{text}").pipe(llm)
});
const result = await parallel.invoke({ text: "LangChain 是一个强大的 AI 应用开发框架..." });
console.log("摘要:", result.summary.content);
console.log("关键词:", result.keywords.content);
console.log("情感:", result.sentiment.content);

Timing comparison:

Serial: 3 × 2 s ≈ 6 s

Parallel: max(2 s, 2 s, 2 s) ≈ 2 s → 3× faster

Technique 5: batch() – bulk request

Processes many inputs in a single API call with controlled concurrency.

const llm = new ChatOpenAI({ model: "gpt-4o-mini" });
const questions = [
  "什么是 LangChain？",
  "什么是 LangGraph？",
  "什么是 RAG？",
  "什么是向量数据库？",
  "什么是 MCP？"
];
// ❌ Serial (≈10 s total)
for (const q of questions) {
  await llm.invoke(q); // each ~2 s
}
// ✅ Batch (≈2 s total, maxConcurrency 5)
const results = await llm.batch(questions, { maxConcurrency: 5 });

03 Prompt slimming – direct token savings

Cost per LLM call = token count × price per token. Halving tokens halves the bill.

Technique 6: Conversation history compression

Uses ConversationSummaryBufferMemory to keep the token budget bounded.

import { ConversationSummaryBufferMemory } from "langchain/memory";
import { ChatOpenAI } from "@langchain/openai";

const memory = new ConversationSummaryBufferMemory({
  llm: new ChatOpenAI({ model: "gpt-4o-mini" }),
  maxTokenLimit: 500, // auto‑summarize when >500 tokens
  returnMessages: true
});
// Long dialogs stay under the token limit, reducing usage by ~70 %

Technique 7: Structured, concise prompts

Replace verbose preambles with short task descriptions.

// ❌ Verbose (≈80 tokens)
const badPrompt = `
请你认真分析以下文本，仔细理解其中的含义，
然后从文本中提取出最重要的关键信息，
用简洁的语言进行总结，控制在100字以内，
注意要保留核心观点，去除冗余信息。
文本：{text}
`;

// ✅ Concise (≈15 tokens, same effect)
const goodPrompt = `
100字内总结以下文本核心观点：
{text}
`;

04 Model tiering and streaming output

任务分级策略：
复杂推理 → GPT‑4o / Claude Sonnet (expensive but accurate)
普通问答 → GPT‑4o‑mini (≈10× cheaper)
简单分类 → GPT‑4o‑mini 或本地模型 (几乎免费)

Technique 8: Route to appropriate model

First assess query complexity with a cheap model, then dispatch to either the cheap or powerful model.

import { ChatOpenAI } from "@langchain/openai";
import { RunnableLambda } from "@langchain/core/runnables";
import { PromptTemplate } from "@langchain/core/prompts";

const cheapModel = new ChatOpenAI({ model: "gpt-4o-mini" });
const powerfulModel = new ChatOpenAI({ model: "gpt-4o" });

const smartRouter = RunnableLambda.from(async (input) => {
  const llm = input.complexity === "complex" ? powerfulModel : cheapModel;
  return llm.invoke(input.query);
});

const complexityChecker = PromptTemplate.fromTemplate(
  `判断以下问题的复杂度，只回答 simple 或 complex：{query}`
).pipe(cheapModel);

const result = await complexityChecker.invoke({ query: "2+2等于几？" });
// result → "simple", subsequent processing uses cheapModel

Technique 9: Streaming output

Streaming does not reduce token usage but improves perceived latency.

import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI({ model: "gpt-4o-mini", streaming: true });
const stream = await llm.stream("解释一下什么是 LangGraph？");
process.stdout.write("回答: ");
for await (const chunk of stream) {
  process.stdout.write(chunk.content as string);
}
console.log(); // newline

Perceived latency: non‑streaming waits ~3 s for a full block; streaming begins showing text after ~100 ms, making the wait feel near‑zero.

05 Connection‑pool reuse

Reusing a singleton ChatOpenAI instance shares the underlying HTTP connection pool, reducing handshake overhead and allowing higher concurrency.

import { ChatOpenAI } from "@langchain/openai";

let llmInstance = null;
function getLLM() {
  if (!llmInstance) {
    llmInstance = new ChatOpenAI({
      model: "gpt-4o-mini",
      maxConcurrency: 10, // up to 10 parallel calls
      maxRetries: 3       // auto‑retry on failure
    });
  }
  return llmInstance;
}
const llm = getLLM();

06 Combined production‑grade configuration (recommended)

Integrating the above techniques yields a ready‑to‑deploy setup.

import { ChatOpenAI } from "@langchain/openai";
import { RedisCache } from "@langchain/community/caches/ioredis";
import { Redis } from "ioredis";
import { ConversationSummaryBufferMemory } from "langchain/memory";

// 1. Redis cache (1 hour TTL)
const redisClient = new Redis({ host: "localhost", port: 6379 });
const cache = new RedisCache(redisClient, { ttl: 3600 });

// 2. Main LLM with streaming, cache, and connection pool
const mainLLM = new ChatOpenAI({
  model: "gpt-4o-mini",
  streaming: true,
  cache,
  maxConcurrency: 10,
  maxRetries: 3
});

// 3. Conversation memory that compresses history
const memory = new ConversationSummaryBufferMemory({
  llm: mainLLM,
  maxTokenLimit: 500,
  returnMessages: true
});

// 4. Powerful model for complex tasks (also uses cache)
const powerLLM = new ChatOpenAI({ model: "gpt-4o", cache, maxRetries: 3 });

Observed outcomes in real projects:

Response speed: cache hits reduce latency from ~2 s to < 1 ms.

Throughput: a single instance handles up to 10 concurrent requests.

Token cost: compared with a naïve implementation, costs drop by 50‑70 %.