Why Are LLM Stacks Becoming Essential for Modern Companies?

ChatGPT’s breakthrough with large language models (LLMs) has sparked a wave of technological innovation, prompting many companies to embed natural‑language interaction into their products and forming a new LLM‑centric tech stack.

We observe impressive auto‑complete features across domains—from programming (Sourcegraph, Warp, GitHub) to data science (Hex)—and increasingly sophisticated chatbots serving customer support, employee assistance, entertainment, and more.

Companies are re‑imagining workflows with AI‑first approaches in visual art (Midjourney), marketing (HubSpot, Jasper), sales (Gong), contact centers (Cresta), legal (Ironclad), accounting (Pilot), productivity (Notion), data engineering (dbt), search (Glean), shopping (Instacart), payments (Klarna), and travel planning (Airbnb). This is just the beginning.

According to recent surveys, 65% of firms have deployed LLM‑based applications to production—up from 50% two months earlier—while the remainder are still experimenting.

94% of companies use foundational model APIs; OpenAI’s GPT dominates with a 91% share, and Anthropic’s usage rose to 15% last quarter, with many firms employing multiple models.

88% consider retrieval technologies, such as vector databases, a critical component of their stack. Retrieval‑augmented models improve result quality, reduce hallucinations, and address data freshness. Companies adopt dedicated vector stores (Pinecone, Weaviate, Chroma, Qdrant, Milvus) or use pgvector/AWS solutions.

38% are interested in LLM orchestration frameworks like LangChain for prototyping or production, with adoption increasing over recent months.

Less than 10% currently seek tools for monitoring LLM output, cost, performance, or A/B testing prompts, though this demand may grow as larger enterprises and regulated industries adopt LLMs.

15% have built custom language models from scratch or using open‑source resources, often alongside LLM APIs. Interest in training custom models has risen, requiring dedicated compute stacks, model hubs, hosting, training frameworks, and experiment tracking—services offered by Hugging Face, Replicate, Foundry, Tecton, Weights & Biases, PyTorch, Scale, and others.

Overall, respondents agree that LLM APIs will remain a core pillar, followed by retrieval mechanisms and development frameworks like LangChain.

Open‑source and custom model training and fine‑tuning are also growing, though other areas of the LLM stack remain less mature.

Companies Want to Customize LLMs to Their Own Context

Enterprises aim to enable natural‑language interaction over their own data—documentation, product catalogs, HR policies, IT rules, personal notes, design layouts, metrics, codebases, etc.

There are three primary ways to customize LLMs:

1. Train a custom model from scratch, which is the most difficult and requires expert ML scientists, large datasets, and substantial compute infrastructure.

2. Fine‑tune a base model, a medium‑difficulty approach that adds domain‑specific data to pre‑trained weights; while increasingly feasible thanks to open‑source tools, it still typically needs a mature team and can introduce unexpected model drift or loss of capabilities.

3. Use a pre‑trained model with retrieval of relevant context, the simplest method. Instead of a fully fine‑tuned model, many solutions supply the right information at inference time via SQL queries, product catalog searches, external APIs, or embedding‑based retrieval.

We face two stack choices: a closed‑source LLM‑API‑centric stack aimed at developers, or an open‑source custom‑model stack designed for more advanced ML teams.

As AI interest grows and open‑source development accelerates, more companies are exploring training and fine‑tuning their own models.

We anticipate convergence of the LLM‑API and custom‑model stacks over time—companies may train models from open‑source projects while leveraging vector databases for retrieval, and startups are building tools to support this hybrid approach.

The community engaging with LLMs is expanding beyond specialist AI researchers to all types of developers, and we expect a richer ecosystem of AI‑powered developer tools to emerge.