How Cerebras Built a Scalable Internal Knowledge Base: Hybrid Search, Slack Ingestion & MCP Integration
Cerebras shares how they built an internal knowledge base handling 15,000+ daily queries by ingesting data from Slack, code repos, and custom sources into a unified Postgres embedding store, using hybrid search (full-text, vector, IDF, time decay), CocoIndex for incremental code embeddings, RRF reranking, and MCP-exposed retrieval primitives for both agents and a web UI.
Employees at Cerebras ask over 15,000 questions per day to an internal knowledge base that has become one of the company's most widely adopted tools since its launch three months ago. The system serves humans, automated workflows, and AI agents across teams spanning data-center operations, chip design, hardware, training, inference, and cloud platform.
Meet Data Where It Lives
Information naturally resides in the tools best suited for each domain: suggested edits in docs, discussions in Slack, code references in GitHub, status metadata in Jira. Forcing everything into a single platform rarely works. Cerebras therefore designed a system that minimizes behavior change by pulling data directly from each platform.
Knowledge Base Anatomy
The core is a single Postgres table storing embeddings, raw excerpts, and metadata from many sources. A simple, uniform interface lets any developer build custom connectors; every source — from Slack threads to netlists — writes rows to the same embedding table, all immediately queryable through one API. Each data source defines what data it holds, how to connect, and the fetch cadence.
Handling Unstructured Slack Conversations
Slack is the primary source for up-to-date engineering discussions. Simple vector search on raw text proved insufficient. Cerebras built a hybrid retrieval pipeline for each thread, combining four complementary techniques:
Full-text search captures exact tokens (error strings, flag names, hostnames) that embeddings blur; lexical matches rank highest when an engineer pastes a raw error.
Embedding search connects paraphrased queries and answers (e.g., "restore hangs after manifest load" vs. "checkpoint stalls on the NFS mount") with no shared vocabulary.
Inverse document frequency (IDF) separates signal from filler; short messages around rare tokens (obscure config flags) score higher, while generic replies like "sounds good, thanks!" approach zero.
Time decay reflects that Slack answers expire; a six-month-old thread describing decommissioned infrastructure loses to a newer one when relevance is otherwise equal.
No single scorer is trusted alone. Each technique produces its own ranked view of the same corpus; the views are fused at query time (see Reranking).
Socket Mode for Real-Time Ingestion
A Slack bot runs in Socket Mode, receiving every message event via a persistent WebSocket. This avoids polling the Web API and its rate limits. On each event, the system acknowledges immediately, deduplicates via stable event IDs, and hands the message to an ingestion consumer. The consumer resolves the full thread (parent + all replies) from the Slack API and writes the entire conversation as one row, ensuring stored content, participant lists, and last-activity timestamps always reflect the complete dialogue. Each Slack channel maps to its own data source, enabling per-channel freshness tuning (e.g., higher ingest frequency for a busy incident channel).
Thread Distillation and Splitting
Raw Slack text is keyword-searchable via a Postgres GIN index on the raw content. For vector search, an LLM distills each full thread into structured fields (summary, key decisions, participants, etc.). These distilled fields are embedded and written to the shared embedding table; the raw transcript is not embedded directly. Experiments showed significantly higher accuracy when threads are normalized to a consistent format. Additional metadata provides stronger semantic signals.
Long threads still hid important messages inside thread-level summaries. To boost per-message signal, Cerebras introduced splits — contiguous message runs from the same author. Each split is embedded with the thread topic prepended as context. This surfaces answers buried in off-topic messages whose vocabulary never reached the thread summary. Every split is scored against weighted signals (reaction count, reply depth, recency, author expertise, etc.) and only embedded if it exceeds a threshold.
Code Repositories with CocoIndex
Initially skeptical about embedding code ("grep is enough"), the team adopted CocoIndex, an open-source document-embedding framework focused on codebases, after reviewing Cursor's findings on semantic search in large repos. Cerebras has many internal repos, some exceeding 40 GB; the priority was efficient incremental updates.
For each repo, code is split by language-specific regex boundaries ordered coarse-to-fine: first class-level, then method-level, then smaller blocks. Chunks are embedded and vectors written to Postgres. A single file can yield multiple embeddings at different granularities (file-level and function-level records). CocoIndex tracks sync metadata in Postgres; on each commit it re-embeds and re-exports only changed chunks, not the entire codebase. This works especially well because sync state and embeddings share the same database. As repo count grew, onboarding moved to config files (allow/deny lists at file-path level) that teams submit via pull requests.
Custom Data Sources as Plugins
Teams with existing databases didn't want to migrate data to Slack or docs. Custom sources are treated as plugin scripts: a team submits a PR containing a small Python module that reads from its system and emits rows matching the embedding table schema, plus a matching data-source entry. As long as the script writes to the shared database using the same schema, the rest of the stack requires no changes; the data becomes queryable alongside Slack, code, and docs.
Planning and Tool Fan-Out
For each query, a brief planning step uses an LLM to decide which tools and data sources are relevant. The planner works from concise descriptions of indexed projects, their available data sources, and each source's strengths. Given the user query and current scope, it outputs tool selections; an executor fans out these calls in parallel, normalizes results into a unified evidence format (score, recency, source hints), and passes the package to a final synthesis LLM.
Reranking
Before reranking, Reciprocal Rank Fusion (RRF) merges incompatible result lists from different retrievers. For each document appearing in a list, add weight / (60 + rank); default weight 1.0, smoothing constant 60. The smoothing constant makes consensus across retrievers outweigh a single strong vote: a document appearing near the top in multiple retrievers can beat one ranked first in only one retriever. Duplicate chunks are merged back to their source, per-file result caps are applied, and a diversified top-20 set emerges.
A small reranker model then scores each candidate 0–10 against the original query; the top 10 are kept. Finally, surrounding context is re-attached to winners (e.g., pulling two adjacent wiki sections when a subsection matches) so headings, preconditions, and caveats lost to chunking are restored. The search output is a rich evidence pack: fused from multiple retrievers, deduplicated at source level, reranked for the actual question, then expanded with neighboring context.
MCP Integration
In the Model Context Protocol (MCP) integration, retrieval primitives are exposed as direct, simple tools rather than hidden behind a "answer this question" endpoint. Tools are intentionally low-LLM-dependency so clients can call them fast and cheap. Each MCP tool maps to a core retrieval primitive (search_slack, search_code, search, who_knows). Inputs and outputs are narrow, structured, and stable, letting any client or agent invoke them without embedding orchestration logic inside the tool. Most tools run a query pipeline (vector, lexical, or ripgrep), apply lightweight scoring heuristics, and return raw evidence rows. The orchestration engine (Claude Code or any MCP-compatible agent) decides which tools to call, in what order, and how to assemble results into a final answer or code edit. The retrieval layer itself does not depend on those LLM decisions.
Web UI
The Web UI uses the same tools but wires them into a full query pipeline that runs end-to-end for each user question. A UI agent handles the planner and executor steps:
Planner: one lightweight LLM pass inspects the query and current project, selects retrieval tools (e.g., search, search_slack, subsystem_index).
Executor: system fans out tool calls in parallel, collects results, normalizes them into a unified evidence schema with scores, recency, and source hints.
Synthesis: a final LLM pass receives the typed evidence pack and original question, generates the displayed answer with citations, caveats, and cross-source synthesis.
From the user perspective it's "ask a question, get an answer"; underneath it runs the same "planner → executor → synthesizer" pattern that MCP clients can explicitly replicate.
Organization: Projects and Scoped Search
As the corpus grew, "search everything everywhere" became noisy. Compiler engineers didn't want infrastructure runbooks, and vice versa. Projects are the primary way to scope queries. A project is a named collection of data sources: specific Slack channels, code repos, internal databases, and doc spaces relevant to a team or initiative. Projects are lightweight; the same data source (e.g., a shared incident channel or central platform repo) can be referenced by multiple projects without duplication.
Onboarding and Defaults
During onboarding, users pick or create a default project matching their work area (ML training infrastructure, compiler, data-center operations, etc.). The default is stored in the user profile and automatically scopes queries. New engineers get high signal-to-noise answers without first learning which Slack channels, repos, or doc spaces matter.
Closing Thoughts
The knowledge base works because it meets people where their information already lives, rather than forcing everything into a rigid system. By combining multiple search techniques, it surfaces evidence quickly. The result is a search experience flexible enough for real company data and structured enough to stay useful as Cerebras keeps growing.
Signed-in readers can open the original source through BestHub's protected redirect.
This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactand we will review it promptly.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.
