Applying Large Language Models in Zhihu’s Jianqiao Enterprise Analytics Platform

Introduction

Jianqiao is Zhihu’s internal enterprise analytics platform that supports scenarios such as finding people, finding content, monitoring, opportunity discovery, and problem investigation. The article explores how large language models (LLMs) are integrated into three core business scenarios: knowledge taxonomy organization, natural‑language‑to‑filter conversion, and natural‑language data analysis.

01 Business Status and Background

The platform aggregates external news into events, generates questions, and creates discussion spaces, while also reorganizing internal content into a structured knowledge system. These three LLM‑driven scenarios each have distinct technical and business characteristics.

02 Knowledge Taxonomy Organization

Two business forms are addressed: event aggregation (clustering news from external sources) and content sedimentation (building multi‑level classification trees for internal knowledge). Challenges include clustering accuracy, max‑token limits, and workflow complexity. A four‑stage MapReduce‑like pipeline was designed:

Extract key information from news and embed it.

Iterative high‑precision clustering with LLM‑generated event names.

High‑recall clustering followed by LLM‑based event merging.

Generate final event‑to‑news mappings for business consumption.

Benefits: automatic event naming, improved accuracy, token‑size reduction, and simplified workflow.

For knowledge sedimentation, a similar MapReduce process splits content, generates classification names (map), merges them iteratively (reduce), and writes merged results, handling max‑token constraints and parallel execution.

03 Natural Language to Filter Conditions

This scenario supports packaging, people search, and content search. Because filter conditions are numerous and logically complex, LLM fine‑tuning was adopted. Data construction involved four stages: atomic condition generation, logical composition (AND/OR/NOT), fuzzy statement creation, and intentionally erroneous examples for robustness.

Three iterative model versions tackled issues such as output‑input mismatch, JSON truncation, output repetition, format errors, missing conditions, and comparison‑operator mistakes. Training speed was improved by reducing epochs.

After deployment, the system reduced usage cost, increased user adoption, lowered onboarding friction, and automated tag generation, thereby improving communication efficiency.

04 Natural Language Data Analysis

Ad‑hoc analysis requires translating varied natural language queries into appropriate SQL while ensuring data relevance and handling token limits. A dynamic prompt strategy was employed:

Encode sample Q‑A pairs into embeddings stored in FAISS.

At query time, retrieve top‑10 similar questions via MMR, then construct a prompt that respects max‑token constraints, highlighting de‑duplicated column names, example queries, and the current user question.

Challenges such as cosine similarity overload, data‑source alignment, and ambiguous user input were mitigated with MMR diversity, user‑selected data sources, and iterative sample augmentation based on feedback.

05 Summary and Outlook

Key pain points identified include lack of mature PE practices, max‑token constraints, absence of prompt best practices, prompt injection risks, LLM latency, insufficient frameworks for large‑scale or complex scenarios, and the difficulty of constructing fine‑tuning data. Future directions involve building dedicated frameworks for complex LLM tasks, leveraging business imagination to push model capabilities, and further fine‑tuning to improve accuracy.

06 Q&A

Q1: How to select LLMs for event aggregation? A: Run multiple models with tuned prompts, compare accuracy on batch tasks, and choose the best performer.

Q2: Evaluation methods for event aggregation? A: Diff outputs of LLM and existing online models on 4 × 100 cases and conduct manual reasonableness assessment.

Q3: When to stop iteration in knowledge organization? A: Based on leaf‑node count and maximum tree depth.

Overall, the sharing demonstrates practical LLM integration in enterprise analytics, highlighting both successes and ongoing challenges.