How to Turn Raw Text Data into an Interactive Searchable Dashboard in One Minute for Pre‑sales POCs

Problem with traditional pre‑sales demos

Manual preparation requires hand‑written mappings, custom data‑loading scripts for CSV/JSON/NDJSON, and on‑the‑fly aggregation DSL. Small errors (e.g., wrong date format, nested object mapped as keyword) cause zero documents to be indexed, Chinese analyzer mismatches produce empty results, and sensitive fields (phone, ID, email) may be written in clear text. After the demo temporary indices and uploaded files remain, requiring manual cleanup.

End‑to‑End pipeline

The workflow is modelled as a linear state machine:

uploaded → scanned → inferring → indexing → ingesting → planning → ready | error

Each stage is implemented as follows.

Upload

The frontend provides three static pages: / (upload), /progress.html (real‑time progress), and /dashboard.html (generated dashboard). Supported formats are .csv, .json, .ndjson, and .jsonl. Files are stored under data/uploads/{sessionId}/ and a session record is created.

Sensitive‑field scanning

~2,000 rows are sampled and regular expressions detect phone numbers, ID numbers, and email addresses. Users confirm which fields to mask or hash. The transformation is applied before indexing, guaranteeing that no clear‑text sensitive data reaches the demo index.

Mapping inference (dual‑path)

The system first calls the _text_structure/find_structure API. If the API is unavailable (e.g., Easysearch does not implement it) a local heuristic fallback runs:

// Prefer text_structure; fallback to local heuristic
const inferred = await inferMappings(client, filePath, format);
// inferred.source === "text_structure" | "local-heuristic"
const enhanced = enhanceMappings(inferred.mappings, { ikAvailable: healthState.ikAvailable });

Heuristic rules derived from real‑world failures:

Map all nested objects to flattened to avoid keyword rejections.

Map long numeric strings (IDs, phone numbers) to keyword to preserve precision.

Map low‑cardinality short strings to keyword for efficient terms aggregation.

Map long or Chinese strings to both text and .keyword to support search and aggregation.

If the IK plugin is detected, Chinese fields automatically receive ik_max_word and ik_smart analyzers; otherwise the standard analyzer is used and the UI shows a notice.

Data ingestion

The core writer reuses node‑es‑transformer (the same library used by elastic‑file‑ingest). CSV and NDJSON are streamed directly; JSON arrays are first converted to temporary NDJSON.

Real‑time progress

The progress page opens a Server‑Sent Events stream from GET /api/sessions/:id/progress. The stream reports the current stage, throughput, logs, and percentage completion, and automatically redirects to the dashboard when ready.

Dashboard generation

Search uses a multi_match query with field wildcards and highlights. Charts are auto‑planned based on field types: distribution bar charts, time‑trend lines, numeric cards, and composite analyses, rendered with ECharts 5 via CDN.

If a DEEPSEEK_KEY is configured, the backend calls DeepSeek to obtain a JSON decision describing chart type, aggregation type, field, and theme. The LLM never emits raw DSL or fabricated field names; the backend validates the decision against a whitelist, builds the actual Elasticsearch query, and falls back to rule‑based charts when the LLM is unavailable, times out, or returns no valid charts.

Architecture and key implementations

Layered design

flowchart TB
  frontend[Frontend: HTML/CSS/JS/ECharts]
  api[API: Express]
  services[Business layer: session/scan/mapping/ingest/dashboard]
  es[Integration layer: Transport + node‑es‑transformer]
  llm[Optional: DeepSeek]
  frontend --> api --> services
  services --> es
  services --> llm

Technology stack (no build steps): Node.js ≥ 22, ESM, Express, @elastic/elasticsearch ^8.17, node‑es‑transformer, optional DeepSeek, ECharts 5 CDN.

Easysearch compatibility layer

Easysearch returns the header X‑Elastic‑Product, which the official client rejects. Subclassing Transport and disabling the product check resolves the issue:

class CompatibleTransport extends Transport {
  constructor(opts) {
    super({ ...opts, productCheck: null }); // disables product check
  }
}

This allows the same client code to communicate with both Easysearch and Elasticsearch.

Ingestion orchestrator

The script ingest‑runner.js runs the full chain: infer → create index → transform (mask/hash) → stream write → refresh/count → dashboard planning.

Index names embed the session ID and timestamp, e.g. poc-demo-{sessionId}-{timestamp}. Ownership is verified with a regular expression to prevent accidental deletion of unrelated indices.

LLM boundary enforcement

DeepSeek only decides which charts are valuable; it never emits raw DSL. The backend validates chart type, aggregation type, field existence, and sensitive‑field exclusion before translating the decision into a concrete query. If validation fails or the LLM is unavailable, the system falls back to the rule‑based dashboard.

Degradation handling

Cluster unreachable – service starts but upload and ingestion are disabled.

IK plugin missing – use the standard analyzer and show a UI notice that Chinese tokenization will be weaker.

No DEEPSEEK_KEY or call fails – generate a rule‑based dashboard. _text_structure unavailable – fall back to local heuristic mapping.

Zero documents ingested – report a clear error instead of pretending success.

Cleanup

Delete the temporary index named poc-demo-{sessionId}-{timestamp}.

Delete the uploaded files under data/uploads/{sessionId}/.

Remove the session record.

Before deletion the index name is matched against the session ID regular expression to avoid removing unrelated indices.