How We Built an Experience Cockpit: From Data Collection to Actionable Insights

Background

The Experience Cockpit is a one‑stop experience data monitoring platform created by 58UXD and ADS, aimed at improving overall user experience by capturing data across all user scenarios, providing monitoring, dashboards, and alerting capabilities.

Why Experience Data Monitoring?

Poor user experiences such as missing entry points or unresponsive buttons can damage product reputation, especially in critical flows like ordering and payment. Continuous experience management helps businesses understand user behavior, identify pain points, and enhance emotional appeal.

Metric Framework

The core North Star metric combines satisfaction and NPS to assess both feature usage and overall recommendation willingness. This is broken down into independent secondary metrics and further into scene metrics aligned with user journeys, allowing granular analysis of each interaction.

Non‑Intrusive User Research

To avoid disrupting normal usage, a non‑intrusive strategy was implemented, controlling IP fatigue by limiting repeat submissions and using rotation rules to ensure diverse question exposure. Users receive surveys only once per session, and a cooldown period prevents repeated prompts.

Data Collection

Experience data is gathered through multiple touchpoints that blend with the product’s visual style, reducing perceived questionnaire burden.

Data Processing

Collected data is weighted and calculated daily via scheduled scripts, updating metrics in real time. For open‑ended text feedback, AI and NLP techniques remove filler words, extract key phrases, cluster topics, and classify sentiment into positive, negative, or neutral, producing sentiment word clouds.

Data Presentation

Dashboard design starts with identifying who needs to see the data and what information they require. Three role‑based dashboards were created, supporting drill‑down analysis to pinpoint issues. Visualizations follow data‑visualization best practices to convey information clearly and efficiently.

Alerting and Data‑Driven Action

When metrics cross predefined thresholds (e.g., NPS below 30 or a 50% drop YoY), the system generates alert tickets, guiding users to the abnormal scene for timely response.

Weekly Reporting and Mobile Support

A weekly report is pushed to product directors, and the cockpit is accessible via enterprise WeChat and other mobile channels, ensuring stakeholders can view data and alerts anytime.

Future Directions

Plans include integrating operational data for multi‑dimensional measurement, deploying QMD and B‑Metric models for consumer and enterprise products, and further automating the experience measurement workflow.

References

Understanding Human Behavior: The Softer Side of Experience Management

“Experience Measurement” – Thoughtworks

“Good Experience, Good Business” – Ipsos

Customer Experience Management – Bernd H. Schmitt

User Experience Measurement – Tom Tullis & Bill Albert

Data Visualization – Chen Wei et al.