How to Build a Multi‑Dimensional Evaluation Framework for AI‑Powered Data Analysis Platforms

Introduction

In the era of rapid AI development, the department launched the DataV‑Note intelligent analysis creation platform two years ago, offering services such as data insight, industry report generation, and AI‑assisted academic/medical report rewriting, aiming to deeply fuse data value with textual expression.

Feedback from sales and users shows two core issues: a lack of unified evaluation standards and ongoing disputes over accuracy and technical maturity, which hinder product value communication and industry standardization.

2. Establishing Quantitative Evaluation Standards and Building the Evaluation Model

2.1 Evaluation Model Objectives

Product verification: establish quantifiable accuracy metrics and output reports meeting industry standards.

Competitive analysis: generate differentiated competition evaluation reports through multi‑dimensional comparison.

Automated testing: perform regression testing for model switching, prompt optimization, and AI engineering improvements.

Accuracy improvement: embed the evaluation model into the product optimization loop to dynamically calibrate hallucinations.

2.2 Preliminary Design of the Evaluation Model

The Qwen VL model is selected for content extraction and the Qwen‑3 model for evaluation, forming the technical architecture shown below.

2.3 Design Details

Key details include visual‑recognition prompt tuning, ensuring complete description of visual elements, clear operation steps, and strict boundary limits to avoid hallucinations.

## Role
You are a professional image analysis expert, adept at extracting charts, tables, code, and text from images and describing their detailed information and values.

## Tasks
### Task 1: Extract chart information
- Identify chart type (e.g., bar, line, pie)
- Capture chart title (or output "None")
- Extract axis metadata and full data values

### Task 2: Extract table information
- Identify column headers and table content

### Task 3: Extract code information
- Detect language (SQL or Python) and content

### Task 4: Extract textual information
- Distinguish content vs. comments

## Output format
{ 'filename': 'xxx', 'title': 'xxx', 'body': [{ 'section_title': 'xxx', 'content': [{ 'type': 'chart', 'chart_type': 'xxx', 'title': 'xxx', 'metadata': 'xxx', 'data': {...} }, ...] }] }

2.4 Evaluation Standards

Two assessment methods are used: vertical evaluation (generating 5‑10 reports per question and scoring them on basic, visualization, and attribution dimensions) and horizontal comparison (aligning themes, conclusions, core metrics, and chart consistency across reports).

3. Future Plans

Plans include automating cross‑platform analysis via browser‑user integration and embedding the evaluation model into knowledge assessment and RAG pipelines to improve accuracy.

4. Conclusion

The evaluation system demonstrates the potential of large models while highlighting challenges in achieving precise data‑analysis control at the product level, inviting further community input.