How to Tame Massive New‑Energy‑Vehicle Logs with Cloud‑Native SLS Data Cleaning

In the new‑energy‑vehicle industry, logs from vehicle, mobile and cloud sides generate massive, weakly‑structured data, making standardisation, analysis and real‑time operations extremely difficult.

Five Core Log Challenges

Mixed log cleaning : Diverse formats (syslog, JSON, KV) and inconsistent fields across sources.

Data aggregation : Need to securely and efficiently gather logs from multiple regions and clouds.

Second‑level response : Real‑time collection and query must meet sub‑second latency for incident handling.

Cost control : Large data flows increase temporary storage costs.

Process transparency : Complex data pipelines obscure lineage and hinder troubleshooting.

SLS Data‑Cleaning Solution

SLS offers a one‑stop data‑cleaning capability that unifies log ingestion from SDK, Logtail, OSS, Kafka, Syslog and other agents, performs intelligent field extraction, supports custom regular expressions, and provides dynamic privacy‑preserving masking for sensitive fields such as VIN, location and phone numbers.

1. Mixed Log Cleaning – Full‑Chain Ingestion

Multi‑source unified ingestion : Logs from vehicle, phone and backend services are written to a single Logstore via SDK, Logtail, OSS Bucket or open protocols.

2. Data Aggregation – Cross‑Region Processing

SLS can process logs across regions, consolidating them into a single Logstore for unified analysis, eliminating data silos.

3. Second‑Level Response – High‑Performance Cleaning

SLS processes up to 2 Million records per second (≈2 GB/s) with linear scalability, enabling sub‑second query and alerting during emergencies.

4. Cost Control – Zero‑Temporary‑Storage Processor

Write‑processor transforms data during ingestion, eliminating the need for intermediate Logstores and reducing storage expenses.

5. Process Transparency – Logstore Lineage Visualization

Using Alibaba Cloud Observability MCP, users can list projects, tasks and generate Mermaid diagrams that illustrate data flow between Logstores.

Write‑Processor Examples

Vehicle arrival log (JSON)

{
  "ts":"2024-06-17T11:26:00Z",
  "vin":"LVSHF6196L1234567",
  "event":"arrive_station",
  "station_id":"SH_CHG_001",
  "location":{"lat":31.2362,"lon":121.4798}
}

Processor:

* | parse-json content
  | parse-json location
  | project-away content,location
  | extend __tidme__ = cast(to_unixtime(date_parse(ts,'%Y-%m-%dT%H:%i:%sZ')) as bigint)

Result:

__time__: 1718623560
vin: LVSHF6196L1234567
event: arrive_station
station_id: SH_CHG_001
lat: 31.2362
lon: 121.4798

Charging plug‑in log (KV)

2024-06-17T11:27:11Z station=SH_CHG_001 event=plug_in vin=LVSHF6196L1234567 result=success connector=DC

Processor:

* | parse-regexp content '(\d+-\d+-\d+T\d{2}:\d{2}:\d{2}Z) (.*)' as time, content
  | parse-kv content ' ' '='
  | project-away content
  | extend __time__ = cast(to_unixtime(date_parse(time,'%Y-%m-%dT%H:%i:%sZ')) as bigint)

__time__: 1718623631
connector: DC
event: plug_in
result: success
station: SH_CHG_001
time: 2024-06-17T11:27:11Z
vin: LVSHF6196L1234567

Charging start log (mobile KV)

2024-06-17 11:27:12 user_id=87234 action=charge_start vin=LVSHF6196L1234567 station_id=SH_CHG_001 os=android13

* | parse-regexp content '(\d+-\d+-\d+ \d{2}:\d{2}:\d{2}) (.*)' as time, content
  | parse-kv content ' ' '='
  | project-away content
  | extend __time__ = cast(to_unixtime(date_parse(time,'%Y-%m-%d %H:%i:%s')) as bigint)

__time__: 1718623632
action: charge_start
os: android13
station_id: SH_CHG_001
time: 2024-06-17 11:27:12
user_id: 87234
vin: LVSHF6196L1234567

Operational SQL Analyses

1. Daily total charging sessions and energy:

SELECT station_id,
       date_trunc('day', __time__) AS day,
       COUNT(*) AS charge_count,
       SUM(total_kwh) AS total_kwh
FROM log
WHERE event = 'charging_stop'
GROUP BY station_id, day
ORDER BY charge_count DESC;

2. Charging failure reasons:

SELECT reason,
       COUNT(*) AS fail_count
FROM log
WHERE event IN ('charging_start','plug_in','charging_auth')
  AND result = 'fail'
GROUP BY reason
ORDER BY fail_count DESC;

3. Full‑chain charging trace (VIN‑ordered):

SELECT * FROM log WHERE vin = 'LVSHF6196L1234567' ORDER BY __time__;

4. SOC growth curve and charging speed:

SELECT date_trunc('minute', __time__) AS dt, battery_soc
FROM log
WHERE vin='LVSHF6196L0034' AND station_id='SH_CHG_008'
ORDER BY dt;

5. Detect abnormally low charging speed (SOC increase < 0.2 per hour):

SELECT date_trunc('hour', __time__) AS dt,
       vin,
       station_id,
       (MAX(battery_soc)-MIN(battery_soc)) AS avg_soc_per_hour
FROM log
WHERE event='charging_status'
GROUP BY dt, vin, station_id
HAVING avg_soc_per_hour < 0.2;

6. Detect frequent interruptions (more than 3 start/stop pairs within an hour):

SELECT vin, station_id,
       COUNT(*) AS interrupt_count
FROM log
WHERE event IN ('charging_start','charging_stop')
GROUP BY vin, station_id
HAVING interrupt_count > 3;

Conclusion

New‑energy‑vehicle logs are massive, heterogeneous and often unstructured, creating challenges in cleaning, cross‑region aggregation, cost control and operational transparency. Alibaba Cloud SLS addresses these issues with cloud‑native, one‑stop ingestion, intelligent parsing, privacy masking, high‑throughput real‑time processing, zero‑temporary‑storage write processors and visual lineage tools. The charging‑scenario demo shows how unified cleaning enables comprehensive fault‑diagnosis, performance monitoring and business insight.

References

SPL syntax – https://help.aliyun.com/zh/sls/spl-overview/

Write processor – https://help.aliyun.com/zh/sls/sls-write-processor/

Comparison of ingest processor, Logtail plug‑in and data transformation – https://help.aliyun.com/zh/sls/comparison-of-ingest-processor-logtail-plug-in-and-data-transformation

Alibaba Cloud Observability MCP – https://github.com/aliyun/alibabacloud-observability-mcp-server