Can You Predict Switch Failures Before They Happen? Inside PreFix’s ML Approach

Introduction

Switch failures are a frequent cause of performance degradation in modern datacenter networks. Traditional fault‑tolerance methods focus on protocol changes or topology redesign, which cannot cover all switch types and often require manual diagnosis. PreFix proposes predicting failures before they occur, enabling proactive remediation.

Problem Definition and Design Challenges

PreFix defines switch‑failure prediction as determining, during operation, whether a switch will fail within a near‑future time window (e.g., 0.5–24 hours). The task is cast as a time‑slice classification problem: each fixed‑length slice (e.g., 15 minutes) is labeled as a “symptom” slice if a failure follows, otherwise as “non‑symptom”. Challenges include noisy log signals, severe class imbalance, and high computational cost of processing hundreds of thousands of log messages per prediction interval.

PreFix System Overview

In the offline phase, PreFix learns log‑message templates from historical system logs using the FT‑tree method, then extracts four template‑sequence features (frequency, periodicity, sequence, burstiness) for each time slice. A random‑forest classifier is trained on labeled slices. During online prediction, real‑time logs are mapped to templates, the same four features are computed, and the trained model decides whether the slice indicates an imminent failure.

Feature Extraction Details

• Frequency : an N‑dimensional vector counting occurrences of each template within the slice. • Periodicity : a metric aj measuring the regularity of a template’s appearance over time. • Sequence : common subsequence patterns extracted via a two‑stage longest common subsequence algorithm (LCS²) to capture shared fault‑precursor sequences while filtering noise. • Burstiness : detection of sudden spikes in template occurrence using singular‑spectrum analysis, indicating abrupt behavior changes.

Experimental Evaluation

Data were collected from over 20 datacenters, covering three switch models (M1, M2, M3) from two manufacturers, amounting to thousands of switches and two years of logs with manually verified hardware failures. PreFix was compared against two existing log‑based fault‑prediction methods, SKSVM and HSMM, using PR‑curve metrics. Results show PreFix consistently outperforms both baselines across all models, achieving higher precision and recall while maintaining reasonable computational overhead.

To demonstrate generality, PreFix was also applied to hard‑disk failure prediction using the Backblaze ST4000DM000 dataset (29,084 disks). The PR‑curve again indicated superior performance over baseline methods.

Conclusion

PreFix introduces a novel framework that learns log‑template patterns and leverages random‑forest classification to predict imminent switch hardware failures. Experiments on large‑scale datacenter deployments confirm its accuracy advantage over prior approaches and its applicability to other hardware‑failure domains such as disk prediction.