Train a fault detector on real charging data, optimize for accuracy, and you'll get a model with excellent accuracy that cannot do the job.
The imbalance problem
In a fleet that's being maintained, failures are rare. Across our pilot data, fault conditions run somewhere between 1 and 3% of session-level observations, depending on where you draw the line on what counts as a fault.
So a model that answers "no fault" every single time scores 97 to 99%. It also catches nothing, ever.
None of this is specific to charging. Fraud detection, medical diagnosis, equipment monitoring of any kind: anywhere the interesting event is rare, accuracy stops being informative. It isn't measuring the thing you care about.
What F1 measures
F1 is the harmonic mean of precision and recall.
Precision asks: of the sessions we flagged, how many were really faults? Recall asks: of the real faults, how many did we flag?
Flag everything and recall is perfect while precision collapses. Flag almost nothing and precision looks respectable while recall is nowhere. F1 punishes both. If you want to deliberately favor one side, F-beta lets you weight it, and we tilt slightly toward recall, since a missed failure costs an operator more than a false alarm does.
Where AUC-ROC comes in
AUC-ROC asks a different question: across every possible threshold, can the model separate the two classes at all? Pick a random fault and a random non-fault. Does it score the fault higher? An AUC of 0.5 means it's guessing. 1.0 means perfect separation.
F1 evaluates one threshold. AUC-ROC ignores thresholds entirely. We report both because a model can post a strong AUC and a mediocre F1 when the decision boundary is badly calibrated, and the reverse happens too.
Handling the imbalance
We intervene in three places. In the data, SMOTE for some model variants, though not all. In training, class-weighted loss that makes minority-class mistakes more expensive. In evaluation, stratified cross-validation so fault cases stay proportionally represented in every fold.
None of that solves anything by itself. The imbalance is a real property of the problem, not an artifact we can preprocess away. What these techniques buy us is that the model can't take the lazy route of predicting the majority class and calling it done. The harder question is whether our features actually encode what separates a pre-fault charger from a healthy one, and that's feature engineering, not metrics.
In practice
Operators ask how accurate the model is. We don't answer that question. We give precision, recall, and AUC-ROC on held-out data from their hardware class, because those are numbers they can act on. The accuracy figure would just be flattering.