We expected the pilot data to be full of obvious failures. Voltage spikes, error codes that name the broken part, connector states flipping through sequences the spec says are impossible. All of that is in there. It just didn't turn out to be the useful part.
The useful part was the traffic that looked like nothing.
The signal before the signal
The best predictor of failure we found wasn't an error code at all. It was drift in how regularly a charger sent MeterValues.
Healthy chargers report on a steady interval, usually every 60 seconds depending on how they're configured. Chargers that died within the next 48 to 72 hours got sloppy about it first. The variance in that interval crept up. We're talking a few seconds, not minutes, but it was consistent enough to build on.
Threshold monitoring will never catch this. Nothing crosses a limit. You only see it if you know what a specific charger's normal looks like and you're watching for departures from it.
Why per-charger baselines matter
Fleet averages hide too much. A unit at a highway rest stop sees completely different load than one in a downtown garage: different temperature swings, different session frequency, different hardware generation, usually different firmware. Normal is local. So we fit a model per charger rather than one model for the fleet.
That's also why the anomaly detection layer uses Isolation Forest. It doesn't need labeled failures, which matters, because a fleet that's being maintained properly doesn't produce many to label. It learns what the bulk of the data looks like and flags the points that are easy to separate from it.
1.4 billion messages, in practice
The first problem at this scale isn't modeling. It's schema.
OCPP is a protocol, not a data model, and vendors interpret it loosely. Some send StatusNotification on every minor state change; others batch them. Some populate StopTransactionReason; others leave it out. We had to write a normalization layer before we could ask a single interesting question of the data.
Once it's normalized, sessions turn out to be pretty distinctive. The ramp-up when power delivery starts, the shape of the plateau, how the teardown goes when the car unplugs. A charger tends to repeat its own pattern. Watching for a session that doesn't match a charger's usual shape has told us more than any individual message ever has.
What we haven't figured out
1.4 billion messages, one fleet, a few months. That's a start.
Some failure modes we can now predict with real confidence. Others we can only find looking backwards. We can point at the historical data and see the signal sitting right there, and we still don't have a model that would have flagged it at the time.
Closing that gap is most of the current work. The information is in the stream. Whether our models are sensitive enough to pull it out early enough to be worth acting on is a separate question, and for several failure modes the answer is still no.