The explainability problem in fraud ML

14 July 2025 · Soo-Yeon Park

Gradient-boosted trees and neural networks can reach impressive AUC scores on fraud datasets. But an analyst looking at a flagged transaction doesn't have time to wonder why the model fired. They need a reason — in plain language, in under a second — to make a decision they can defend.

Why accuracy alone fails in production

We ran an experiment early on: we deployed a high-accuracy model with no explanations and gave analysts only a risk score and a threshold. False positive rates looked good on paper. But analysts started overriding alerts at a high rate — not because the model was wrong, but because they didn't trust what they couldn't understand. A model that analysts routinely override is not a useful model.

"Trust is a product requirement, not a nice-to-have. A black box will be ignored — even a correct one."

Our approach: attribution at inference time

We use a combination of SHAP values for tree-based models and attention weights for our sequence models, computed at inference time and mapped to human-readable feature names. "This event scored high because: (1) IP first seen 4 minutes ago, (2) third transaction from this device in 90 seconds, (3) shipping address added 8 minutes before checkout."

The computation adds about 40ms to our latency budget. That's a deliberate trade-off we're happy with: the alternative is a score that gets ignored.

What we're still working on

Contrastive explanations — "this would not have flagged if the IP had been seen before" — are genuinely useful and we ship them in limited cases, but generating them reliably across our model cascade is still an open problem. It's the next big thing on our ML roadmap.