A better fraud score can hide a worse model

Begin with the decision, not the metric

A fraud model usually participates in a decision process: an analyst reviews an alert, a transaction is delayed, or a case is escalated. These actions have different costs and different tolerance for uncertainty. Before choosing an imbalance technique, I would define the action the score supports and the information available when that action is taken. A model can improve a convenient offline metric while sending investigators more unhelpful alerts.

This protocol studies a hypothetical transaction-review queue. It compares an unweighted baseline, class weighting, and SMOTE under the same feature and evaluation rules. There are no measured fraud outcomes in this article. The intended result is a reproducible comparison that could justify using or rejecting each technique. In particular, I would not assume that making the training distribution more balanced makes the operational system better.

Build the timeline before building features

Every record needs at least an event time and a label-availability time. A transaction may occur today while its eventual disposition is known much later. If the experiment trains on labels that would not have existed at the historical training cutoff, it answers an easier question than deployment faces. The training snapshot must therefore include only eligible events whose labels had matured by that cutoff.

I would then allocate later periods to model selection, calibration or threshold selection, and final testing. The exact interval lengths should reflect the business cycle and label delay rather than a convenient percentage split. Repeated rolling evaluations help reveal whether a result depends on one favorable interval. TimeSeriesSplit documents ordered evaluation and a gap parameter, but applying a generic splitter does not by itself implement domain-specific label maturity or entity separation.

Point-in-time features must follow the same logic. A customer total should include only transactions known before the prediction, not a current warehouse aggregate rebuilt after the outcome. Shared devices, merchants, and accounts introduce another dependence: the same entity may occur in several periods. I would report both the intended recurring-entity deployment setting and a separate new-entity slice, instead of pretending that one split answers both questions.

Keep resampling inside the training boundary

The imbalanced-learn documentation identifies resampling before the train/test split as a leakage pitfall. The evaluation set should retain the distribution appropriate to the deployment question, while synthetic samples belong only to training. This is a narrow but essential rule. It does not mean that every resampling pipeline is automatically free from leakage; the feature table itself may already contain future information.

The example below assumes binary labels, a minority-to-majority ratio below 0.2, numeric features, a separately constructed time-valid training set, and enough minority observations for the neighbor setting. It fits scaling and SMOTE inside an imbalanced-learn pipeline. The later validation set is only scored. The sampling ratio and neighbor count are illustrative candidate settings, not recommended defaults or measured optimal values. They should be selected within a temporal model-selection procedure.

Ordinary SMOTE interpolates within a feature geometry. That geometry may be a poor description of business plausibility. A synthetic amount combined with interpolated behavioral counts could describe no possible transaction, and categorical identities should not be treated as arbitrary continuous coordinates. I would inspect generated samples, consider suitable categorical methods when appropriate, and compare against weighting before accepting the extra modeling assumptions.

1from imblearn.over_sampling import SMOTE
2from imblearn.pipeline import Pipeline
3from sklearn.linear_model import LogisticRegression
4from sklearn.preprocessing import StandardScaler
5 
6# X_train/y_train must already obey event and label-availability cutoffs.
7model = Pipeline([
8    ("scale", StandardScaler()),
9    ("sample", SMOTE(sampling_strategy=0.2, k_neighbors=3, random_state=17)),
10    ("classify", LogisticRegression(max_iter=1000)),
11])
12model.fit(X_train, y_train)
13validation_scores = model.predict_proba(X_validation)[:, 1]
14# Keep validation rows at their observed prevalence. Do not resample them.

Compare interventions on the same evidence

I would hold the eligible records, feature definitions, candidate model family, tuning budget, and evaluation windows constant for the initial comparison. The baseline uses the original training distribution. A second arm changes class weights. A third changes the training sample through SMOTE. A fourth, only if justified, tests a different model family. Changing all these ingredients together may produce a stronger model, but it does not reveal which intervention helped.

Average precision or a precision-recall curve can describe ranking behavior for a rare positive class, but they do not settle the operating decision. Report precision at the review capacity, recall at that capacity, alert counts, and the distribution of alert value or harm when reliable labels support those calculations. The queue has a capacity measured in reviews per period; a threshold has no stable workload meaning if the traffic volume or score distribution shifts.

Ties need a declared rule. If many transactions share the threshold score, selecting the first rows in a file can make the experiment depend on ingestion order. Use a deterministic tie-breaker or evaluate the full tied set and report the ambiguity. Also retain a simple policy baseline where one is available. A complex classifier should demonstrate value over the actual decision process it proposes to replace.

Treat probabilities and priorities as different outputs

A score can rank cases usefully without being a trustworthy probability. Resampling changes the distribution the learner sees, and the resulting probability estimates require evaluation on representative held-out data. Scikit-learn’s calibration documentation separates probability calibration from classification performance. I would inspect reliability by score range and relevant operational slices, alongside proper scoring rules, before attaching a probability interpretation to the output.

Calibration data must be independent of fitting the base model and must match the intended evaluation design. Threshold selection also consumes information. If the same small holdout is used repeatedly to choose the model, fit a calibrator, choose the threshold, and celebrate the final result, it is no longer a convincing final test. I would reserve a later untouched period for the final comparison and preserve the selection history.

For the review queue, a capacity-based policy may be more interpretable than an asserted universal probability threshold. The policy can rank eligible cases, apply documented exclusions, and allocate a fixed review budget. That still requires monitoring: the mix of reviewed cases can change, and feedback from the review policy can influence which labels are observed. Missing labels should not automatically be treated as evidence that an unreviewed transaction was legitimate.

Investigate failures before accepting a higher score

After the aggregate comparison, I would examine false positives and missed positives by time, entity novelty, data completeness, and relevant transaction categories. Any subgroup analysis needs sufficient support and a clear interpretation; small slices can produce unstable estimates. Use uncertainty estimates that respect the dependence structure where possible. Repeated records from one account are not the same as independent observations from many accounts.

A failure report should distinguish a bad decision from an unavailable feature, a delayed label, a duplicate case, and a disagreement in the labeling process. Those causes imply different remedies. More synthetic training data will not fix a feature that arrives after inference. A better classifier will not resolve inconsistent investigative definitions. This is where the experiment becomes a systems investigation rather than a competition between model names.

The release argument should state the chosen operating policy, the untouched test period, the limitations of the labels, and the evidence that would trigger reconsideration. It should also preserve a negative result if SMOTE fails to help. I am interested in a model that earns its place in a decision process. A higher score is useful only when the evaluation makes clear what improved, for whom, and under which constraints.