Explainable Fraud Detection and Investigation Platform

ML Model Selection, Tuning & Evaluation

Objective

This notebook selects, tunes, and evaluates supervised machine learning models for fraud detection using the saved outputs of 02_data_preparation.ipynb.

The comparison follows the MIA 5100 themes of parametric and non-parametric learning, decision trees, ensemble learning, chronological validation, hyperparameter tuning, and model evaluation. Because fraud is rare, average precision (area under the precision-recall curve) is the primary model-selection metric. ROC-AUC, recall, precision, F-scores, balanced accuracy, confusion matrices, and alert volume provide complementary evidence.

Deliverables

• Reproducible baselines and candidate models
• Validation comparison at the default 0.50 threshold
• Precision-recall and ROC curves
• Validation-only decision-threshold selection
• Expanding-window cross-validated hyperparameter tuning
• One final, untouched test-set evaluation
• Persisted model, threshold, metrics, and transaction-level predictions

Validation set: used to compare models and choose the decision threshold—the risk score above which a transaction is labelled fraudulent. Test set: used only at the end to estimate how the finalized approach performs on unseen data. “Decisions are frozen” which means that before examining test results, the following have already been chosen. The winning model Its hyperparameters The decision threshold The evaluation procedure

1. Import Libraries and Configure the Experiment

from pathlib import Path
from time import perf_counter
import hashlib
import json
import platform
import subprocess
import warnings

import joblib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

from sklearn import __version__ as sklearn_version
from sklearn.base import clone
from sklearn.calibration import CalibrationDisplay
from sklearn.dummy import DummyClassifier
from sklearn.ensemble import HistGradientBoostingClassifier, RandomForestClassifier
from sklearn.exceptions import ConvergenceWarning
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    average_precision_score,
    balanced_accuracy_score,
    classification_report,
    confusion_matrix,
    f1_score,
    fbeta_score,
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
    roc_curve,
)
from sklearn.model_selection import RandomizedSearchCV, TimeSeriesSplit
from sklearn.neural_network import MLPClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.utils.class_weight import compute_sample_weight

warnings.filterwarnings("ignore", category=ConvergenceWarning)
sns.set_theme(style="whitegrid", context="notebook")
pd.set_option("display.max_columns", 100)

RANDOM_STATE = 42
N_JOBS = -1

# Set True for a faster end-to-end rehearsal. Set False for final reported results.
QUICK_RUN = False
QUICK_TRAIN_ROWS = 120_000

# Optional, computationally heavier experiments.
RUN_MLP = True
RUN_TUNING = True
RUN_XGBOOST = True

print(f"Python: {platform.python_version()}")
print(f"pandas: {pd.__version__}")
print(f"scikit-learn: {sklearn_version}")
print(f"Quick run: {QUICK_RUN}")

Python: 3.14.6
pandas: 3.0.3
scikit-learn: 1.9.0
Quick run: False

Design choices

• Dummy classifier: establishes the no-skill reference.
• Logistic regression: interpretable parametric baseline.
• Decision tree: non-parametric baseline with nonlinear interactions.
• Random forest: bagging ensemble that reduces tree variance.
• Histogram gradient boosting: efficient boosting model available in scikit-learn.
• XGBoost: external boosting benchmark.
• Multi-layer perceptron: neural-network comparison

Class weighting is learned from the training labels only. No synthetic observations are created, which avoids distorting already encoded transaction patterns. Resampling could be examined later as a controlled ablation.

The processed splits are chronological. Model selection and threshold selection use validation data from a later period than training, and the final test period remains later still.

2. Define Project Paths

def find_project_root(start=None):
    '''Find the repository from the notebook directory or current directory.'''
    candidates = [Path(start or Path.cwd()).resolve(), Path.cwd().resolve()]
    for candidate in candidates:
        for parent in [candidate, *candidate.parents]:
            if (parent / "data" / "processed").exists() and (parent / "notebooks").exists():
                return parent
    raise FileNotFoundError(
        "Could not locate the project root. Run this notebook from the repository "
        "or update PROJECT_ROOT manually."
    )


PROJECT_ROOT = find_project_root()
PROCESSED_DATA_PATH = PROJECT_ROOT / "data" / "processed"
MODEL_PATH = PROJECT_ROOT / "models" / "trained"
RESULTS_PATH = PROJECT_ROOT / "results" / "model_comparison"

MODEL_PATH.mkdir(parents=True, exist_ok=True)
RESULTS_PATH.mkdir(parents=True, exist_ok=True)

print(f"Project root: {PROJECT_ROOT}")
print(f"Processed data: {PROCESSED_DATA_PATH}")
print(f"Models: {MODEL_PATH}")
print(f"Results: {RESULTS_PATH}")

Project root: /Users/hshazel/Projects/explainable-fraud-investigation-platform
Processed data: /Users/hshazel/Projects/explainable-fraud-investigation-platform/data/processed
Models: /Users/hshazel/Projects/explainable-fraud-investigation-platform/models/trained
Results: /Users/hshazel/Projects/explainable-fraud-investigation-platform/results/model_comparison

3. Load Training and Validation Data

required_files = [
    "X_train.parquet", "X_validation.parquet",
    "y_train.parquet", "y_validation.parquet",
    "id_train.parquet", "id_validation.parquet",
]
missing_files = [name for name in required_files if not (PROCESSED_DATA_PATH / name).exists()]
if missing_files:
    raise FileNotFoundError(
        "Run 02_data_preparation.ipynb first. Missing processed files: "
        + ", ".join(missing_files)
    )

X_train = pd.read_parquet(PROCESSED_DATA_PATH / "X_train.parquet")
X_validation = pd.read_parquet(PROCESSED_DATA_PATH / "X_validation.parquet")
y_train = pd.read_parquet(PROCESSED_DATA_PATH / "y_train.parquet")["isFraud"].astype("int8")
y_validation = pd.read_parquet(PROCESSED_DATA_PATH / "y_validation.parquet")["isFraud"].astype("int8")
id_train = pd.read_parquet(PROCESSED_DATA_PATH / "id_train.parquet")["TransactionID"]
id_validation = pd.read_parquet(PROCESSED_DATA_PATH / "id_validation.parquet")["TransactionID"]

print(f"Training:   X={X_train.shape}, fraud rate={y_train.mean():.3%}")
print(f"Validation: X={X_validation.shape}, fraud rate={y_validation.mean():.3%}")

Training:   X=(413378, 359), fraud rate=3.517%
Validation: X=(88581, 359), fraud rate=3.434%

3.1 Data-contract integrity checks

assert len(X_train) == len(y_train) == len(id_train)
assert len(X_validation) == len(y_validation) == len(id_validation)
assert X_train.columns.equals(X_validation.columns)
assert X_train.columns.is_unique
assert set(y_train.unique()).issubset({0, 1})
assert set(y_validation.unique()).issubset({0, 1})
assert not id_train.isin(set(id_validation)).any()
assert not X_train.isna().to_numpy().any()
assert not X_validation.isna().to_numpy().any()
assert np.isfinite(X_train.to_numpy(dtype=np.float32)).all()
assert np.isfinite(X_validation.to_numpy(dtype=np.float32)).all()

data_contract = pd.DataFrame({
    "Split": ["Training", "Validation"],
    "Rows": [len(X_train), len(X_validation)],
    "Features": [X_train.shape[1], X_validation.shape[1]],
    "Fraud cases": [int(y_train.sum()), int(y_validation.sum())],
    "Fraud rate": [y_train.mean(), y_validation.mean()],
})
display(data_contract.style.format({"Fraud rate": "{:.3%}"}))
print("All integrity checks passed.")

	Split	Rows	Features	Fraud cases	Fraud rate
0	Training	413378	359	14538	3.517%
1	Validation	88581	359	3042	3.434%

All integrity checks passed.

3.2 Optional chronological quick-run sample

def chronological_subsample(X, y, ids, n_rows):
    """Keep the most recent training rows while preserving time order."""
    if n_rows >= len(X):
        return X, y, ids
    start = len(X) - n_rows
    return (
        X.iloc[start:].reset_index(drop=True),
        y.iloc[start:].reset_index(drop=True),
        ids.iloc[start:].reset_index(drop=True),
    )


if QUICK_RUN:
    X_fit, y_fit, id_fit = chronological_subsample(
        X_train, y_train, id_train, QUICK_TRAIN_ROWS
    )
else:
    X_fit, y_fit, id_fit = X_train, y_train, id_train

print(f"Fitting rows: {len(X_fit):,}")
print(f"Fitting fraud rate: {y_fit.mean():.3%}")

Fitting rows: 413,378
Fitting fraud rate: 3.517%

4. Define Evaluation Functions

def positive_class_score(model, X):
    """Return an uncalibrated ranking score; larger values indicate greater fraud risk."""
    if hasattr(model, "predict_proba"):
        return model.predict_proba(X)[:, 1]
    if hasattr(model, "decision_function"):
        scores = model.decision_function(X)
        return 1 / (1 + np.exp(-np.clip(scores, -500, 500)))
    raise TypeError(f"{type(model).__name__} does not expose continuous scores.")


def metric_row(y_true, risk_score, threshold=0.50):
    prediction = (risk_score >= threshold).astype("int8")
    tn, fp, fn, tp = confusion_matrix(y_true, prediction, labels=[0, 1]).ravel()
    return {
        "Threshold": threshold,
        "ROC-AUC": roc_auc_score(y_true, risk_score),
        "Average precision": average_precision_score(y_true, risk_score),
        "Balanced accuracy": balanced_accuracy_score(y_true, prediction),
        "Precision": precision_score(y_true, prediction, zero_division=0),
        "Recall": recall_score(y_true, prediction, zero_division=0),
        "F1": f1_score(y_true, prediction, zero_division=0),
        "F2": fbeta_score(y_true, prediction, beta=2, zero_division=0),
        "Alerts": int(prediction.sum()),
        "Alert rate": float(prediction.mean()),
        "TN": int(tn), "FP": int(fp), "FN": int(fn), "TP": int(tp),
    }


def threshold_table(y_true, risk_score):
    precision, recall, thresholds = precision_recall_curve(y_true, risk_score)
    table = pd.DataFrame({
        "Threshold": thresholds,
        "Precision": precision[:-1],
        "Recall": recall[:-1],
    })
    table["F1"] = (
        2 * table["Precision"] * table["Recall"] /
        (table["Precision"] + table["Recall"] + 1e-12)
    )
    table["F2"] = (
        5 * table["Precision"] * table["Recall"] /
        (4 * table["Precision"] + table["Recall"] + 1e-12)
    )
    sorted_scores = np.sort(np.asarray(risk_score))
    alert_counts = len(sorted_scores) - np.searchsorted(
        sorted_scores, thresholds, side="left"
    )
    table["Predicted alert rate"] = alert_counts / len(sorted_scores)
    return table


def plot_confusion(y_true, risk_score, threshold, title):
    prediction = (risk_score >= threshold).astype("int8")
    matrix = confusion_matrix(y_true, prediction, labels=[0, 1])
    fig, ax = plt.subplots(figsize=(5.2, 4.3))
    sns.heatmap(matrix, annot=True, fmt=",d", cmap="Blues", cbar=False, ax=ax)
    ax.set(xlabel="Predicted class", ylabel="Actual class", title=title)
    ax.set_xticklabels(["Legitimate", "Fraud"])
    ax.set_yticklabels(["Legitimate", "Fraud"], rotation=0)
    plt.tight_layout()
    return ax

Metric interpretation

• Average precision (primary): summarizes the precision-recall trade-off and is informative under severe class imbalance.
• ROC-AUC: measures ranking discrimination but can look optimistic when negatives dominate.
• Recall: fraction of fraud cases detected; missed fraud appears as false negatives.
• Precision: fraction of alerts that are truly fraudulent; low precision increases investigator workload.
• F2: weights recall more than precision and is used here for threshold selection.
• Balanced accuracy: averages fraud and legitimate-class recall.

Accuracy is intentionally omitted from model ranking because predicting every transaction as legitimate would appear strong on this imbalanced dataset while detecting no fraud.

5. Define Candidate Models

models = {
    "Dummy (prior)": DummyClassifier(strategy="prior", random_state=RANDOM_STATE),
    "Logistic regression": LogisticRegression(
        class_weight="balanced", solver="saga", l1_ratio=0.0,
        C=1.0, max_iter=300, tol=1e-3, random_state=RANDOM_STATE,
    ),
    "Decision tree": DecisionTreeClassifier(
        class_weight="balanced", max_depth=12, min_samples_leaf=50,
        random_state=RANDOM_STATE,
    ),
    "Random forest": RandomForestClassifier(
        n_estimators=300 if not QUICK_RUN else 100,
        class_weight="balanced_subsample", max_depth=18,
        min_samples_leaf=5, max_features="sqrt", n_jobs=N_JOBS,
        random_state=RANDOM_STATE,
    ),
    "Histogram gradient boosting": HistGradientBoostingClassifier(
        learning_rate=0.08, max_iter=250 if not QUICK_RUN else 100,
        max_leaf_nodes=31, min_samples_leaf=30, l2_regularization=1.0,
        early_stopping=True, validation_fraction=0.10,
        random_state=RANDOM_STATE,
    ),
}

if RUN_MLP:
    models["Multi-layer perceptron"] = MLPClassifier(
        hidden_layer_sizes=(64, 32), activation="relu", alpha=1e-4,
        batch_size=1024, learning_rate_init=1e-3,
        max_iter=60, early_stopping=True, validation_fraction=0.10,
        n_iter_no_change=6, random_state=RANDOM_STATE,
    )

if RUN_XGBOOST:
    try:
        from xgboost import XGBClassifier
        negative, positive = np.bincount(y_fit)
        models["XGBoost"] = XGBClassifier(
            n_estimators=500 if not QUICK_RUN else 150,
            max_depth=6, learning_rate=0.05, subsample=0.8,
            colsample_bytree=0.8, min_child_weight=5,
            reg_lambda=1.0, scale_pos_weight=negative / positive,
            objective="binary:logistic", eval_metric="aucpr",
            tree_method="hist", n_jobs=N_JOBS, random_state=RANDOM_STATE,
        )
    except Exception as error:
        print(
            "XGBoost could not be loaded; continuing with scikit-learn models.\n"
            f"Reason: {type(error).__name__}: {error}"
        )

pd.DataFrame({"Candidate model": list(models)}).style.hide(axis="index")

Candidate model
Dummy (prior)
Logistic regression
Decision tree
Random forest
Histogram gradient boosting
Multi-layer perceptron
XGBoost

6. Train Models and Compare Validation Performance

fitted_models = {}
validation_risk_scores = {}
comparison_rows = []

balanced_sample_weight = compute_sample_weight(class_weight="balanced", y=y_fit)

for name, estimator in models.items():
    print(f"Training {name} ...")
    model = clone(estimator)
    start = perf_counter()

    # HistGradientBoosting accepts sample_weight; the MLP remains an explicitly
    # unweighted course comparison because it lacks class_weight support.
    if name == "Histogram gradient boosting":
        model.fit(X_fit, y_fit, sample_weight=balanced_sample_weight)
    else:
        model.fit(X_fit, y_fit)

    seconds = perf_counter() - start
    risk_score = positive_class_score(model, X_validation)
    row = metric_row(y_validation, risk_score, threshold=0.50)
    row.update({"Model": name, "Fit seconds": seconds})

    fitted_models[name] = model
    validation_risk_scores[name] = risk_score
    comparison_rows.append(row)
    print(
        f"  AP={row['Average precision']:.4f} | "
        f"ROC-AUC={row['ROC-AUC']:.4f} | {seconds:.1f}s"
    )

validation_results = (
    pd.DataFrame(comparison_rows)
    .set_index("Model")
    .sort_values("Average precision", ascending=False)
)

display(
    validation_results[[
        "Average precision", "ROC-AUC", "Balanced accuracy",
        "Precision", "Recall", "F1", "F2", "Alert rate", "Fit seconds",
    ]].style.format({
        "Average precision": "{:.4f}", "ROC-AUC": "{:.4f}",
        "Balanced accuracy": "{:.4f}", "Precision": "{:.4f}",
        "Recall": "{:.4f}", "F1": "{:.4f}", "F2": "{:.4f}",
        "Alert rate": "{:.2%}", "Fit seconds": "{:.1f}",
    }).background_gradient(subset=["Average precision"], cmap="YlGn")
)

Training Dummy (prior) ...
  AP=0.0343 | ROC-AUC=0.5000 | 0.0s
Training Logistic regression ...
  AP=0.1704 | ROC-AUC=0.7691 | 182.8s
Training Decision tree ...
  AP=0.3766 | ROC-AUC=0.8203 | 9.4s
Training Random forest ...
  AP=0.4527 | ROC-AUC=0.8795 | 43.6s
Training Histogram gradient boosting ...
  AP=0.5336 | ROC-AUC=0.9098 | 37.5s
Training Multi-layer perceptron ...
  AP=0.2809 | ROC-AUC=0.6650 | 20.3s
Training XGBoost ...
  AP=0.5425 | ROC-AUC=0.9153 | 16.6s

	Average precision	ROC-AUC	Balanced accuracy	Precision	Recall	F1	F2	Alert rate	Fit seconds
Model
XGBoost	0.5425	0.9153	0.8047	0.2844	0.6693	0.3992	0.5268	8.08%	16.6
Histogram gradient boosting	0.5336	0.9098	0.8148	0.2191	0.7209	0.3361	0.4945	11.30%	37.5
Random forest	0.4527	0.8795	0.7236	0.4003	0.4724	0.4334	0.4560	4.05%	43.6
Decision tree	0.3766	0.8203	0.7720	0.1502	0.6811	0.2461	0.3990	15.57%	9.4
Multi-layer perceptron	0.2809	0.6650	0.6086	0.6074	0.2222	0.3254	0.2545	1.26%	20.3
Logistic regression	0.1704	0.7691	0.6649	0.0572	0.7978	0.1067	0.2222	47.93%	182.8
Dummy (prior)	0.0343	0.5000	0.5000	0.0000	0.0000	0.0000	0.0000	0.00%	0.0

6.1 Precision-recall and ROC curves

fig, axes = plt.subplots(1, 2, figsize=(14, 5.2))
prevalence = y_validation.mean()

for name, risk_score in validation_risk_scores.items():
    precision, recall, _ = precision_recall_curve(y_validation, risk_score)
    fpr, tpr, _ = roc_curve(y_validation, risk_score)
    ap = average_precision_score(y_validation, risk_score)
    auc = roc_auc_score(y_validation, risk_score)
    axes[0].plot(recall, precision, label=f"{name} (AP={ap:.3f})")
    axes[1].plot(fpr, tpr, label=f"{name} (AUC={auc:.3f})")

axes[0].axhline(prevalence, color="black", linestyle="--", label=f"No skill ({prevalence:.3f})")
axes[0].set(xlabel="Recall", ylabel="Precision", title="Validation precision-recall curves", xlim=(0, 1), ylim=(0, 1))
axes[1].plot([0, 1], [0, 1], color="black", linestyle="--", label="No skill")
axes[1].set(xlabel="False-positive rate", ylabel="True-positive rate", title="Validation ROC curves", xlim=(0, 1), ylim=(0, 1))
for ax in axes:
    ax.legend(fontsize=8, loc="best")
plt.tight_layout()
plt.show()

6.2 Select the champion by validation average precision

eligible_models = validation_results.drop(index="Dummy (prior)", errors="ignore")
champion_name = eligible_models["Average precision"].idxmax()
champion_model = fitted_models[champion_name]
champion_validation_risk_score = validation_risk_scores[champion_name]

print(f"Champion model: {champion_name}")
print(f"Validation average precision: {eligible_models.loc[champion_name, 'Average precision']:.4f}")
print(f"Validation ROC-AUC: {eligible_models.loc[champion_name, 'ROC-AUC']:.4f}")

Champion model: XGBoost
Validation average precision: 0.5425
Validation ROC-AUC: 0.9153

7. Select an Operating Threshold on Validation Data

The best ranking model is not automatically the best operational classifier at a 0.50 cutoff. The threshold is selected only on validation data. This notebook uses maximum F2 as a recall-oriented default, while also reporting workload-constrained alternatives.

For deployment, the threshold should ultimately reflect the relative cost of missed fraud and unnecessary investigations, plus the investigation team’s alert capacity.

thresholds = threshold_table(y_validation, champion_validation_risk_score)

f2_threshold = float(thresholds.loc[thresholds["F2"].idxmax(), "Threshold"])

# Useful operating points: highest recall while meeting a minimum precision.
precision_targets = [0.20, 0.40, 0.60]
operating_points = []
for target in precision_targets:
    feasible = thresholds[thresholds["Precision"] >= target]
    if not feasible.empty:
        point = feasible.loc[feasible["Recall"].idxmax()].copy()
        point["Rule"] = f"Precision >= {target:.0%}"
        operating_points.append(point)

f2_point = thresholds.loc[thresholds["F2"].idxmax()].copy()
f2_point["Rule"] = "Maximum F2 (selected)"
operating_points.append(f2_point)

operating_points = pd.DataFrame(operating_points).set_index("Rule")
display(operating_points[["Threshold", "Precision", "Recall", "F1", "F2", "Predicted alert rate"]].style.format({
    "Threshold": "{:.4f}", "Precision": "{:.3f}", "Recall": "{:.3f}",
    "F1": "{:.3f}", "F2": "{:.3f}", "Predicted alert rate": "{:.2%}",
}))

print(f"Frozen validation-selected threshold: {f2_threshold:.6f}")

	Threshold	Precision	Recall	F1	F2	Predicted alert rate
Rule
Precision >= 20%	0.3775	0.200	0.759	0.317	0.487	13.03%
Precision >= 40%	0.6225	0.400	0.582	0.474	0.533	5.00%
Precision >= 60%	0.7649	0.600	0.468	0.526	0.489	2.68%
Maximum F2 (selected)	0.5794	0.355	0.613	0.450	0.535	5.93%

Frozen validation-selected threshold: 0.579397

7.1 Threshold trade-off and confusion matrix

fig, ax = plt.subplots(figsize=(10, 5))
plot_table = thresholds.iloc[::max(1, len(thresholds) // 3000)]
ax.plot(plot_table["Threshold"], plot_table["Precision"], label="Precision")
ax.plot(plot_table["Threshold"], plot_table["Recall"], label="Recall")
ax.plot(plot_table["Threshold"], plot_table["F2"], label="F2")
ax.axvline(f2_threshold, color="black", linestyle="--", label=f"Selected = {f2_threshold:.3f}")
ax.set(xlabel="Decision threshold", ylabel="Score", title=f"{champion_name}: validation threshold trade-off", ylim=(0, 1))
ax.legend()
plt.tight_layout()
plt.show()

plot_confusion(
    y_validation, champion_validation_risk_score, f2_threshold,
    f"Validation confusion matrix: {champion_name}\nthreshold={f2_threshold:.4f}",
)
plt.show()

print(classification_report(
    y_validation,
    (champion_validation_risk_score >= f2_threshold).astype("int8"),
    target_names=["Legitimate", "Fraud"],
    digits=4,
))

              precision    recall  f1-score   support

  Legitimate     0.9859    0.9604    0.9730     85539
       Fraud     0.3550    0.6134    0.4497      3042

    accuracy                         0.9484     88581
   macro avg     0.6704    0.7869    0.7113     88581
weighted avg     0.9642    0.9484    0.9550     88581

7.2 Risk-score calibration diagnostic

Class weighting improves minority-class learning but generally changes score calibration. The chart below is diagnostic: until an explicit calibration experiment is performed, outputs are described as fraud risk scores rather than probabilities.

fig, ax = plt.subplots(figsize=(6, 5))
CalibrationDisplay.from_predictions(
    y_validation, champion_validation_risk_score,
    n_bins=10, strategy="quantile", name=champion_name, ax=ax,
)
ax.set_xlabel("Mean fraud risk score")
ax.set_title("Validation risk-score calibration diagnostic")
plt.tight_layout()
plt.show()
print(
    "The weighted model score is used for ranking and thresholding, not as a calibrated "
    "probability. A future calibration experiment must use training-only data."
)

The weighted model score is used for ranking and thresholding, not as a calibrated probability. A future calibration experiment must use training-only data.

8. Expanding-Window Hyperparameter Tuning

This section is computationally heavier and is enabled for the final experiment. It uses three expanding-window time-series folds on the training period and optimizes average precision. The later validation period remains external to the search, and the test period remains untouched.

tuning_metadata = {
    "requested": bool(RUN_TUNING),
    "performed": False,
    "cv": "3-fold expanding-window TimeSeriesSplit",
    "scoring": "average_precision",
}

if RUN_TUNING:
    tuning_estimators = {
        "Logistic regression": (
            LogisticRegression(
                class_weight="balanced", solver="saga", max_iter=300,
                tol=1e-3, random_state=RANDOM_STATE,
            ),
            {"C": np.logspace(-2, 1, 8), "l1_ratio": [0.0, 1.0]},
        ),
        "Decision tree": (
            DecisionTreeClassifier(class_weight="balanced", random_state=RANDOM_STATE),
            {
                "max_depth": [5, 8, 12, 16, None],
                "min_samples_leaf": [10, 25, 50, 100, 250],
                "criterion": ["gini", "entropy"],
            },
        ),
        "Random forest": (
            RandomForestClassifier(
                class_weight="balanced_subsample", n_jobs=N_JOBS,
                random_state=RANDOM_STATE,
            ),
            {
                "n_estimators": [200, 350, 500],
                "max_depth": [10, 16, 24, None],
                "min_samples_leaf": [2, 5, 10, 25],
                "max_features": ["sqrt", 0.3, 0.5],
            },
        ),
    }

    if "XGBoost" in fitted_models:
        negative, positive = np.bincount(y_fit)
        tuning_estimators["XGBoost"] = (
            XGBClassifier(
                objective="binary:logistic", eval_metric="aucpr",
                scale_pos_weight=negative / positive, tree_method="hist",
                n_jobs=N_JOBS, random_state=RANDOM_STATE,
            ),
            {
                "n_estimators": [250, 400, 550],
                "max_depth": [4, 6, 8],
                "learning_rate": [0.03, 0.05, 0.08],
                "min_child_weight": [3, 5, 10],
                "subsample": [0.70, 0.85, 1.0],
                "colsample_bytree": [0.70, 0.85, 1.0],
                "reg_lambda": [1.0, 5.0, 10.0],
            },
        )

    if champion_name not in tuning_estimators:
        print(f"No tuning grid specified for {champion_name}; keep the fitted champion.")
    else:
        estimator, parameter_distributions = tuning_estimators[champion_name]
        cv = TimeSeriesSplit(n_splits=3)
        search = RandomizedSearchCV(
            estimator=estimator,
            param_distributions=parameter_distributions,
            n_iter=8 if not QUICK_RUN else 3,
            scoring="average_precision",
            cv=cv,
            n_jobs=N_JOBS,
            verbose=2,
            random_state=RANDOM_STATE,
            refit=True,
            return_train_score=True,
        )
        search.fit(X_fit, y_fit)
        tuned_risk_score = positive_class_score(search.best_estimator_, X_validation)
        tuned_metrics = metric_row(y_validation, tuned_risk_score, threshold=0.50)
        tuning_metadata.update({
            "performed": True,
            "best_cv_average_precision": float(search.best_score_),
            "best_parameters": search.best_params_,
        })
        print(f"Best time-aware CV AP: {search.best_score_:.4f}")
        print("Best parameters:", search.best_params_)
        display(pd.DataFrame([tuned_metrics]).style.format(precision=4))

        current_ap = average_precision_score(
            y_validation, champion_validation_risk_score
        )
        if tuned_metrics["Average precision"] > current_ap:
            original_name = champion_name
            champion_model = search.best_estimator_
            champion_validation_risk_score = tuned_risk_score
            champion_name = f"{original_name} (tuned)"
            thresholds = threshold_table(y_validation, champion_validation_risk_score)
            f2_threshold = float(thresholds.loc[thresholds["F2"].idxmax(), "Threshold"])
            tuned_row = {**tuned_metrics, "Fit seconds": np.nan}
            validation_results.loc[champion_name] = tuned_row
            validation_results = validation_results.sort_values(
                "Average precision", ascending=False
            )
            tuning_metadata["selected_after_external_validation"] = True
            print("The tuned model becomes the champion; threshold recomputed on validation.")
        else:
            tuning_metadata["selected_after_external_validation"] = False
            print("Tuning did not improve validation AP; retaining the original champion.")
else:
    print("Tuning skipped.")

Fitting 3 folds for each of 8 candidates, totalling 24 fits
[CV] END colsample_bytree=0.85, learning_rate=0.03, max_depth=6, min_child_weight=5, n_estimators=550, reg_lambda=5.0, subsample=1.0; total time= 2.3min
[CV] END colsample_bytree=1.0, learning_rate=0.08, max_depth=8, min_child_weight=10, n_estimators=400, reg_lambda=1.0, subsample=0.7; total time= 2.0min
[CV] END colsample_bytree=0.85, learning_rate=0.03, max_depth=6, min_child_weight=5, n_estimators=550, reg_lambda=5.0, subsample=1.0; total time= 1.4min
[CV] END colsample_bytree=1.0, learning_rate=0.03, max_depth=8, min_child_weight=3, n_estimators=550, reg_lambda=1.0, subsample=0.7; total time= 2.9min
[CV] END colsample_bytree=0.85, learning_rate=0.03, max_depth=6, min_child_weight=5, n_estimators=550, reg_lambda=5.0, subsample=1.0; total time=  38.1s
[CV] END colsample_bytree=0.85, learning_rate=0.05, max_depth=6, min_child_weight=5, n_estimators=400, reg_lambda=10.0, subsample=0.7; total time= 1.3min
[CV] END colsample_bytree=1.0, learning_rate=0.08, max_depth=8, min_child_weight=10, n_estimators=400, reg_lambda=1.0, subsample=0.7; total time= 1.6min
[CV] END colsample_bytree=0.85, learning_rate=0.08, max_depth=4, min_child_weight=3, n_estimators=550, reg_lambda=5.0, subsample=1.0; total time=  58.9s
Best time-aware CV AP: 0.5448
Best parameters: {'subsample': 0.7, 'reg_lambda': 1.0, 'n_estimators': 400, 'min_child_weight': 10, 'max_depth': 8, 'learning_rate': 0.08, 'colsample_bytree': 1.0}

	Threshold	ROC-AUC	Average precision	Balanced accuracy	Precision	Recall	F1	F2	Alerts	Alert rate	TN	FP	FN	TP
0	0.5000	0.9240	0.5904	0.7606	0.6137	0.5332	0.5706	0.5476	2643	0.0298	84518	1021	1420	1622

The tuned model becomes the champion; threshold recomputed on validation.

9. Freeze the Decision and Evaluate the Test Set Once

At this point the model family, fitted model, and decision threshold are frozen. The test set is now loaded for the first and only time. It must not be used to revise the model or threshold.

X_test = pd.read_parquet(PROCESSED_DATA_PATH / "X_test.parquet")
y_test = pd.read_parquet(PROCESSED_DATA_PATH / "y_test.parquet")["isFraud"].astype("int8")
id_test = pd.read_parquet(PROCESSED_DATA_PATH / "id_test.parquet")["TransactionID"]

assert X_test.columns.equals(X_train.columns)
assert len(X_test) == len(y_test) == len(id_test)
assert not id_test.isin(set(id_train)).any()
assert not id_test.isin(set(id_validation)).any()

test_risk_score = positive_class_score(champion_model, X_test)
test_metrics = metric_row(y_test, test_risk_score, threshold=f2_threshold)

final_comparison = pd.DataFrame([
    {"Split": "Validation", **metric_row(
        y_validation, champion_validation_risk_score, threshold=f2_threshold
    )},
    {"Split": "Test", **test_metrics},
]).set_index("Split")

display(final_comparison[[
    "Threshold", "Average precision", "ROC-AUC", "Balanced accuracy",
    "Precision", "Recall", "F1", "F2", "Alert rate", "TP", "FP", "FN", "TN",
]].style.format({
    "Threshold": "{:.4f}", "Average precision": "{:.4f}", "ROC-AUC": "{:.4f}",
    "Balanced accuracy": "{:.4f}", "Precision": "{:.4f}", "Recall": "{:.4f}",
    "F1": "{:.4f}", "F2": "{:.4f}", "Alert rate": "{:.2%}",
}))

	Threshold	Average precision	ROC-AUC	Balanced accuracy	Precision	Recall	F1	F2	Alert rate	TP	FP	FN	TN
Split
Validation	0.2757	0.5904	0.9240	0.8160	0.3856	0.6700	0.4895	0.5839	5.97%	2038	3247	1004	82292
Test	0.2757	0.5213	0.8996	0.7850	0.3258	0.6160	0.4262	0.5228	6.58%	1899	3930	1184	81568

9.1 Final test confusion matrix

plot_confusion(
    y_test, test_risk_score, f2_threshold,
    f"Final test confusion matrix: {champion_name}\nthreshold={f2_threshold:.4f}",
)
plt.show()

print(classification_report(
    y_test,
    (test_risk_score >= f2_threshold).astype("int8"),
    target_names=["Legitimate", "Fraud"],
    digits=4,
))

              precision    recall  f1-score   support

  Legitimate     0.9857    0.9540    0.9696     85498
       Fraud     0.3258    0.6160    0.4262      3083

    accuracy                         0.9423     88581
   macro avg     0.6557    0.7850    0.6979     88581
weighted avg     0.9627    0.9423    0.9507     88581

10. Persist the Champion Model and Evaluation Outputs

safe_model_name = (
    champion_name.lower().replace(" ", "_").replace("(", "").replace(")", "")
)
model_file = MODEL_PATH / f"{safe_model_name}.joblib"
metadata_file = MODEL_PATH / f"{safe_model_name}_metadata.json"

joblib.dump(champion_model, model_file)

preprocessing_metadata_file = (
    PROJECT_ROOT / "models" / "preprocessing" / "preprocessing_metadata.json"
)
preprocessing_metadata = (
    json.loads(preprocessing_metadata_file.read_text(encoding="utf-8"))
    if preprocessing_metadata_file.exists() else {}
)

git_commit = subprocess.run(
    ["git", "rev-parse", "HEAD"], cwd=PROJECT_ROOT,
    capture_output=True, text=True, check=False,
).stdout.strip() or None
git_dirty = bool(subprocess.run(
    ["git", "status", "--porcelain"], cwd=PROJECT_ROOT,
    capture_output=True, text=True, check=False,
).stdout.strip())
feature_schema_sha256 = hashlib.sha256(
    "\n".join(X_fit.columns).encode("utf-8")
).hexdigest()

try:
    import xgboost as xgb_package
    xgboost_version = xgb_package.__version__
except ImportError:
    xgboost_version = None

metadata = {
    "model_name": champion_name,
    "decision_threshold": f2_threshold,
    "threshold_rule": "maximum validation F2",
    "primary_selection_metric": "validation average precision",
    "score_column": "fraud_risk_score",
    "score_semantics": "uncalibrated ranking score; not a fraud probability",
    "random_state": RANDOM_STATE,
    "quick_run": QUICK_RUN,
    "training_rows_used": len(X_fit),
    "feature_count": X_fit.shape[1],
    "feature_schema_sha256": feature_schema_sha256,
    "data_split": preprocessing_metadata,
    "tuning": tuning_metadata,
    "environment": {
        "python": platform.python_version(),
        "pandas": pd.__version__,
        "scikit_learn": sklearn_version,
        "xgboost": xgboost_version,
    },
    "git_commit": git_commit,
    "git_worktree_dirty_at_training": git_dirty,
    "test_metrics": {
        key: float(value) if isinstance(value, (float, np.floating)) else int(value)
        for key, value in test_metrics.items()
    },
}
metadata_file.write_text(json.dumps(metadata, indent=2), encoding="utf-8")

manifest = {
    "model_name": champion_name,
    "model_file": model_file.name,
    "metadata_file": metadata_file.name,
}
(MODEL_PATH / "champion_manifest.json").write_text(
    json.dumps(manifest, indent=2), encoding="utf-8"
)

validation_predictions = pd.DataFrame({
    "TransactionID": id_validation.to_numpy(),
    "isFraud": y_validation.to_numpy(),
    "fraud_risk_score": champion_validation_risk_score,
    "predicted_fraud": (champion_validation_risk_score >= f2_threshold).astype("int8"),
    "split": "validation",
})
test_predictions = pd.DataFrame({
    "TransactionID": id_test.to_numpy(),
    "isFraud": y_test.to_numpy(),
    "fraud_risk_score": test_risk_score,
    "predicted_fraud": (test_risk_score >= f2_threshold).astype("int8"),
    "split": "test",
})
predictions = pd.concat([validation_predictions, test_predictions], ignore_index=True)
predictions.to_parquet(RESULTS_PATH / "champion_predictions.parquet", index=False)

validation_results.reset_index().to_csv(
    RESULTS_PATH / "validation_model_comparison.csv", index=False
)
final_comparison.reset_index().to_csv(
    RESULTS_PATH / "champion_validation_test_metrics.csv", index=False
)

print(f"Saved model: {model_file}")
print(f"Saved metadata: {metadata_file}")
print(f"Saved champion manifest: {MODEL_PATH / 'champion_manifest.json'}")
print(f"Saved predictions and metrics: {RESULTS_PATH}")

Saved model: /Users/hshazel/Projects/explainable-fraud-investigation-platform/models/trained/xgboost_tuned.joblib
Saved metadata: /Users/hshazel/Projects/explainable-fraud-investigation-platform/models/trained/xgboost_tuned_metadata.json
Saved champion manifest: /Users/hshazel/Projects/explainable-fraud-investigation-platform/models/trained/champion_manifest.json
Saved predictions and metrics: /Users/hshazel/Projects/explainable-fraud-investigation-platform/results/model_comparison

11. ML Model Selection, Tuning & Evaluation Summary and Next Steps

The executed experiment compared a no-skill baseline, linear and tree baselines, ensemble methods, a neural network, and XGBoost on chronological data. Untuned XGBoost ranked first initially, and the expanding-window search improved the external validation average precision from 0.5425 to 0.5904. The tuned model’s best three-fold time-aware cross-validation average precision was 0.5448.

Final executed results

• Champion: XGBoost (tuned)
• Validation-selected risk-score threshold: 0.2757 using maximum F2
• Later-period test average precision: 0.5213
• Later-period test ROC-AUC: 0.8996
• Test precision: 0.3258
• Test recall: 0.6160
• Test F1 / F2: 0.4262 / 0.5228
• Test alerts: 5,829 of 88,581 transactions (6.58%)
• Test confusion counts: 1,899 TP, 3,930 FP, 1,184 FN, and 81,568 TN

The decline from validation AP 0.5904 to test AP 0.5213 indicates a meaningful later-period generalization gap. The chronological result is therefore more conservative and operationally credible than a random-split estimate. At the selected threshold, recall is prioritized, but approximately two thirds of alerts are false positives; deployment would require an explicit investigation-capacity and error-cost policy. Model outputs remain uncalibrated risk scores rather than fraud probabilities.

Reporting checklist

• Report the class prevalence and explain why accuracy is misleading.
• Compare average precision and ROC-AUC, not either metric in isolation.
• Discuss the 1,184 missed fraud cases and 3,930 unnecessary alerts in business terms.
• Report both the default 0.50 threshold and selected 0.2757 threshold.
• State that QUICK_RUN=False, MLP and XGBoost were enabled, and time-aware tuning was completed.
• Treat the final test result as confirmatory, not another tuning signal.

XAI extension

04_post_hoc_explainability.ipynb uses the frozen champion, processed data, and saved transaction-level predictions for post-hoc global and local SHAP and LIME analysis. It does not retrain the model or change the selected threshold.