Explainable Fraud Detection and Investigation Platform

Data Wrangling, Preprocessing & Feature Engineering

Objective

This notebook focuses on preparing the IEEE-CIS Fraud Detection dataset for machine learning model development.

Building upon the insights obtained during the Exploratory Data Analysis (EDA) phase, this stage transforms the raw transaction and identity data into a clean, consistent, and model-ready dataset.

The data-wrangling, preprocessing, and feature-engineering workflow includes handling missing values, removing low-information features, encoding categorical variables, scaling numerical features where appropriate, engineering new predictive features, selecting relevant variables, and creating reproducible training, validation, and testing datasets.

The output of this notebook serves as the input for the subsequent ML Model Selection, Tuning & Evaluation phase of the Explainable Fraud Detection and Investigation Platform.

Deliverables

By the end of this notebook, the following artifacts will be produced:

• A cleaned and preprocessed dataset suitable for machine learning.
• Engineered features designed to improve predictive performance.
• Encoded and transformed numerical and categorical variables.
• Train, validation, and test datasets.
• A reproducible preprocessing pipeline that can be reused for future model training and inference.

1.Import Libraries

# ==========================================================
# 1. Import Libraries
# ==========================================================

# Standard Libraries
import hashlib
import json
import random
from pathlib import Path

# Data Manipulation
import numpy as np
import pandas as pd

# Data Visualization
import matplotlib.pyplot as plt
import seaborn as sns

# Scikit-Learn - Preprocessing
from sklearn.preprocessing import (
    RobustScaler,
    OneHotEncoder
)

# Scikit-Learn - Imputation
from sklearn.impute import SimpleImputer


# Save preprocessing objects
import joblib

# ----------------------------------------------------------
# Display Configuration
# ----------------------------------------------------------

pd.set_option("display.max_columns", None)
pd.set_option("display.max_rows", 100)
pd.set_option("display.width", 1200)
pd.set_option("display.float_format", "{:.2f}".format)

# ----------------------------------------------------------
# Plot Configuration
# ----------------------------------------------------------

sns.set_theme(style="whitegrid")
plt.rcParams["figure.figsize"] = (10, 6)

# ----------------------------------------------------------
# Reproducibility
# ----------------------------------------------------------

RANDOM_STATE = 42

np.random.seed(RANDOM_STATE)
random.seed(RANDOM_STATE)


print("Libraries imported successfully.")
print(f"Random State: {RANDOM_STATE}")

Libraries imported successfully.
Random State: 42

# Versions check
import sys
import sklearn

print(f"Python: {sys.version.split()[0]}")
print(f"Pandas: {pd.__version__}")
print(f"NumPy: {np.__version__}")
print(f"Scikit-learn: {sklearn.__version__}")

Python: 3.14.6
Pandas: 3.0.3
NumPy: 2.4.6
Scikit-learn: 1.9.0

2. Import dataset

2.1 Define Project Paths

# ==========================================================
# 2. Load the Dataset
# ==========================================================

# Define project paths

CURRENT_PATH = Path.cwd().resolve()

if CURRENT_PATH.name == "notebooks":
    PROJECT_ROOT = CURRENT_PATH.parent
elif (CURRENT_PATH / "data").exists():
    PROJECT_ROOT = CURRENT_PATH
else:
    raise FileNotFoundError(
        "Could not identify the project root. "
        "Run the notebook from the repository root or notebooks folder."
    )

RAW_DATA_PATH = PROJECT_ROOT / "data" / "raw"
PROCESSED_DATA_PATH = PROJECT_ROOT / "data" / "processed"

PROCESSED_DATA_PATH.mkdir(parents=True, exist_ok=True)

print(f"Project Root: {PROJECT_ROOT}")
print(f"Raw Data Path: {RAW_DATA_PATH}")

required_files = [
    RAW_DATA_PATH / "train_transaction.csv",
    RAW_DATA_PATH / "train_identity.csv"
]

missing_files = [
    path for path in required_files
    if not path.exists()
]

if missing_files:
    raise FileNotFoundError(
        "Missing required files:\n"
        + "\n".join(str(path) for path in missing_files)
    )

Project Root: /Users/hshazel/Projects/explainable-fraud-investigation-platform
Raw Data Path: /Users/hshazel/Projects/explainable-fraud-investigation-platform/data/raw

2.2 Load Raw Data

# Load raw input tables

train_transaction = pd.read_csv(
    RAW_DATA_PATH / "train_transaction.csv",
    #low_memory=False
)

train_identity = pd.read_csv(
    RAW_DATA_PATH / "train_identity.csv",
    #low_memory=False # For large datasets, this prevents dtype inference and speeds up loading
)

print("Datasets loaded successfully.")

Datasets loaded successfully.

2.3 Merge Datasets

# Merge transaction and identity tables
assert train_transaction["TransactionID"].is_unique
assert train_identity["TransactionID"].is_unique

rows_before_merge = len(train_transaction)

df = train_transaction.merge(
    train_identity,
    on="TransactionID",
    how="left",
    validate="one_to_one"
)

assert len(df) == rows_before_merge

print("Datasets merged successfully.")
print(f"Rows preserved after merge: {len(df):,}")

# Release memory by deleting the original DataFrames
del train_transaction
del train_identity

# Release memory by invoking garbage collection
import gc
gc.collect()

Datasets merged successfully.
Rows preserved after merge: 590,540

16

# ==========================================================
# Create Working Copy
# ==========================================================

df_processed = df.copy()

print("Working copy created successfully.")
print(f"Shape: {df_processed.shape}")

# Saving the original feature count for reference
feature_counts = {}
feature_counts["Original merged dataset"] = df_processed.shape[1]

Working copy created successfully.
Shape: (590540, 434)

2.4 Dataset Overview

print("=" * 60)
print("Dataset Overview")
print("=" * 60)

print(f"Rows: {df_processed.shape[0]:,}")
print(f"Columns: {df_processed.shape[1]:,}")

print("\nData Types")

display(df_processed.dtypes.value_counts())

============================================================
Dataset Overview
============================================================
Rows: 590,540
Columns: 434

Data Types

float64    399
str         31
int64        4
Name: count, dtype: int64

2.5 Verify Target Variable

print("=" * 60)
print("Target Variable")
print("=" * 60)

display(df_processed["isFraud"].value_counts())

fraud_rate = df_processed["isFraud"].mean() * 100

print(f"\nFraud Rate: {fraud_rate:.2f}%")

============================================================
Target Variable
============================================================

isFraud
0    569877
1     20663
Name: count, dtype: int64

Fraud Rate: 3.50%

2.6 Preview Dataset

preview_columns = [
    "TransactionID",
    "isFraud",
    "TransactionDT",
    "TransactionAmt",
    "ProductCD",
    "card1",
    "card4",
    "card6",
    "P_emaildomain",
    "DeviceType",
    "DeviceInfo"
]

display(df_processed[preview_columns].head())

	TransactionID	isFraud	TransactionDT	TransactionAmt	ProductCD	card1	card4	card6	P_emaildomain	DeviceType	DeviceInfo
0	2987000	0	86400	68.50	W	13926	discover	credit	NaN	NaN	NaN
1	2987001	0	86401	29.00	W	2755	mastercard	credit	gmail.com	NaN	NaN
2	2987002	0	86469	59.00	W	4663	visa	debit	outlook.com	NaN	NaN
3	2987003	0	86499	50.00	W	18132	mastercard	debit	yahoo.com	NaN	NaN
4	2987004	0	86506	50.00	H	4497	mastercard	credit	gmail.com	mobile	SAMSUNG SM-G892A Build/NRD90M

# Small random sample of the dataset for preview

display(
    df_processed[preview_columns].sample(
        5,
        random_state=RANDOM_STATE
    )
)

	TransactionID	isFraud	TransactionDT	TransactionAmt	ProductCD	card1	card4	card6	P_emaildomain	DeviceType	DeviceInfo
470624	3457624	0	12153579	724.00	W	7826	mastercard	debit	aol.com	NaN	NaN
565820	3552820	0	15005886	108.50	W	12544	visa	debit	yahoo.com	NaN	NaN
284083	3271083	0	6970178	47.95	W	9400	mastercard	debit	gmail.com	NaN	NaN
239689	3226689	0	5673658	100.60	C	15885	visa	debit	gmail.com	NaN	NaN
281855	3268855	0	6886780	107.95	W	15497	visa	debit	hotmail.com	NaN	NaN

2.7 Data Integrity Checks

print("=" * 60)
print("Data Integrity Checks")
print("=" * 60)

print(f"Duplicate TransactionID: {df_processed['TransactionID'].duplicated().sum()}")

print(f"Duplicate Rows: {df_processed.duplicated().sum()}")

print(f"Missing Target Values: {df_processed['isFraud'].isna().sum()}")

============================================================
Data Integrity Checks
============================================================
Duplicate TransactionID: 0
Duplicate Rows: 0
Missing Target Values: 0

2.8 Dataset Memory Usage

memory_usage = (
    df_processed.memory_usage(deep=True).sum()
    / 1024**2
)

print(f"Dataset Memory Usage: {memory_usage:.2f} MB")

Dataset Memory Usage: 1984.20 MB

Dataset Loading Summary

The transaction and identity tables from the IEEE-CIS Fraud Detection dataset were successfully loaded and merged using the TransactionID field.

Basic integrity checks confirmed that the target variable is present, the dataset contains no duplicate transaction identifiers, and the merged dataset is ready for preprocessing.

The resulting dataset will serve as the foundation for the data wrangling, preprocessing, feature engineering, and model preparation steps performed throughout this notebook.

3. Review Data Quality

Before applying preprocessing techniques, it is important to verify the overall quality and consistency of the merged dataset.

This review confirms that the dataset is structurally sound and identifies any issues that must be addressed during preprocessing, such as duplicate observations, missing values, inconsistent data types, low-information variables, and potential infinite values.

The findings from this section will guide the preprocessing strategy implemented throughout the remainder of this notebook.

3.1 Dataset Shape

# ==========================================================
# 3.1 Dataset Shape
# ==========================================================

print("=" * 60)
print("Dataset Shape")
print("=" * 60)

print(f"Rows: {df_processed.shape[0]:,}")
print(f"Columns: {df_processed.shape[1]:,}")

============================================================
Dataset Shape
============================================================
Rows: 590,540
Columns: 434

3.2 Data Types

# ==========================================================
# 3.2 Data Types
# ==========================================================

print("=" * 60)
print("Data Types")
print("=" * 60)

display(df_processed.dtypes.value_counts())

dtype_summary = (
    df_processed.dtypes
      .astype(str)
      .value_counts()
      .rename_axis("Data Type")
      .reset_index(name="Count")
)

display(dtype_summary)

============================================================
Data Types
============================================================

float64    399
str         31
int64        4
Name: count, dtype: int64

	Data Type	Count
0	float64	399
1	str	31
2	int64	4

3.3 Missing Values Overview

Missing values are summarized here but will be handled later during preprocessing.

Just confirming the amount.

# ==========================================================
# 3.3 Missing Values Overview
# ==========================================================

missing_summary = (
    df_processed.isna()
      .sum()
      .to_frame("Missing Values")
)

missing_summary["Missing (%)"] = (
    missing_summary["Missing Values"]
    / len(df_processed)
    * 100
)

missing_summary = (
    missing_summary
    .sort_values("Missing (%)", ascending=False)
)

display(missing_summary.head(15))

	Missing Values	Missing (%)
id_24	585793	99.20
id_25	585408	99.13
id_07	585385	99.13
id_08	585385	99.13
id_21	585381	99.13
id_26	585377	99.13
id_27	585371	99.12
id_23	585371	99.12
id_22	585371	99.12
dist2	552913	93.63
D7	551623	93.41
id_18	545427	92.36
D13	528588	89.51
D14	528353	89.47
D12	525823	89.04

3.4 Duplicate Check

# ==========================================================
# 3.4 Duplicate Check
# ==========================================================

print("=" * 60)
print("Duplicate Check")
print("=" * 60)

print(f"Duplicate Rows: {df_processed.duplicated().sum()}")

print(f"Duplicate TransactionID: {df_processed['TransactionID'].duplicated().sum()}")

============================================================
Duplicate Check
============================================================
Duplicate Rows: 0
Duplicate TransactionID: 0

3.5 Infinite Values

# ==========================================================
# 3.5 Infinite Values
# ==========================================================

numeric_df = df_processed.select_dtypes(include=np.number)

infinite_values = np.isinf(numeric_df).sum().sum()

print("=" * 60)
print("Infinite Values")
print("=" * 60)

print(f"Infinite Values: {infinite_values}")

============================================================
Infinite Values
============================================================
Infinite Values: 0

3.6 Constant Features

These carry no predictive information.

# ==========================================================
# 3.6 Constant Features
# ==========================================================

constant_features = [
    column
    for column in df_processed.columns
    if df_processed[column].nunique(dropna=False) == 1
]

print("=" * 60)
print("Constant Features")
print("=" * 60)

print(f"Constant Features: {len(constant_features)}")

if constant_features:
    display(pd.DataFrame({"Feature": constant_features}))

============================================================
Constant Features
============================================================
Constant Features: 0

3.7 Near-Constant Features

These features contribute very little predictive information.

# ==========================================================
# 3.7 Near-Constant Features
# ==========================================================

threshold = 0.995

near_constant_features = []

for column in df_processed.columns:

    top_frequency = (
        df_processed[column]
        .value_counts(normalize=True, dropna=False)
        .iloc[0]
    )

    if top_frequency >= threshold:
        near_constant_features.append(column)

print("=" * 60)
print("Near-Constant Features")
print("=" * 60)

print(f"Near-Constant Features: {len(near_constant_features)}")

display(
    pd.DataFrame(
        {"Feature": near_constant_features}
    ).head(20)
)

============================================================
Near-Constant Features
============================================================
Near-Constant Features: 11

	Feature
0	C3
1	V107
2	V111
3	V113
4	V117
5	V118
6	V119
7	V120
8	V121
9	V122
10	V305

Data Quality Findings

The dataset passed the initial quality review and is suitable for preprocessing.

The integrity checks confirmed that no duplicate transactions or duplicate observations are present, the target variable contains no missing values, and no infinite numerical values were detected.

A substantial number of variables contain missing values, which is expected for the IEEE-CIS Fraud Detection dataset and will be addressed during the missing value treatment stage. Several variables also exhibit little or no variability, making them potential candidates for removal during feature selection.

Overall, the dataset is structurally consistent and ready for data wrangling, preprocessing, and feature engineering.

4. Deterministic Feature Engineering

This section creates features using row-level transformations that do not learn parameters from the overall dataset. These transformations can therefore be performed before splitting without introducing data leakage.

Any transformation that learns population statistics—such as imputation values, frequency maps, category sets, scaling parameters, low-variance filters, or correlation thresholds—is postponed until after the train, validation, and test split.

# ==========================================================
# 4.1 Create Deterministic Features
# ==========================================================

# Conversion constants used to transform TransactionDT from seconds into
# elapsed day and hour components.
SECONDS_PER_DAY = 24 * 60 * 60
SECONDS_PER_HOUR = 60 * 60

# Create a separate feature frame with the same index as df_processed so
# every engineered value remains aligned with its original transaction.
engineered_df = pd.DataFrame(index=df_processed.index)

# TransactionAmt_Log: apply log(1 + amount) to preserve zero-valued amounts
# while reducing the right skew of the original transaction amount.
engineered_df["TransactionAmt_Log"] = np.log1p(
    df_processed["TransactionAmt"]
)

# TransactionAmt_Rounded: round the transaction amount to the nearest whole
# monetary unit to capture coarse amount patterns and common price points.
engineered_df["TransactionAmt_Rounded"] = (
    df_processed["TransactionAmt"].round()
)

# TransactionDay: use floor division to count complete 24-hour periods since
# the dataset's unspecified time origin. This is an elapsed day, not a date.
engineered_df["TransactionDay"] = (
    df_processed["TransactionDT"] // SECONDS_PER_DAY
).astype("int32")

# TransactionHour: take the seconds remaining within each 24-hour period and
# floor-divide by 3,600 to obtain an hour value from 0 through 23. Because the
# origin and timezone are unknown, this is an elapsed-hour component only.
engineered_df["TransactionHour"] = (
    (df_processed["TransactionDT"] % SECONDS_PER_DAY) // SECONDS_PER_HOUR
).astype("int8")

# TransactionDT has an unspecified origin, so an actual weekday/weekend
# cannot be inferred safely from the elapsed-day value.
# EmailMatch: assign 1 only when both email domains are present and exactly
# equal; otherwise assign 0.
engineered_df["EmailMatch"] = (
    df_processed["P_emaildomain"].notna()
    & df_processed["R_emaildomain"].notna()
    & (df_processed["P_emaildomain"] == df_processed["R_emaildomain"])
).astype("int8")

# P_EmailProvider: replace a missing purchaser email domain with 'unknown'
# and keep the text before the first period (for example, gmail.com -> gmail).
engineered_df["P_EmailProvider"] = (
    df_processed["P_emaildomain"].fillna("unknown").str.split(".").str[0]
)

# R_EmailProvider: apply the same provider extraction to the recipient email
# domain, again using 'unknown' when the original value is missing.
engineered_df["R_EmailProvider"] = (
    df_processed["R_emaildomain"].fillna("unknown").str.split(".").str[0]
)

# CardID: convert the anonymized card1 and card2 fields to strings, replace
# missing components with 'unknown', and join them with an underscore. This
# creates a composite anonymized card grouping, not a real card number.
engineered_df["CardID"] = (
    df_processed["card1"].astype("string").fillna("unknown")
    + "_"
    + df_processed["card2"].astype("string").fillna("unknown")
)

# IsMobile: assign 1 when DeviceType is exactly 'mobile' and 0 otherwise.
# Missing DeviceType values therefore receive 0 and are not assumed mobile.
engineered_df["IsMobile"] = (
    df_processed["DeviceType"] == "mobile"
).astype("int8")

# UnknownDevice: assign 1 when the original DeviceInfo value is missing and
# 0 when it is present. This preserves missingness before later imputation.
engineered_df["UnknownDevice"] = (
    df_processed["DeviceInfo"].isna()
).astype("int8")

# Append all ten deterministic features to the working dataset by index.
df_processed = pd.concat([df_processed, engineered_df], axis=1)

print("=" * 60)
print("Deterministic Feature Engineering")
print("=" * 60)
print(f"New features created: {engineered_df.shape[1]}")
print(f"Dataset shape: {df_processed.shape}")

feature_counts["After deterministic feature engineering"] = df_processed.shape[1]

============================================================
Deterministic Feature Engineering
============================================================
New features created: 10
Dataset shape: (590540, 444)

Deterministic Feature Engineering Findings

Ten row-level features were created without using information from other observations. Missing-device status was captured before categorical imputation so that the original missingness signal was preserved. A weekend flag is intentionally not created because TransactionDT is elapsed time from an unspecified origin rather than a known calendar timestamp.

The dataset is now ready for chronological training, validation, and test partitioning before any data-dependent preprocessing is fitted.

5. Train / Validation / Test Split

Transactions are sorted by TransactionDT and split chronologically: the earliest 70% form training, the next 15% form validation, and the latest 15% form the final test set. This better represents deployment on future transactions and allows natural class-prevalence drift to remain visible.

All subsequent preprocessing parameters are learned exclusively from the training period and applied unchanged to later periods.

# ==========================================================
# 5.1 Separate Features, Target, and Transaction IDs
# ==========================================================

TARGET_COLUMN = "isFraud"
ID_COLUMN = "TransactionID"

y = df_processed[TARGET_COLUMN].astype("int8").copy()
transaction_ids = df_processed[ID_COLUMN].copy()

X = df_processed.drop(
    columns=[TARGET_COLUMN, ID_COLUMN]
).copy()

print("=" * 60)
print("Features, Target, and IDs")
print("=" * 60)
print(f"Raw feature matrix shape: {X.shape}")
print(f"Target shape: {y.shape}")
print(f"Fraud rate: {y.mean() * 100:.2f}%")

============================================================
Features, Target, and IDs
============================================================
Raw feature matrix shape: (590540, 442)
Target shape: (590540,)
Fraud rate: 3.50%

# ==========================================================
# 5.2 Create Chronological 70% / 15% / 15% Split
# ==========================================================

chronological_order = X["TransactionDT"].sort_values(kind="mergesort").index
X_chronological = X.loc[chronological_order]
y_chronological = y.loc[chronological_order]
id_chronological = transaction_ids.loc[chronological_order]

train_end = int(len(X_chronological) * 0.70)
validation_end = int(len(X_chronological) * 0.85)

X_train = X_chronological.iloc[:train_end].copy()
X_validation = X_chronological.iloc[train_end:validation_end].copy()
X_test = X_chronological.iloc[validation_end:].copy()

y_train = y_chronological.iloc[:train_end].copy()
y_validation = y_chronological.iloc[train_end:validation_end].copy()
y_test = y_chronological.iloc[validation_end:].copy()

id_train = id_chronological.iloc[:train_end].copy()
id_validation = id_chronological.iloc[train_end:validation_end].copy()
id_test = id_chronological.iloc[validation_end:].copy()

split_boundaries_elapsed_days = {
    "training_start": float(X_train["TransactionDT"].min() / SECONDS_PER_DAY),
    "training_end": float(X_train["TransactionDT"].max() / SECONDS_PER_DAY),
    "validation_start": float(X_validation["TransactionDT"].min() / SECONDS_PER_DAY),
    "validation_end": float(X_validation["TransactionDT"].max() / SECONDS_PER_DAY),
    "test_start": float(X_test["TransactionDT"].min() / SECONDS_PER_DAY),
    "test_end": float(X_test["TransactionDT"].max() / SECONDS_PER_DAY),
}

print("Chronological dataset split completed successfully.")

# After removing target and TransactionID
feature_counts["After removing target and TransactionID"] = X.shape[1]

Chronological dataset split completed successfully.

# ==========================================================
# 5.3 Verify Split Sizes and Fraud Rates
# ==========================================================

split_summary = pd.DataFrame({
    "Dataset": ["Training", "Validation", "Test"],
    "Rows": [
        len(X_train),
        len(X_validation),
        len(X_test)
    ],
    "Columns": [
        X_train.shape[1],
        X_validation.shape[1],
        X_test.shape[1]
    ],
    "Fraud Cases": [
        int(y_train.sum()),
        int(y_validation.sum()),
        int(y_test.sum())
    ],
    "Fraud Rate (%)": [
        y_train.mean() * 100,
        y_validation.mean() * 100,
        y_test.mean() * 100
    ]
})

display(split_summary.round(2))

	Dataset	Rows	Columns	Fraud Cases	Fraud Rate (%)
0	Training	413378	442	14538	3.52
1	Validation	88581	442	3042	3.43
2	Test	88581	442	3083	3.48

# ==========================================================
# 5.4 Verify Identifier and Temporal Independence
# ==========================================================

train_ids = set(id_train)
validation_ids = set(id_validation)
test_ids = set(id_test)

assert not train_ids.intersection(validation_ids)
assert not train_ids.intersection(test_ids)
assert not validation_ids.intersection(test_ids)
assert X_train["TransactionDT"].max() <= X_validation["TransactionDT"].min()
assert X_validation["TransactionDT"].max() <= X_test["TransactionDT"].min()

temporal_ranges = pd.DataFrame({
    "Dataset": ["Training", "Validation", "Test"],
    "Start day": [
        X_train["TransactionDT"].min() / SECONDS_PER_DAY,
        X_validation["TransactionDT"].min() / SECONDS_PER_DAY,
        X_test["TransactionDT"].min() / SECONDS_PER_DAY,
    ],
    "End day": [
        X_train["TransactionDT"].max() / SECONDS_PER_DAY,
        X_validation["TransactionDT"].max() / SECONDS_PER_DAY,
        X_test["TransactionDT"].max() / SECONDS_PER_DAY,
    ],
})

print("No TransactionID overlap and chronological boundaries verified.")
display(temporal_ranges.style.format({"Start day": "{:.1f}", "End day": "{:.1f}"}))

No TransactionID overlap and chronological boundaries verified.

	Dataset	Start day	End day
0	Training	1.0	120.8
1	Validation	120.8	152.2
2	Test	152.2	183.0

Dataset Split Findings

The chronological split preserves transaction order, keeps every TransactionID in exactly one subset, and exposes changes in fraud prevalence across time. It does not force equal class proportions. From this point forward, only the earliest training period is used to learn preprocessing rules and transformation parameters.

6. Missing Value Treatment

Missing-value thresholds and imputation parameters are learned exclusively from the training dataset. The resulting feature-removal list and fitted imputers are then applied unchanged to the validation and test datasets.

# ==========================================================
# 6.1 Create Preprocessing Artifact Directory
# ==========================================================

PREPROCESSING_PATH = (
    PROJECT_ROOT
    / "models"
    / "preprocessing"
)

PREPROCESSING_PATH.mkdir(
    parents=True,
    exist_ok=True
)

print(f"Preprocessing artifacts directory:\n{PREPROCESSING_PATH}")

Preprocessing artifacts directory:
/Users/hshazel/Projects/explainable-fraud-investigation-platform/models/preprocessing

# ==========================================================
# 6.2 Identify High-Missing Features from Training Data
# ==========================================================

MISSING_THRESHOLD = 95

training_missing_percentage = (
    X_train.isna().mean() * 100
)

high_missing_features = (
    training_missing_percentage[
        training_missing_percentage > MISSING_THRESHOLD
    ]
    .index
    .tolist()
)

print("=" * 60)
print("High-Missing Features")
print("=" * 60)
print(f"Threshold: {MISSING_THRESHOLD}%")
print(f"Features identified: {len(high_missing_features)}")

display(
    pd.DataFrame({
        "Feature": high_missing_features,
        "Training Missing (%)": (
            training_missing_percentage[
                high_missing_features
            ].values
        )
    }).sort_values(
        "Training Missing (%)",
        ascending=False
    )
)

============================================================
High-Missing Features
============================================================
Threshold: 95%
Features identified: 9

	Feature	Training Missing (%)
5	id_24	99.13
6	id_25	99.05
0	id_07	99.05
1	id_08	99.05
7	id_26	99.05
2	id_21	99.05
3	id_22	99.05
4	id_23	99.05
8	id_27	99.05

# ==========================================================
# 6.3 Remove Training-Selected High-Missing Features
# ==========================================================

for dataset in [X_train, X_validation, X_test]:
    dataset.drop(
        columns=high_missing_features,
        inplace=True
    )

joblib.dump(
    high_missing_features,
    PREPROCESSING_PATH / "high_missing_features.pkl"
)

print(f"Remaining training features: {X_train.shape[1]}")

# After removing high-missing features
feature_counts["After removing >95% missing features"] = X_train.shape[1]

Remaining training features: 433

# ==========================================================
# 6.4 Identify Numerical and Categorical Features
# ==========================================================

numeric_columns = (
    X_train
    .select_dtypes(include=np.number)
    .columns
    .tolist()
)

categorical_columns = (
    X_train
    .select_dtypes(include=["object", "string"])
    .columns
    .tolist()
)

print(f"Numerical columns: {len(numeric_columns)}")
print(f"Categorical columns: {len(categorical_columns)}")

Numerical columns: 401
Categorical columns: 32

# ==========================================================
# 6.5 Fit Numerical Imputer on Training Data
# ==========================================================

median_imputer = SimpleImputer(
    strategy="median"
)

X_train.loc[:, numeric_columns] = (
    median_imputer.fit_transform(
        X_train[numeric_columns]
    )
)

X_validation.loc[:, numeric_columns] = (
    median_imputer.transform(
        X_validation[numeric_columns]
    )
)

X_test.loc[:, numeric_columns] = (
    median_imputer.transform(
        X_test[numeric_columns]
    )
)

print("Numerical imputation completed.")

Numerical imputation completed.

# ==========================================================
# 6.6 Fit Categorical Imputer on Training Data
# ==========================================================

mode_imputer = SimpleImputer(
    strategy="most_frequent"
)

X_train.loc[:, categorical_columns] = (
    mode_imputer.fit_transform(
        X_train[categorical_columns]
    )
)

X_validation.loc[:, categorical_columns] = (
    mode_imputer.transform(
        X_validation[categorical_columns]
    )
)

X_test.loc[:, categorical_columns] = (
    mode_imputer.transform(
        X_test[categorical_columns]
    )
)

print("Categorical imputation completed.")

Categorical imputation completed.

# ==========================================================
# 6.7 Save Imputation Artifacts
# ==========================================================

joblib.dump(
    median_imputer,
    PREPROCESSING_PATH / "median_imputer.pkl"
)

joblib.dump(
    mode_imputer,
    PREPROCESSING_PATH / "mode_imputer.pkl"
)

joblib.dump(
    numeric_columns,
    PREPROCESSING_PATH / "imputed_numeric_features.pkl"
)

joblib.dump(
    categorical_columns,
    PREPROCESSING_PATH / "imputed_categorical_features.pkl"
)

print("Imputation artifacts saved.")

Imputation artifacts saved.

# ==========================================================
# 6.8 Verify Missing Value Treatment
# ==========================================================

print("=" * 60)
print("Missing Value Verification")
print("=" * 60)
print(
    "Training missing values:",
    int(X_train.isna().to_numpy().sum())
)
print(
    "Validation missing values:",
    int(X_validation.isna().to_numpy().sum())
)
print(
    "Test missing values:",
    int(X_test.isna().to_numpy().sum())
)

============================================================
Missing Value Verification
============================================================
Training missing values: 0
Validation missing values: 0
Test missing values: 0

Missing Value Treatment Findings

Features exceeding 95% missingness were identified using the training dataset only and removed consistently from all three subsets.

Numerical and categorical imputers were fitted exclusively on the training data. The learned medians and most-frequent categories were then applied unchanged to the validation and test data, preventing information from those subsets from influencing preprocessing.

7. Remove Low-Information Features

Constant and near-constant features are detected using the imputed training dataset only. The resulting removal list is then applied consistently to the validation and test datasets.

# ==========================================================
# 7.1 Identify Constant and Near-Constant Features
# ==========================================================

NEAR_CONSTANT_THRESHOLD = 0.995

constant_features = [
    column
    for column in X_train.columns
    if X_train[column].nunique(dropna=False) <= 1
]

near_constant_features = []

for column in X_train.columns:
    top_frequency = (
        X_train[column]
        .value_counts(
            normalize=True,
            dropna=False
        )
        .iloc[0]
    )

    if top_frequency >= NEAR_CONSTANT_THRESHOLD:
        near_constant_features.append(column)

low_information_features = sorted(
    set(constant_features + near_constant_features)
)

print("=" * 60)
print("Low-Information Features")
print("=" * 60)
print(f"Constant features: {len(constant_features)}")
print(f"Near-constant features: {len(near_constant_features)}")
print(f"Unique features to remove: {len(low_information_features)}")

display(
    pd.DataFrame({
        "Feature": low_information_features
    })
)

============================================================
Low-Information Features
============================================================
Constant features: 1
Near-constant features: 33
Unique features to remove: 33

	Feature
0	C3
1	M1
2	V1
3	V107
4	V108
5	V111
6	V112
7	V113
8	V117
9	V118
10	V119
11	V120
12	V121
13	V122
14	V138
15	V14
16	V191
17	V240
18	V241
19	V247
20	V252
21	V269
22	V27
23	V28
24	V305
25	V41
26	V65
27	V68
28	V88
29	V89
30	id_03
31	id_04
32	id_10

# ==========================================================
# 7.2 Remove Training-Selected Low-Information Features
# ==========================================================

for dataset in [X_train, X_validation, X_test]:
    dataset.drop(
        columns=low_information_features,
        inplace=True
    )

joblib.dump(
    low_information_features,
    PREPROCESSING_PATH / "low_information_features.pkl"
)

print(f"Remaining training features: {X_train.shape[1]}")

# After removing low-information features
feature_counts["After removing low-information features"] = X_train.shape[1]

Remaining training features: 400

Low-Information Feature Findings

Low-information features were selected exclusively from the training data using a 99.5% dominant-value threshold. Removing the same columns from every subset reduces unnecessary dimensionality while preserving identical feature spaces.

8. Encode Categorical Variables

Categorical preprocessing uses a hybrid strategy. Low-cardinality variables are one-hot encoded, while high-cardinality variables are frequency encoded.

Feature cardinality, frequency maps, and one-hot categories are all learned from the training dataset only. Validation and test observations are transformed using those training-derived artifacts.

# ==========================================================
# 8.1 Split Categorical Features by Training Cardinality
# ==========================================================

MAX_ONEHOT_CATEGORIES = 20

categorical_features = (
    X_train
    .select_dtypes(include=["object", "string"])
    .columns
    .tolist()
)

low_cardinality_features = [
    column
    for column in categorical_features
    if X_train[column].nunique(dropna=False)
    <= MAX_ONEHOT_CATEGORIES
]

high_cardinality_features = [
    column
    for column in categorical_features
    if X_train[column].nunique(dropna=False)
    > MAX_ONEHOT_CATEGORIES
]

cardinality_summary = pd.DataFrame({
    "Feature": categorical_features,
    "Training Unique Categories": [
        X_train[column].nunique(dropna=False)
        for column in categorical_features
    ],
    "Encoding": [
        "One-Hot"
        if column in low_cardinality_features
        else "Frequency"
        for column in categorical_features
    ]
}).sort_values(
    "Training Unique Categories",
    ascending=False
)

print(f"Low-cardinality features: {len(low_cardinality_features)}")
print(f"High-cardinality features: {len(high_cardinality_features)}")
display(cardinality_summary)

Low-cardinality features: 22
High-cardinality features: 9

	Feature	Training Unique Categories	Encoding
30	CardID	12955	Frequency
27	DeviceInfo	1546	Frequency
20	id_33	183	Frequency
19	id_31	108	Frequency
18	id_30	71	Frequency
4	R_emaildomain	60	Frequency
3	P_emaildomain	59	Frequency
29	R_EmailProvider	46	Frequency
28	P_EmailProvider	45	Frequency
0	ProductCD	5	One-Hot
2	card6	4	One-Hot
1	card4	4	One-Hot
21	id_34	4	One-Hot
7	M4	3	One-Hot
14	id_15	3	One-Hot
22	id_35	2	One-Hot
5	M2	2	One-Hot
6	M3	2	One-Hot
26	DeviceType	2	One-Hot
25	id_38	2	One-Hot
24	id_37	2	One-Hot
23	id_36	2	One-Hot
10	M7	2	One-Hot
11	M8	2	One-Hot
8	M5	2	One-Hot
9	M6	2	One-Hot
17	id_29	2	One-Hot
16	id_28	2	One-Hot
13	id_12	2	One-Hot
12	M9	2	One-Hot
15	id_16	2	One-Hot

# ==========================================================
# 8.2 Frequency Encode High-Cardinality Features
# ==========================================================

frequency_maps = {}

for column in high_cardinality_features:
    frequency_map = (
        X_train[column]
        .value_counts(
            normalize=True,
            dropna=False
        )
        .to_dict()
    )

    frequency_maps[column] = frequency_map

    for dataset in [X_train, X_validation, X_test]:
        dataset[f"{column}_Frequency"] = (
            dataset[column]
            .map(frequency_map)
            .fillna(0)
            .astype(np.float32)
        )

for dataset in [X_train, X_validation, X_test]:
    dataset.drop(
        columns=high_cardinality_features,
        inplace=True
    )

print(
    f"Frequency-encoded "
    f"{len(high_cardinality_features)} features."
)

Frequency-encoded 9 features.

# ==========================================================
# 8.3 Fit One-Hot Encoder on Training Data
# ==========================================================

encoder = OneHotEncoder(
    handle_unknown="ignore",
    sparse_output=False,
    dtype=np.float32
)

train_encoded_array = encoder.fit_transform(
    X_train[low_cardinality_features]
)

validation_encoded_array = encoder.transform(
    X_validation[low_cardinality_features]
)

test_encoded_array = encoder.transform(
    X_test[low_cardinality_features]
)

onehot_feature_names = encoder.get_feature_names_out(
    low_cardinality_features
)

print(
    f"One-hot encoded features created: "
    f"{len(onehot_feature_names)}"
)

One-hot encoded features created: 55

# ==========================================================
# 8.4 Replace Original Low-Cardinality Variables
# ==========================================================

train_encoded_df = pd.DataFrame(
    train_encoded_array,
    index=X_train.index,
    columns=onehot_feature_names
)

validation_encoded_df = pd.DataFrame(
    validation_encoded_array,
    index=X_validation.index,
    columns=onehot_feature_names
)

test_encoded_df = pd.DataFrame(
    test_encoded_array,
    index=X_test.index,
    columns=onehot_feature_names
)

X_train = pd.concat(
    [
        X_train.drop(columns=low_cardinality_features),
        train_encoded_df
    ],
    axis=1
)

X_validation = pd.concat(
    [
        X_validation.drop(columns=low_cardinality_features),
        validation_encoded_df
    ],
    axis=1
)

X_test = pd.concat(
    [
        X_test.drop(columns=low_cardinality_features),
        test_encoded_df
    ],
    axis=1
)

print(f"Training shape: {X_train.shape}")
print(f"Validation shape: {X_validation.shape}")
print(f"Test shape: {X_test.shape}")

# After categorical encoding
feature_counts["After categorical encoding"] = X_train.shape[1]

Training shape: (413378, 433)
Validation shape: (88581, 433)
Test shape: (88581, 433)

# ==========================================================
# 8.5 Save Categorical Preprocessing Artifacts
# ==========================================================

joblib.dump(
    encoder,
    PREPROCESSING_PATH / "onehot_encoder.pkl"
)

joblib.dump(
    onehot_feature_names,
    PREPROCESSING_PATH / "encoded_feature_names.pkl"
)

joblib.dump(
    frequency_maps,
    PREPROCESSING_PATH / "frequency_maps.pkl"
)

joblib.dump(
    low_cardinality_features,
    PREPROCESSING_PATH / "low_cardinality_features.pkl"
)

joblib.dump(
    high_cardinality_features,
    PREPROCESSING_PATH / "high_cardinality_features.pkl"
)

print("Categorical preprocessing artifacts saved.")

Categorical preprocessing artifacts saved.

# ==========================================================
# 8.6 Verify Encoded Feature Consistency
# ==========================================================

assert list(X_train.columns) == list(X_validation.columns)
assert list(X_train.columns) == list(X_test.columns)

print("=" * 60)
print("Encoded Feature Verification")
print("=" * 60)
print(
    "Train/Validation columns match:",
    list(X_train.columns) == list(X_validation.columns)
)
print(
    "Train/Test columns match:",
    list(X_train.columns) == list(X_test.columns)
)
print(
    "Training missing values:",
    int(X_train.isna().to_numpy().sum())
)
print(
    "Validation missing values:",
    int(X_validation.isna().to_numpy().sum())
)
print(
    "Test missing values:",
    int(X_test.isna().to_numpy().sum())
)

============================================================
Encoded Feature Verification
============================================================
Train/Validation columns match: True
Train/Test columns match: True
Training missing values: 0
Validation missing values: 0
Test missing values: 0

Categorical Encoding Findings

Categorical encoding is now leakage-free. Cardinality decisions, frequency mappings, and one-hot categories were learned from the training dataset only.

Unknown validation or test categories are handled safely: one-hot encoding ignores unseen categories, while unseen high-cardinality values receive a frequency of zero.

9. Scale Numerical Features

The numerical variables in the dataset operate on substantially different scales. For example, transaction amounts, elapsed-time variables, count-based features, and identity-related measurements have different numerical ranges.

Feature scaling is particularly important for algorithms such as Logistic Regression and neural networks, which are sensitive to differences in feature magnitude. Tree-based algorithms such as Decision Trees, Random Forest, and XGBoost are generally less sensitive to scaling.

RobustScaler is used because several numerical variables contain highly skewed distributions and extreme values. The scaler uses the median and interquartile range, making it less sensitive to outliers than standardization based on the mean and standard deviation.

To reduce data leakage, the scaler is fitted exclusively on the training dataset. The fitted transformation is then applied unchanged to the validation and test datasets.

9.1 Identify features to scale

One-hot encoded variables and binary indicators are intentionally excluded from scaling.

# ==========================================================
# 9.1 Identify Numerical Features to Scale
# ==========================================================

numerical_columns = (
    X_train
    .select_dtypes(include=np.number)
    .columns
    .tolist()
)

# One-hot columns created by the encoder
onehot_columns = [
    column
    for column in onehot_feature_names
    if column in X_train.columns
]

# Binary features should remain coded as 0 and 1
binary_columns = [
    column
    for column in numerical_columns
    if X_train[column].nunique(dropna=False) <= 2
]

numerical_features_to_scale = [
    column
    for column in numerical_columns
    if column not in onehot_columns
    and column not in binary_columns
]

print("=" * 60)
print("Numerical Feature Selection")
print("=" * 60)

print(f"Total model features: {X_train.shape[1]}")
print(f"Numerical features: {len(numerical_columns)}")
print(f"One-hot features excluded: {len(onehot_columns)}")
print(f"Binary features excluded: {len(binary_columns)}")
print(
    f"Numerical features selected for scaling: "
    f"{len(numerical_features_to_scale)}"
)

============================================================
Numerical Feature Selection
============================================================
Total model features: 433
Numerical features: 433
One-hot features excluded: 55
Binary features excluded: 59
Numerical features selected for scaling: 374

display(
    pd.DataFrame({
        "Feature": numerical_features_to_scale
    }).head(30)
)

	Feature
0	TransactionDT
1	TransactionAmt
2	card1
3	card2
4	card3
5	card5
6	addr1
7	addr2
8	dist1
9	dist2
10	C1
11	C2
12	C4
13	C5
14	C6
15	C7
16	C8
17	C9
18	C10
19	C11
20	C12
21	C13
22	C14
23	D1
24	D2
25	D3
26	D4
27	D5
28	D6
29	D7

9.2 Convert selected columns to float32

This reduces memory usage and prevents dtype-assignment warnings.

# ==========================================================
# 9.2 Prepare Numerical Columns
# ==========================================================

for dataset in [X_train, X_validation, X_test]:
    dataset[numerical_features_to_scale] = (
        dataset[numerical_features_to_scale]
        .astype(np.float32)
    )

print("Numerical features converted to float32.")

Numerical features converted to float32.

9.3 Fit the scaler on training data only

# ==========================================================
# 9.3 Fit RobustScaler
# ==========================================================

robust_scaler = RobustScaler()

robust_scaler.fit(
    X_train[numerical_features_to_scale] # We use X_train, not df_processed, to avoid data leakage
)

print("RobustScaler fitted on the training dataset.")

RobustScaler fitted on the training dataset.

9.4 Apply Scaling

# ==========================================================
# 9.4 Transform Train, Validation, and Test Data
# ==========================================================

X_train.loc[:, numerical_features_to_scale] = (
    robust_scaler.transform(
        X_train[numerical_features_to_scale]
    )
)

X_validation.loc[:, numerical_features_to_scale] = (
    robust_scaler.transform(
        X_validation[numerical_features_to_scale]
    )
)

X_test.loc[:, numerical_features_to_scale] = (
    robust_scaler.transform(
        X_test[numerical_features_to_scale]
    )
)

print("Numerical scaling completed successfully.")

Numerical scaling completed successfully.

9.5 Save the scaler and feature list

# ==========================================================
# 9.5 Save Scaling Artifacts
# ==========================================================

PREPROCESSING_PATH = (
    PROJECT_ROOT
    / "models"
    / "preprocessing"
)

PREPROCESSING_PATH.mkdir(
    parents=True,
    exist_ok=True
)

joblib.dump(
    robust_scaler,
    PREPROCESSING_PATH / "robust_scaler.pkl"
)

joblib.dump(
    numerical_features_to_scale,
    PREPROCESSING_PATH / "scaled_feature_names.pkl"
)

print("Scaling artifacts saved successfully.")
print(
    f"Scaler path: "
    f"{PREPROCESSING_PATH / 'robust_scaler.pkl'}"
)

Scaling artifacts saved successfully.
Scaler path: /Users/hshazel/Projects/explainable-fraud-investigation-platform/models/preprocessing/robust_scaler.pkl

9.6 Verify the transformation

# ==========================================================
# 9.6 Scaling Verification
# ==========================================================

scaling_verification = pd.DataFrame({
    "Scaled Median": (
        X_train[numerical_features_to_scale]
        .median()
    ),
    "Scaled Q1": (
        X_train[numerical_features_to_scale]
        .quantile(0.25)
    ),
    "Scaled Q3": (
        X_train[numerical_features_to_scale]
        .quantile(0.75)
    )
})

scaling_verification["Scaled IQR"] = (
    scaling_verification["Scaled Q3"]
    - scaling_verification["Scaled Q1"]
)

display(
    scaling_verification
    .head(20)
    .round(3)
)

	Scaled Median	Scaled Q1	Scaled Q3	Scaled IQR
TransactionDT	0.00	-0.48	0.52	1.00
TransactionAmt	0.00	-0.32	0.68	1.00
card1	0.00	-0.45	0.56	1.00
card2	0.00	-0.49	0.51	1.00
card3	0.00	0.00	0.00	0.00
card5	0.00	-1.00	0.00	1.00
addr1	0.00	-0.77	0.23	1.00
addr2	0.00	0.00	0.00	0.00
dist1	0.00	0.00	0.00	0.00
dist2	0.00	0.00	0.00	0.00
C1	0.00	0.00	1.00	1.00
C2	0.00	0.00	1.00	1.00
C4	0.00	0.00	1.00	1.00
C5	0.00	0.00	1.00	1.00
C6	0.00	0.00	1.00	1.00
C7	0.00	0.00	0.00	0.00
C8	0.00	0.00	1.00	1.00
C9	0.00	-0.50	0.50	1.00
C10	0.00	0.00	1.00	1.00
C11	0.00	0.00	1.00	1.00

9.7 Final integrity checks

# ==========================================================
# 9.7 Scaling Integrity Checks
# ==========================================================

print("=" * 60)
print("Scaling Integrity Checks")
print("=" * 60)

print(
    "Train/Validation columns match:",
    list(X_train.columns)
    == list(X_validation.columns)
)

print(
    "Train/Test columns match:",
    list(X_train.columns)
    == list(X_test.columns)
)

print(
    "Missing values in training:",
    int(X_train.isna().to_numpy().sum())
)

print(
    "Missing values in validation:",
    int(X_validation.isna().to_numpy().sum())
)

print(
    "Missing values in test:",
    int(X_test.isna().to_numpy().sum())
)

def count_infinite_values(dataframe):
    return int(
        np.isinf(
            dataframe.to_numpy(dtype=np.float32)
        ).sum()
    )


print(
    "Infinite values in training:",
    count_infinite_values(X_train)
)

print(
    "Infinite values in validation:",
    count_infinite_values(X_validation)
)

print(
    "Infinite values in test:",
    count_infinite_values(X_test)
)

print(f"Training shape: {X_train.shape}")
print(f"Validation shape: {X_validation.shape}")
print(f"Test shape: {X_test.shape}")

============================================================
Scaling Integrity Checks
============================================================
Train/Validation columns match: True
Train/Test columns match: True
Missing values in training: 0
Missing values in validation: 0
Missing values in test: 0
Infinite values in training: 0
Infinite values in validation: 0
Infinite values in test: 0
Training shape: (413378, 433)
Validation shape: (88581, 433)
Test shape: (88581, 433)

Numerical Feature Scaling Findings

A total of 374 numerical features were transformed using RobustScaler.

One-hot encoded variables and binary indicator variables were excluded from scaling to preserve their original binary representation. The remaining numerical variables were scaled using the median and interquartile range calculated from the training dataset, reducing the influence of outliers and differences in feature magnitude.

The scaler was fitted exclusively on the training data and then applied unchanged to the validation and test datasets, preventing data leakage.

Post-scaling verification confirmed that: - All datasets retained identical feature sets consisting of 433 features before correlation filtering. - No missing values or infinite values were introduced during scaling. - The median of most scaled numerical variables is approximately 0, while variables with sufficient variability have an interquartile range close to 1, indicating that the RobustScaler transformation was applied successfully. - Several variables retained an interquartile range of 0 because the majority of their observations share the same value after preprocessing. This behavior is expected for low-variance or highly concentrated features and does not indicate an error.

The scaled datasets are now prepared for machine learning models that are sensitive to feature magnitude, including Logistic Regression and neural networks.

10. Feature Selection

The objective of this section is to reduce unnecessary model complexity by removing features that contribute little predictive information while preserving variables that may improve fraud detection.

Feature selection is performed using data-driven criteria rather than manual inspection. Constant features are removed because they contain no information, and highly correlated numerical variables are eliminated to reduce redundancy.

Model-based feature importance will be applied later during model development to further evaluate feature relevance.

10.1 Remove Constant Features

# ==========================================================
# 10.1 Remove Constant Features
# ==========================================================

constant_features = [
    column
    for column in X_train.columns
    if X_train[column].nunique(dropna=False) <= 1
]

print("=" * 60)
print("Constant Features")
print("=" * 60)

print(f"Constant features found: {len(constant_features)}")

if constant_features:
    display(pd.DataFrame({
        "Feature": constant_features
    }))

============================================================
Constant Features
============================================================
Constant features found: 0

10.2 Remove Constant Features from All Sets

# ==========================================================
# 10.2 Remove Constant Features
# ==========================================================

X_train.drop(
    columns=constant_features,
    inplace=True
)

X_validation.drop(
    columns=constant_features,
    inplace=True
)

X_test.drop(
    columns=constant_features,
    inplace=True
)

print(f"Remaining features: {X_train.shape[1]}")

Remaining features: 433

10.3 Correlation Analysis

Only evaluate continuous numerical variables.

# ==========================================================
# 10.3 Correlation Analysis
# ==========================================================

continuous_features = [
    column
    for column in numerical_features_to_scale
    if column in X_train.columns
]

correlation_matrix = (
    X_train[continuous_features]
    .corr()
    .abs()
)

upper_triangle = correlation_matrix.where(
    np.triu(
        np.ones(correlation_matrix.shape),
        k=1
    ).astype(bool)
)

correlated_features = [
    column
    for column in upper_triangle.columns
    if any(upper_triangle[column] > 0.95)
]

print("=" * 60)
print("Highly Correlated Features")
print("=" * 60)

print(f"Features to remove: {len(correlated_features)}")

============================================================
Highly Correlated Features
============================================================
Features to remove: 74

10.4 Remove Correlated Features

# ==========================================================
# 10.4 Remove Correlated Features
# ==========================================================

X_train.drop(
    columns=correlated_features,
    inplace=True
)

X_validation.drop(
    columns=correlated_features,
    inplace=True
)

X_test.drop(
    columns=correlated_features,
    inplace=True
)

joblib.dump(
    correlated_features,
    PREPROCESSING_PATH / "correlated_features_removed.pkl"
)

print(f"Remaining features: {X_train.shape[1]}")

# After correlation filtering
feature_counts["After correlation filtering"] = X_train.shape[1]

Remaining features: 359

10.5 Save Selected Features

# ==========================================================
# 10.5 Save Selected Features
# ==========================================================

selected_features = X_train.columns.tolist()

joblib.dump(
    selected_features,
    PROJECT_ROOT
    / "models"
    / "preprocessing"
    / "selected_features.pkl"
)

print("Selected feature list saved.")

Selected feature list saved.

10.6 Final Dataset Summary

# ==========================================================
# 10.6 Dataset Summary
# ==========================================================

summary = pd.DataFrame({
    "Dataset": [
        "Training",
        "Validation",
        "Test"
    ],
    "Rows": [
        len(X_train),
        len(X_validation),
        len(X_test)
    ],
    "Features": [
        X_train.shape[1],
        X_validation.shape[1],
        X_test.shape[1]
    ]
})

display(summary)

	Dataset	Rows	Features
0	Training	413378	359
1	Validation	88581	359
2	Test	88581	359

# ==========================================================
# Feature Count Throughout the Preprocessing Pipeline
# ==========================================================

pipeline_summary = pd.DataFrame(
    feature_counts.items(),
    columns=["Stage", "Features"]
)

pipeline_summary["Features Removed / Added"] = (
    pipeline_summary["Features"]
    .diff()
    .fillna(0)
    .astype(int)
)

display(pipeline_summary)

	Stage	Features	Features Removed / Added
0	Original merged dataset	434	0
1	After deterministic feature engineering	444	10
2	After removing target and TransactionID	442	-2
3	After removing >95% missing features	433	-9
4	After removing low-information features	400	-33
5	After categorical encoding	433	33
6	After correlation filtering	359	-74

Feature Selection Findings

Feature selection was performed to reduce redundancy and model complexity while retaining one representative feature from each highly correlated group.

First, constant features containing no variability were identified and removed because they cannot contribute to machine learning models. Next, highly correlated numerical features (absolute Pearson correlation greater than 0.95) were eliminated to reduce multicollinearity, simplify the feature space, and improve computational efficiency while retaining much of the shared linear information.

The same features were removed consistently from the training, validation, and test datasets, ensuring that all datasets maintained identical feature spaces throughout the machine learning pipeline.

After the earlier low-information filter, the final constant-feature check found 0 additional constant columns. Correlation filtering then removed 74 features, reducing the model matrix from 433 to 359 features and creating a more compact dataset for model development. Tree-based feature importance will be applied during model training and evaluation rather than during preprocessing.

11. Save Processed Data

The final step in the preprocessing pipeline is to persist the processed datasets and preprocessing artifacts for reuse in subsequent notebooks.

Saving the processed training, validation, and test datasets ensures that model development can begin immediately without repeating the computationally intensive preprocessing steps. This also improves reproducibility and guarantees that every model is trained using exactly the same feature set.

In addition to the processed datasets, all preprocessing objects—including imputers, encoders, scalers, and the final feature list—have been saved to support future inference and deployment.

11.1 Create Output Directory

# ==========================================================
# 11.1 Create Output Directory
# ==========================================================

PROCESSED_DATA_PATH = (
    PROJECT_ROOT
    / "data"
    / "processed"
)

PROCESSED_DATA_PATH.mkdir(
    parents=True,
    exist_ok=True
)

print(f"Output directory:\n{PROCESSED_DATA_PATH}")

Output directory:
/Users/hshazel/Projects/explainable-fraud-investigation-platform/data/processed

11.2 Save the Datasets

import pyarrow as pa

import pandas as pd

print("PyArrow:", pa.__version__)

print("Pandas:", pd.__version__)

PyArrow: 24.0.0
Pandas: 3.0.3

# ==========================================================
# 11.2 Save Processed Datasets
# ==========================================================
X_train.to_parquet(
    PROCESSED_DATA_PATH / "X_train.parquet",
    index=False
)

X_validation.to_parquet(
    PROCESSED_DATA_PATH / "X_validation.parquet",
    index=False
)

X_test.to_parquet(
    PROCESSED_DATA_PATH / "X_test.parquet",
    index=False
)

y_train.to_frame("isFraud").to_parquet(
    PROCESSED_DATA_PATH / "y_train.parquet",
    index=False
)

y_validation.to_frame("isFraud").to_parquet(
    PROCESSED_DATA_PATH / "y_validation.parquet",
    index=False
)

y_test.to_frame("isFraud").to_parquet(
    PROCESSED_DATA_PATH / "y_test.parquet",
    index=False
)

print("Processed datasets saved successfully.")

Processed datasets saved successfully.

11.3 Save Transaction IDs

They will be used later for fraud investigations

# ==========================================================
# 11.3 Save Transaction IDs
# ==========================================================

id_train.to_frame("TransactionID").to_parquet(
    PROCESSED_DATA_PATH / "id_train.parquet",
    index=False
)

id_validation.to_frame("TransactionID").to_parquet(
    PROCESSED_DATA_PATH / "id_validation.parquet",
    index=False
)

id_test.to_frame("TransactionID").to_parquet(
    PROCESSED_DATA_PATH / "id_test.parquet",
    index=False
)

print("Transaction IDs saved successfully.")

# Save the split policy and feature contract used by later notebooks and the app.
feature_schema_sha256 = hashlib.sha256(
    "\n".join(selected_features).encode("utf-8")
).hexdigest()

preprocessing_metadata = {
    "split_policy": "chronological 70/15/15 by TransactionDT",
    "split_boundaries_elapsed_days": split_boundaries_elapsed_days,
    "rows": {
        "training": len(X_train),
        "validation": len(X_validation),
        "test": len(X_test),
    },
    "fraud_rates": {
        "training": float(y_train.mean()),
        "validation": float(y_validation.mean()),
        "test": float(y_test.mean()),
    },
    "feature_count": len(selected_features),
    "feature_schema_sha256": feature_schema_sha256,
    "calendar_warning": (
        "TransactionDT has an unspecified origin; calendar weekday/weekend was not inferred."
    ),
}
(PREPROCESSING_PATH / "preprocessing_metadata.json").write_text(
    json.dumps(preprocessing_metadata, indent=2), encoding="utf-8"
)
print("Preprocessing metadata saved successfully.")

Transaction IDs saved successfully.
Preprocessing metadata saved successfully.

11.4 Summary

# ==========================================================
# 11.4 Saved Files Summary
# ==========================================================

summary = pd.DataFrame({
    "Dataset": [
        "X_train",
        "X_validation",
        "X_test",
        "y_train",
        "y_validation",
        "y_test",
        "id_train",
        "id_validation",
        "id_test"
    ],
    "Rows": [
        len(X_train),
        len(X_validation),
        len(X_test),
        len(y_train),
        len(y_validation),
        len(y_test),
        len(id_train),
        len(id_validation),
        len(id_test)
    ]
})

display(summary)

print()
print(f"Total model features: {X_train.shape[1]}")
print("All processed datasets successfully exported.")

	Dataset	Rows
0	X_train	413378
1	X_validation	88581
2	X_test	88581
3	y_train	413378
4	y_validation	88581
5	y_test	88581
6	id_train	413378
7	id_validation	88581
8	id_test	88581

Total model features: 359
All processed datasets successfully exported.

Processed Data Export Findings

The fully preprocessed datasets were successfully exported in Parquet format for use during the model development phase. The training, validation, and test datasets preserve identical feature spaces consisting of 359 processed model features, ensuring consistency throughout the machine learning pipeline.

The corresponding target variables and transaction identifiers were also saved separately to support model evaluation, fraud investigation, and explainability analyses.

By exporting the processed datasets and preprocessing artifacts, the project substantially improves reproducibility and eliminates the need to repeat computationally intensive preprocessing during subsequent experiments.

12. Data Wrangling, Preprocessing & Feature Engineering Summary and Next Steps

The data-wrangling, preprocessing, and feature-engineering workflow included data quality assessment, deterministic feature engineering, chronological dataset partitioning, removal of high-missing features, training-fitted imputation, removal of low-information features, categorical encoding, robust numerical scaling, correlation-based filtering, and export of the processed datasets and preprocessing artifacts.

Data Wrangling and Preprocessing Pipeline

• ✓ Data quality assessment
• ✓ Deterministic feature engineering without unsupported calendar assumptions
• ✓ Chronological train / validation / test split
• ✓ Identifier and temporal-boundary verification
• ✓ Removal of high-missing and low-information features
• ✓ Training-fitted numerical and categorical imputation
• ✓ Frequency and one-hot encoding fitted on training only
• ✓ Robust scaling fitted on training only
• ✓ Correlation-based feature selection
• ✓ Export of processed data, preprocessing objects, and feature-contract metadata

The next notebook will compare multiple classifiers under class imbalance, tune the selected boosting model with time-aware cross-validation, choose an operating threshold on validation data, and evaluate the frozen decision once on the later test period.