Lecture 22: Data Preparation & Feature Engineering

High-quality data beats clever modeling. Today we systematize data cleaning, splitting, encoding, scaling, handling imbalance/missingness, and building robust feature pipelines.

1) Data Quality Dimensions

Dimension	Questions	Remedies
Completeness	Missingness pattern MCAR/MAR/MNAR?	Impute (mean/median/mode), model-based, indicator flags
Consistency	Units, duplicated ids?	Standardize units, de-duplicate, canonicalize categories
Validity	Schema ranges respected?	Clamp, winsorize, domain rules
Timeliness	Stale or future values?	Cut by time, roll-forward features
Bias	Sampling/label bias?	Audit distributions, stratify, fairness metrics

2) Splits & Leakage (Patterns)

Random / Stratified Split – IID data, keep label proportions.
Group Split – keep all samples of an entity together (patients/users).
Time-based Split – train on past, test on future.

Leakage Watchlist

Fit scalers/encoders on full data → don’t.
Features computed using future info.
Duplicates of test rows in train.

3) Encoding Categorical Variables

One-hot – safe, may increase dimensionality.
Ordinal – for true order (small<medium<large).
Target / Mean Encoding – powerful, but use CV to avoid leakage.
Hashing – for high-cardinality features (stable width).

4) Scaling & Transformations

Standardize (z-score) – good for linear/SVM/kNN.
Min–Max – keep 0..1 range (NNs/interpretability).
Robust Scaler – median/IQR for outliers.
Log / Box-Cox / Yeo-Johnson – handle skew.

5) Missing Values & Outliers

Imputation

Simple: mean/median/mode + is_missing flag.
Model-based: kNN, iterative imputer.
Domain: carry-forward (time-series), clinical ranges.

Outliers

Detect: z-score, IQR, isolation forest.
Handle: cap/winsorize, transform, robust models.

6) Class Imbalance

Resampling – Random under/over, SMOTE/ADASYN.
Class weights – cost-sensitive learning.
Metrics – prefer ROC-AUC/PR-AUC, F1, recall at k.

7) Feature Engineering Patterns

Numeric

Ratios, differences, interactions (x1·x2)
Rolling stats (mean/var) for sequences

Categorical

Count/frequency encoding
Crossed features (city×device)

Datetime

Hour/Day/Month, weekend/holiday
Cyclical sin/cos for hour-of-day

Text (NLP)

Bag-of-Words, TF-IDF, n-grams
Embeddings (word/sentence) – use pre-trained

Images

Resize/normalize
Augment: flip, crop, jitter

Time-series

Lags, rolling window features
Seasonal indicators

8) Safe Pipelines

Principle: Put all preprocessing inside a pipeline so that fitting uses train-only statistics and transforms are applied identically to validation/test.

# Pseudocode (sklearn-style)
Pipeline([
  ("impute", SimpleImputer(strategy="median")),
  ("encode", OneHotEncoder(handle_unknown="ignore")),
  ("scale", StandardScaler(with_mean=False)),
  ("model", LogisticRegression(class_weight="balanced"))
])

Try This: Build two versions of your dataset: (A) raw; (B) with engineered features (ratios, lags, TF-IDF). Compare ROC-AUC via 5-fold CV. Report which features moved the needle most.