Lecture 24: Evaluation, Tuning, Deployment & Ethics

1) Model Evaluation

Classification Metrics

Accuracy, Precision, Recall, F1
ROC-AUC, PR-AUC for imbalanced data
Confusion Matrix for class-wise errors

Regression Metrics

MSE, RMSE, MAE
R² – proportion of variance explained
MAPE for interpretability in % terms

2) Validation Techniques

Train/Validation/Test Split – holdout method
k-Fold Cross-Validation – robust evaluation
Stratified CV – preserve class proportions
Time-Series CV – rolling or expanding windows

3) Hyperparameter Tuning

Grid Search

Exhaustive search over combinations
Simple but expensive

Random Search

Samples random combinations
Faster, often effective

Bayesian Optimization

Models the objective function
Efficient with fewer trials

4) Deployment

Packaging

Pickle/Joblib for model serialization
Export to ONNX/TF-Lite for portability

Serving

Flask/FastAPI endpoints
Batch vs Real-time inference
Monitor latency, throughput, errors

5) Monitoring in Production

Drift Detection – input distribution shift
Performance Decay – evaluate with fresh labels
Logging & Retraining pipelines

6) Ethics in Machine Learning

Bias & Fairness

Bias from skewed data or features
Fairness metrics: demographic parity, equal opportunity

Transparency & Interpretability

SHAP, LIME for local explanations
Feature importance plots

Privacy

Differential privacy & federated learning
Secure data handling & compliance (GDPR, HIPAA)

Accountability

Human-in-the-loop decision systems
Auditability & documentation (Model Cards, Datasheets)

7) Practical Workflow Example

# Hyperparameter tuning with CV (pseudocode)
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

param_grid = {"n_estimators": [100,200,500],
              "max_depth": [5,10,None],
              "min_samples_split": [2,5,10]}

rf = RandomForestClassifier(class_weight="balanced")
cv = GridSearchCV(rf, param_grid, cv=5, scoring="f1")
cv.fit(X_train, y_train)
print(cv.best_params_, cv.best_score_)

8) Exercise

Hands-on Challenge: Tune hyperparameters for Logistic Regression, RandomForest, and XGBoost on the same dataset. Compare evaluation metrics (F1, ROC-AUC, PR-AUC). Deploy the best model as a FastAPI service and simulate drift by changing test distribution.