Handle missing data and outliers with deterministic imputation, robust statistical filters, and automated validation to keep downstream metrics unbiased.
Step‑by‑step workflow
1. Profile the raw table using pandas_profiling.ProfileReport(df) or Great Expectations suites; capture %null, %unique, and distribution plots.
2. Missing‑value policy
- If null_rate > 0.05 → drop column.
- If 0 < null_rate ≤ 0.05 → impute: numeric → sklearn.impute.IterativeImputer (max_iter=10, random_state=42); categorical → SimpleImputer(strategy='most_frequent').
3. Outlier detection
```python
from sklearn.ensemble import IsolationForest
iso = IsolationForest(contamination=0.01, random_state=0)
outlier_mask = iso.fit_predict(df[numeric_cols]) == -1
```
or classic IQR: Q1, Q3 = df[col].quantile([0.25,0.75]); IQR = Q3-Q1; lower, upper = Q1-1.5IQR, Q3+1.5IQR.
4. Mitigation – apply winsorization at the 1st/99th percentile or log‑transform heavy‑tailed fields: df[col] = np.log1p(df[col].clip(lower, upper)).
5. Metric calculation – replace mean/STD with median/MAD or use np.nanmedian and stats.median_abs_deviation. For KPI aggregates, weight rows by 1/(1+outlier_score).
6. Validation – run Kolmogorov‑Smirnov test between original and cleaned distributions; log drift >0.02 as a warning.
Quick comparison
| Method | When to use | Bias risk |
|--------|-------------|-----------|
| Mean imputation | ≤2 % missing, normal | High |
| Iterative imputer | 2‑5 % missing, correlated vars | Low |
| Median / MAD | Skewed metrics | Very low |
| IsolationForest | Multivariate outliers | Low |
Automate the pipeline in Airflow or Prefect, and version‑control the validation suite with Great Expectations to guarantee reproducibility.