· 14 min deep-divemachine-learningfundamentals
Article 1 in your session
Feature Engineering — The Art That Makes or Breaks Your Model
A scroll-driven visual deep dive into feature engineering. Learn transformations, encoding, interaction features, handling missing data, and why feature engineering matters more than model choice.
Introduction
🎯 0/3 0%
Data Preparation
Better data beats
better algorithms.
Feature engineering transforms raw data into features that make ML models more powerful. The top Kaggle winners spend 80% of their time on features, 20% on models. A simple model with great features outperforms a complex model with raw data. Every time.
Encoding
Encoding Categorical Variables
↑ Answer the question above to continue ↑
🟢 Quick Check Knowledge Check
You have a 'city' column with 10,000 unique cities. One-hot encoding would create 10,000 new columns. What's the better approach?
💡 What encoding creates just ONE column from 10,000 categories while adding predictive signal?
10,000 one-hot columns = massive sparse matrix, slow training, and likely overfitting. Label encoding implies a false ordering (city 5000 isn't 'more' than city 1). Target encoding replaces each city with a single number (mean target for that city), adding predictive signal without dimension explosion. The key: compute target means using cross-validation folds to avoid target leakage, where the encoding would 'memorize' the training labels.
Transforms
Numerical Transformations
Common feature transforms
Log Transform
Compresses right-skewed distributions like income or price by squashing the long tail. Makes relationships more linear and reduces the outsized influence of extreme values.
x → log(1 + x) Standardization
Centers data to zero mean and scales to unit variance. Required for distance-based models (KNN, SVM), PCA, and any model using gradient descent or regularization.
x → (x − μ) / σ Min-Max Scaling
Rescales features to a fixed [0, 1] range. Good for neural networks and algorithms sensitive to feature magnitude, but outliers compress the useful range.
x → (x − min) / (max − min) Power Transforms
Box-Cox and Yeo-Johnson automatically find the optimal power transformation to make data more Gaussian. Use sklearn's PowerTransformer when you don't know which transform to pick.
↑ Answer the question above to continue ↑
🟡 Checkpoint Knowledge Check
Why does log-transforming a right-skewed feature (e.g., income) improve linear regression?
💡 What happens when 99% of values are 0-100K but one outlier is 20M?
Income might range from $20K to $20M — the richest person has 1000x the poorest. A linear model would be dominated by the extreme values (high leverage). log(income) compresses this: log(20K) ≈ 10, log(20M) ≈ 17 — now the range is only 1.7x. The relationship between log(income) and the target is often much more linear than income vs target. This is why economists, epidemiologists, and data scientists log-transform skewed variables by default.
Interaction Features
Creating New Features from Existing Ones
Feature interactions and domain features
Interaction Features
Multiply two features together to capture effects that depend on BOTH variables. For example, bedrooms × bathrooms creates a house quality proxy that neither feature captures alone.
x₁ × x₂ Ratio Features
Dividing one feature by another often yields more meaningful signals than raw values. Price per square foot, clicks per impression (CTR), and revenue per user are classic examples.
x₁ / x₂ Polynomial Features
Squaring, cubing, or taking roots of features captures nonlinear relationships. For example, age² models the U-shaped relationship between age and income.
x², x³, √x Date Features
Raw timestamps are useless to models. Extract year, month, day_of_week, is_weekend, days_since_event — these temporal patterns are where the predictive power lives.
Missing Data
Handling Missing Values
↑ Answer the question above to continue ↑
🔴 Challenge Knowledge Check
You compute mean imputation on the FULL dataset (train + test) before splitting. What's wrong?
💡 Should your training process 'see' any information from the test set?
If you compute the mean on the full dataset, information from the test set 'leaks' into the training process through the imputed values. This gives optimistically biased evaluation. The correct pipeline: (1) split data, (2) fit imputer on training set only, (3) transform both train and test using the training-set statistics. This is why sklearn's Pipeline is essential — it ensures proper fit/transform ordering in cross-validation.
Feature Selection
Less Can Be More
🎓 What You Now Know
✓ One-hot encode nominal, label encode ordinal — Target encoding for high cardinality.
✓ Log-transform skewed features — Standardize for distance/gradient-based models.
✓ Create interactions and ratios — Domain knowledge encoded as features beats model complexity.
✓ Impute properly: train stats only — Add “is_missing” indicators for free signal.
✓ Features > algorithms — Time spent on features gives better ROI than model tuning.
Feature engineering is where science meets art. The science: mathematical transformations, proper encoding, statistical imputation. The art: knowing which features to create from domain expertise. Master both, and you’ll outperform any AutoML tool. 🎨
↗ Keep Learning
deep-divemachine-learningregression
Polynomial Regression — When Lines Aren't Enough
A scroll-driven visual deep dive into polynomial regression. See why straight lines fail, how curves capture nonlinear patterns, and when you're overfitting vs underfitting.
12 min
Read →
deep-divemachine-learningunsupervised
PCA — Compressing Reality Without Losing the Plot
A scroll-driven visual deep dive into Principal Component Analysis. Learn eigenvectors, variance maximization, dimensionality reduction, and when PCA transforms your data — and when it doesn't.
9 min
Read →
deep-divemachine-learningmath
Linear Regression — The Foundation of Machine Learning
A scroll-driven visual deep dive into linear regression. From data points to loss functions to gradient descent — understand the building block behind all of ML.
12 min
Read →
Comments
No comments yet. Be the first!