· 10 min deep-divemachine-learningensemble
Article 1 in your session
Gradient Boosting & XGBoost — The Kaggle King
A scroll-driven visual deep dive into gradient boosting. Learn how weak learners combine sequentially, how XGBoost optimizes the process, and why it dominates tabular ML competitions.
Introduction
🎯 0/3 0%
Ensemble Methods · Sequential
Each tree fixes the mistakes
of the one before it.
Random Forest trains trees independently. Gradient Boosting trains them sequentially — each tree learns from the errors of the previous ensemble. This greedy, additive strategy wins over 70% of ML competitions on tabular data.
Fitting Residuals
Step 1: Learn the Errors
Gradient Boosting: additive model
1
F₀(x) = ȳ Start with the simplest prediction: the mean of training labels
2
F₁(x) = F₀(x) + η · h₁(x) Add a small tree h₁ fitted to the residuals (y - F₀(x))
3
Fₘ(x) = Fₘ₋₁(x) + η · hₘ(x) Each step adds a tree fitted to current residuals, scaled by learning rate η
4
η (learning rate) is typically 0.01 − 0.3 Small η = more trees needed but better generalization. Classic bias-variance tradeoff
↑ Answer the question above to continue ↑
🟢 Quick Check Knowledge Check
Why do we multiply each tree's prediction by a learning rate η < 1?
💡 What happens if each tree tries to fix ALL remaining error at once?
Without a learning rate (η = 1), each tree tries to fully correct all remaining errors in one step. This leads to overfitting because later trees memorize noise. With η = 0.1, each tree only corrects 10% of the residuals, requiring more trees but making the ensemble more robust. Think of it as taking small, careful steps toward the optimum instead of large, reckless jumps.
Gradient View
The ‘Gradient’ in Gradient Boosting
Gradient descent in function space
1
L(y, F) = ½(y − F(x))² For MSE loss...
2
−∂L/∂F = y − F(x) = residuals The negative gradient of MSE is exactly the residuals!
3
For log-loss: −∂L/∂F = y − p(x) For classification, the gradient is (true label − predicted probability)
4
hₘ(x) ≈ −∂L/∂F Each tree approximates the negative gradient → steepest descent in function space
↑ Answer the question above to continue ↑
🟡 Checkpoint Knowledge Check
For MSE loss, fitting a tree to residuals is equivalent to fitting a tree to negative gradients. For cross-entropy loss, what are the 'pseudo-residuals'?
💡 Take the derivative of cross-entropy loss with respect to the function output F(x)...
For cross-entropy (log-loss), L = −[y·log(p) + (1−y)·log(1−p)] where p = sigmoid(F(x)). The negative gradient with respect to F is y − p: the difference between the true label (0 or 1) and the predicted probability. This has a beautiful interpretation: if the model predicts p = 0.9 for a positive sample (y = 1), the pseudo-residual is 0.1 — there's little left to learn. If it predicts p = 0.1 for a positive sample, the pseudo-residual is 0.9 — lots to learn!
XGBoost
XGBoost: Gradient Boosting, Engineered
XGBoost's regularized objective
1
Obj = Σ L(yᵢ, ŷᵢ) + Σ Ω(fₖ) Loss + regularization over all trees
2
Ω(f) = γT + ½λ||w||² T = #leaves (penalizes complex trees), w = leaf weights (L2 penalty)
3
Uses 2nd-order Taylor expansion of loss Needs both gradient gᵢ and hessian hᵢ → more precise split finding
4
Gain = ½[G²_L/(H_L+λ) + G²_R/(H_R+λ) − (G_L+G_R)²/(H_L+H_R+λ)] − γ Split gain formula: if gain < 0, don't split → built-in pruning
Key Parameters
The Parameters That Matter
↑ Answer the question above to continue ↑
🔴 Challenge Knowledge Check
In XGBoost, individual trees are typically much shallower (depth 3-6) than in Random Forests (often grown to full depth). Why?
💡 Random Forest reduces variance (average strong learners). Boosting reduces... what? And how?
This is fundamental to boosting: it reduces BIAS by sequentially adding weak learners. Each tree should be a simple 'stump' or shallow tree that captures one pattern. Deep trees would overfit to residuals (memorize noise). The ensemble gains complexity gradually through addition. In contrast, Random Forest reduces VARIANCE by averaging independent strong learners — so deep trees are fine because variance averages out.
RF vs GB
Random Forest vs Gradient Boosting
🎓 What You Now Know
✓ Boosting fits residuals sequentially — Each tree corrects the errors of the ensemble so far.
✓ It’s gradient descent in function space — Residuals = negative gradient of the loss.
✓ XGBoost adds regularization + 2nd-order optimization — Penalizes tree complexity, uses hessians.
✓ Key: learning rate + early stopping — Small η, many shallow trees, stop when validation plateaus.
✓ Wins competitions, but needs tuning — Not as “set-and-forget” as Random Forest.
XGBoost is the tool that turned “good” into “winning” for tabular machine learning. But it’s also a tool that rewards understanding: the better you understand the bias-variance tradeoff, the better you’ll tune it. 🏆
📄 XGBoost: A Scalable Tree Boosting System (Chen & Guestrin, 2016)
↗ Keep Learning
deep-divemachine-learningclassification
Decision Trees — How Machines Learn to Ask Questions
A scroll-driven visual deep dive into decision trees. Learn how trees split data, what Gini impurity and information gain mean, and why trees overfit like crazy.
13 min
Read →
deep-divemachine-learningensemble
Random Forests — Why 1000 Bad Models Beat 1 Good One
A scroll-driven visual deep dive into Random Forests. Learn bagging, feature randomness, out-of-bag error, and why ensembles are the most reliable ML technique.
14 min
Read →
deep-divemachine-learningensemble
Bagging vs Boosting — The Two Philosophies of Ensemble Learning
A scroll-driven visual deep dive comparing bagging and boosting. Learn when to average independent models vs sequentially correct errors, and why ensembles dominate ML.
12 min
Read →
Comments
No comments yet. Be the first!