Bagging vs Boosting
| Bagging | Boosting | |
|---|---|---|
| 1.. | Base estimators Runs in Parallel | Base estimators Runs in Sequence |
| 2.. | Base estimators are Independent of each other | Base estimators are Dependent of each other |
| 3.. | Individual Trees are Deep Trees that can overfit | Individual Trees are Stumps i.e. Max-Depth=1 |
| 4.. | Each Tree have equal importance | Some Trees have more importance or Amount of Say as compared to others |
| 5.. | Needs more CPU cores and RAM | Needs less CPU cores (because its sequential) and less RAM (because of stumps) |
| 6.. | Examples – Random Forest, ExtraTrees Classifier | Examples – AdaBoost, GBM, XGBoost |
| AdaBoost | GBM | |
|---|---|---|
| 1.. | Adaptive Boosting uses SAMME or SAMME.R Algorithm | Gradient Boosting uses Gradient Descent Algorithm for Convergence |
| 2.. | Some Trees have more importance or Amount of Say as compared to others | No mention of importance or Amount of Say at individual tree level |
| 3.. | No mention of learning rate here | Learning Rate plays a important role here |
| 4.. | Sample weights are adjusted at each iteration | Sample weights are NOT adjusted at each iteration |
| 5.. | Each tree predicts for actual output | Each tree predicts for Residual value except the first one that uses Averaging |
Leave a Reply