Ensemble Models

When we want to use more than 1 ML model for our predictions, we use Ensemble models. For example, we can use 10 KNN models together inside an ensemble model. 

These are also known as Meta Models/Estimators

They tend to avoid overfitting by lowering the Overall Variance

• Types of Ensemble Models:
  1. Bagging Models
     • Random Forest
     • ExtraTree Classifier/Regressor
  2. Boosting Models
     • AdaBoost (Adaptive Boosting)
     • GBM (Gradient Boosting Machine)
     • XGBoost (Xtreme Gradient Boosting)
     • LightGBM
     • CatBoost
  3. Stacking Models

---

BAGGING MODELS

1. All Base Models like DTrees runs in Parallel
2. Base Models are independent of each other
3. All Base Models have equal say in prediction

Bagging Classifier : A Bagging classifier is an ensemble meta-estimator that fits base classifiers each on random subsets of the original dataset and then aggregate their individual predictions (either by voting or by averaging) to form a final prediction. 

class sklearn.ensemble.BaggingClassifier(
base_estimator=None, 
n_estimators=10, *, 
max_samples=1.0, max_features=1.0, 
bootstrap=True, bootstrap_features=False, 
oob_score=False, 
warm_start=False, n_jobs=None, 
random_state=None, verbose=0
)

RandomForest Classifier : A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. 

class sklearn.ensemble.RandomForestClassifier(
n_estimators=100, *, 
criterion='gini', 
max_depth=None, 
min_samples_split=2, min_samples_leaf=1, 
min_weight_fraction_leaf=0.0, 
max_features='sqrt', max_leaf_nodes=None, 
min_impurity_decrease=0.0, 
bootstrap=True, oob_score=False, 
n_jobs=None, random_state=None, verbose=0, 
warm_start=False, class_weight=None, 
ccp_alpha=0.0, max_samples=None
)

ExtraTrees Classifier: This class implements a meta estimator that fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.

class sklearn.ensemble.ExtraTreesClassifier(
n_estimators=100, *, 
criterion='gini', 
max_depth=None, 
min_samples_split=2, min_samples_leaf=1, 
min_weight_fraction_leaf=0.0, 
max_features='sqrt', max_leaf_nodes=None, 
min_impurity_decrease=0.0, 
bootstrap=False, oob_score=False, 
n_jobs=None, random_state=None, verbose=0, 
warm_start=False, class_weight=None, 
ccp_alpha=0.0, max_samples=None
)

---

BOOSTING MODELS

• Base Models like DTrees runs in Sequence, and each one is known as weak learner
• Base Models are dependent of each other i.e. previous models adjusted weights of samples to decide the data subset created for next model
• All Base Models don’t have equal say in prediction i.e. some models have more importance over others during the final prediction

from sklearn.ensemble import AdaBoostClassifier, AdaBoostRegressor

ada = AdaBoostClassifier(n_estimators=800)

from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier

gbm = GradientBoostingClassifier(verbose=True, n_estimators=800)

from xgboost import XGBClassifier 
import numpy as np

nan=np.nan

xgb = XGBClassifier(
              base_score=0.5, booster='dart', 
              colsample_bylevel=1,                  
              colsample_bynode=1, colsample_bytree=1, 
              gamma=0, gpu_id=-1,                
              importance_type='gain', 
              interaction_constraints='',                  
              learning_rate=0.1, #eta
              max_delta_step=0, max_depth=6,     
              min_child_weight=1, 
              missing=np.nan, 
              monotone_constraints='()',                  
              n_estimators=100, 
              n_jobs=0, 
              num_parallel_tree=1,                  
              objective='binary:logistic', 
              random_state=0,
              reg_alpha=1, #l1                  
              reg_lambda=0, #l2
              scale_pos_weight=None, subsample=0.8,                  
              tree_method='exact', validate_parameters=1, 
              verbosity=3
              )

---

STACKING MODELS

• Here we have multiple layers of models i.e. layer 1 models feed their output to layer 2 models for prediction on layer 1 output

---

Refer here for Jupyter Notebooks on Bagging and Boosting

---

Sources: https://scikit-learn.org/stable/modules/classes.html#module-sklearn.ensemble

---