The Evolution of Tree-Based Algorithms

This chapter provides an overview of the evolution of tree-based algorithms. In addition to the Summary, there are six sections. Section 11.1 introduces the bagging and boosting algorithms and briefly compares and discusses the challenges of these two algorithms. Section 1.2 introduces gradient boosting, including function estimation, numerical optimization in parameter space and in function space with particular descriptions of stagewise additive expansions, and steepest-descent minimization. This section also describes the steps of implementing the gradient boosting algorithm and presents the gradient boosting least squares regression as an example of gradient boosting. Section 11.3 introduces gradient tree boosting, including the steps of implementing the gradient tree boosting algorithm as well as the mechanisms for preventing overfitting in gradient tree boosting (such as tree depth and tree size, learning rate, and randomization). Section 11.4 introduces Newton boosting and Newton tree boosting. Section 11.5 briefly summarizes the evolution of tree-based algorithms from decision trees to extreme gradient boosting (XGBoost), including the limitation of gradient tree boosting. Section 11.6 provides some remarks on boosting methods.