Most decision tree algorithms, including those used in standard Gradient Boosting implementations and XGBoost, grow trees level-wise (or depth-wise). In this strategy, the tree expands layer by layer. All nodes at a given depth are split before moving to the next deeper level. This approach maintains a balanced tree structure, which can be easier to manage and less prone to immediate overfitting.

LightGBM, however, adopts a different strategy by default: leaf-wise (or best-first) tree growth. Instead of expanding level by level, the leaf-wise strategy focuses on finding the single leaf node, anywhere in the current tree structure, that will yield the largest reduction in the loss function when split. It then splits that specific leaf. This process repeats: find the leaf with the maximum potential loss reduction and split it.

Comparing Level-Wise and Leaf-Wise Growth

Imagine building a tree.

Level-wise: You find the best split for the root node. Then, you find the best splits for both resulting children. Then, you find the best splits for all four nodes at the next level, and so on. You build the tree horizontally, one complete layer at a time.
Leaf-wise: You find the best split for the root node. Now you have two leaves. You evaluate the potential loss reduction from splitting either of these leaves. Let's say splitting the left leaf gives a greater loss reduction than splitting the right leaf. You split the left leaf. Now you have three leaves. You again evaluate the potential gain from splitting any of these three leaves and split the one that promises the best improvement. The tree tends to grow deeper down one path before exploring others.

Example splitting order difference between level-wise and leaf-wise growth. Leaf-wise prioritizes splitting the node that yields the largest loss reduction, often leading to asymmetric trees initially.

Advantages of Leaf-Wise Growth

The primary advantage of the leaf-wise strategy is faster convergence. By always splitting the node that provides the largest immediate improvement (reduction in loss), the model can potentially reach a lower loss value with fewer splits compared to the level-wise approach. This is because it doesn't waste computation splitting nodes that offer minimal gain just to complete a level. For large datasets, this can translate into significantly faster training times and potentially more accurate models for the same number of leaves, as the algorithm focuses its efforts where they matter most.

Combined with histogram-based split finding, which quickly evaluates potential splits, leaf-wise growth allows LightGBM to construct effective trees rapidly.

Disadvantages and Regularization

The main drawback of leaf-wise growth is its tendency to produce deeper, potentially unbalanced trees and its increased risk of overfitting, especially on smaller datasets. Since it greedily pursues the path of maximum loss reduction, it might create very deep branches based on splits that fit the training data noise exceptionally well, before exploring other parts of the feature space.

To combat this, LightGBM relies heavily on specific regularization parameters:

num_leaves: This is the primary parameter used to control the complexity of the tree model in LightGBM when using leaf-wise growth. It directly limits the total number of leaf nodes in each tree. A smaller num_leaves value results in simpler trees and helps prevent overfitting. This is often more effective than max_depth for controlling complexity in a leaf-wise strategy.
max_depth: While still available, max_depth acts as a safeguard. It puts an explicit limit on the tree depth, preventing individual branches from becoming excessively deep, even if num_leaves hasn't been reached. Setting max_depth can sometimes help prevent overfitting that num_leaves alone might not catch, but tuning num_leaves is typically the first step.
min_child_samples (or min_data_in_leaf): This parameter specifies the minimum number of data points required in a leaf node. Splits that would result in a leaf with fewer samples than this threshold are rejected, effectively preventing the model from fitting noise on very small groups of data points.

Effectively tuning num_leaves is essential when using LightGBM. The default value (31) is often a good starting point, but it frequently requires adjustment based on the dataset size and complexity.

In summary, leaf-wise growth is a significant factor contributing to LightGBM's speed and efficiency. It allows the model to converge quickly by focusing on the most promising splits. However, this aggressive strategy necessitates careful regularization, primarily through the num_leaves parameter, to avoid overfitting and ensure good generalization performance.