How does Node splits in Decision Tree for Classification and Regression

Part 1: Decision Tree – Classification

How does a node split happen?

In classification trees, node splitting is done based on purity. The goal is to reduce impurity (like Gini or Entropy) as much as possible at each split.

Gini Impurity Formula:

Gini=1−∑i=1Cpi2Gini = 1 – \sum_{i=1}^{C} p_i^2Gini=1−i=1∑C​pi2​

Where pip_ipi​ is the proportion of class iii in the node.

Sample Data (Classification)

We will try to split based on X.

Try a Split: X <= 5

• Left node: X ≤ 5 → IDs 1, 2 → [A, A]
• Right node: X > 5 → IDs 3, 4, 5 → [B, B, B]

Best Split: This gives 0 total impurity, so it’s the best possible split.

Try another Split: X <= 15

• Left node: X ≤ 15 → IDs 1, 2, 3 → [A, A, B]
• Right node: X > 15 → IDs 4, 5 → [B, B]

Higher than 0 → worse than first split.

Part 2: Decision Tree – Regression

How does a node split happen?

In regression trees, splitting is done to minimize variance (or MSE – Mean Squared Error) of the target variable.

Sample Data (Regression)

Try a Split: X <= 5

• Left Node (X ≤ 5): IDs 1, 2 → [5, 6]
• Right Node (X > 5): IDs 3, 4, 5 → [13, 25, 26]

Try a Better Split: X <= 10

• Left Node: IDs 1, 2, 3 → [5, 6, 13]
• Right Node: IDs 4, 5 → [25, 26]

Much lower than 21.06 – better split.

Summary: