How to Calculate Entropy in Decision Tree?

In decision tree algorithms, entropy is a critical measure used to evaluate the impurity or uncertainty within a dataset. By understanding and calculating entropy, you can determine how to split data into more homogenous subsets, ultimately building a better decision tree that leads to accurate predictions. Concept of entropy originates from information theory, where it quantifies the amount of "surprise" or unpredictability in a set of data.

Understanding Entropy

Entropy is a measure of uncertainty or disorder. In the terms of decision trees, it helps us understand how mixed the data is. If all instances in a dataset belong to one class, entropy is zero, meaning the data is perfectly pure. On the other hand, when the data is evenly distributed across multiple classes, entropy is maximum, indicating high uncertainty.

• High Entropy: Dataset has a mix of classes, meaning it's uncertain and impure.
• Low Entropy: Dataset is homogeneous, with most of the data points belonging to one class.

Entropy helps in choosing which feature to split on at each decision node in the tree. Goal is to reduce entropy with each split to create subsets that are as pure as possible.

Lets understand how we can calculate Entropy:

To calculate entropy, we need to use the following formula:

Entropy(S) = - \sum_{i=1}^{n} p_i \log_2 p_i

Where:

• S is the dataset (set of data points).
• pi​ is the probability of class i in the dataset.
• n is the number of unique classes in the dataset.

Steps to Calculate Entropy:

1. Find the Probability of Each Class: Calculate the proportion of each class in the dataset. For example, if we have a dataset with 10 data points and 6 of them are cats and 4 are dogs, probabilities would be:

• p(\text{cat}) = \frac{6}{10} = 0.6
• p(\text{dog}) = \frac{4}{10} = 0.4

2. Apply the Entropy Formula: Entropy for this dataset is calculated by plugging these probabilities into the entropy formula. Formula becomes:

Entropy(S) = - \left( 0.6 \times \log_2 0.6 + 0.4 \times \log_2 0.4 \right)

3. Compute the Logarithms: We can compute the logarithmic values for each probability:

• \log_2 0.6 \approx -0.737
• \log_2 0.4 \approx -1.322

4. Calculate Final Entropy: Now multiply each probability by its respective log value and sum the results:

Entropy(S) = - \left( 0.6 \times -0.737 + 0.4 \times -1.322 \right)

This results in an entropy value of approximately 0.971, which reflects the uncertainty of the dataset.

By mastering the concept of entropy, you’ll be equipped to build more accurate decision trees and improve your machine learning models with a deeper understanding of data purity and uncertainty.