How to Prune a Tree in R?

Pruning a decision tree in R involves reducing its size by removing sections that do not provide significant improvements in predictive accuracy. Decision trees are particularly intuitive and easy to interpret, but they can often grow too complex, leading to overfitting.

What is Pruning?

Pruning is a technique used in machine learning, particularly in decision tree algorithms, to simplify the model by reducing its size and complexity. The main goals of pruning are to enhance the model's generalization ability, prevent overfitting, and improve interpretability. Pruning removes parts of the tree that provide little to no additional power in predicting target variables, ultimately resulting in a more robust and manageable model.

Understanding the Need for Pruning

Decision trees can grow very deep, leading to complex models that perfectly fit the training data but perform poorly on new, unseen data due to overfitting. Pruning helps to reduce this complexity by removing sections of the tree that provide little power in predicting target variables.

Creating a Decision Tree for Prune

First, let's create a decision tree using the popular rpart package in R. We'll use the built-in iris dataset for this example.

# Load necessary libraries
library(rpart)
library(rpart.plot)

# Load the iris dataset
data(iris)

# Create a decision tree model
set.seed(123)
tree_model <- rpart(Species ~ ., data = iris, method = "class")

# Plot the decision tree
rpart.plot(tree_model)

Output:

First we Load the rpart and rpart.plot libraries for creating and visualizing decision trees.

• Load the built-in iris dataset.
• Set a seed for reproducibility.
• Create a decision tree model to classify iris species using the rpart function.
• Plot the decision tree using rpart.plot for a visual representation of the model.

Pruning in the rpart package can be done using the cp (complexity parameter) value. The complexity parameter is a measure of the cost-complexity of the tree, where a lower cp value indicates a less complex tree.

Choosing the Complexity Parameter (cp)

We can visualize the cost-complexity parameter using the plotcp function to determine an optimal cp value.

# Plot the complexity parameter
plotcp(tree_model)

Output:

The plot shows the relationship between the cross-validated error and the complexity parameter. The optimal cp value is typically chosen where the error is minimized.

Pruning the Tree

Once we identify the optimal cp value, we can prune the tree using the prune function.

# Get the optimal cp value
optimal_cp <- tree_model$cptable[which.min(tree_model$cptable[,"xerror"]), "CP"]

# Prune the tree
pruned_tree <- prune(tree_model, cp = optimal_cp)

# Plot the pruned tree
rpart.plot(pruned_tree)

Output:

After pruning, it's important to evaluate the pruned tree's performance to ensure it generalizes well to new data. We can use a confusion matrix to compare the predictions of the pruned tree to the actual values.

# Predict on the training data
predictions <- predict(pruned_tree, iris, type = "class")

# Create a confusion matrix
confusion_matrix <- table(iris$Species, predictions)
print(confusion_matrix)

Output:

            predictions
             setosa versicolor virginica
  setosa         50          0         0
  versicolor      0         49         1
  virginica       0          5        45

The confusion matrix provides a summary of prediction results, showing the counts of true positives, false positives, true negatives, and false negatives.

Conclusion

Pruning a decision tree is a crucial step in creating robust models that generalize well to new data. By following these steps in R, you can efficiently prune your decision trees and improve their performance.