It is vain to do with more what can be done with few
Occam’s razor can be successfully extended to data science, when choosing the most suitable model for your data. See for example variable selection or pruning which are model complexity reduction techniques.
How much of the model’s complexity can you “shave away” safely, without harming the predictive power of your model?
A model that is overly simplified, won’t capture the relevant signal or patterns in the data, resulting in large errors at train and test time, so poor overall performance. On the other hand,
A model that is too complex will mistakenly take noise for relevant signal, fit the training data perfectly and fail to predict similarly on unseen data.
Giving away complexity introduces bias
Bias, along with variance are the two major sources of error in machine learning models and they go hand in hand.
Bias is the error coming from fitting a too simple model for the data, unable to capture the relevant patterns in the data. This leads to a phenomenon called underfitting, indicating that the model’s complexity is below the relationships’ complexity in the data. High bias will result in poor overall performance at both train and test time.
Variance is the error coming from fitting a model that is too complex, sensitive to the small fluctuations in the data, to the point where the signal is lost, covered in the “details” of the dataset. The corresponding phenomenon is called overfitting indicating that the model fits the training dataset too closely.
Generalization power is the key
100% accuracy on the training dataset is useless. We want models able to generalize well on unseen data, models with high predictive power, so we will, on purpose, trade variance for bias. We will give away model features for predictive power.
The term ‘trade-off’ underlines that a compromise must be reached between the model’s complexity and its generalization capacity.
When you intentionally simplify the model by:
- moving from a high-degree polynomial to a linear model, or
- reduce the number of variables in linear regression),
you reduce the variance but increase the bias. This increase in bias means that the model can no longer capture all of the complex relationships in the data.
The model simplifies the relationships in the data, and as a result, it ignores some of the variability that a more complex model would have been able to explain.
This leftover variability or systematic error becomes bias in the model’s predictions.
Final thoughts
Understanding the bias-variance tradeoff and comparing the performance metrics at train and validation time, helps address underfitting and overfitting by:
- Increasing the model’s complexity by adding more variables
- Or choosing a more complex model architecture
as well as,
- Perform variable selection or choose a less complex
model architecture.
At Giraffa Analytics, we prioritize building reliable AI solutions by leveraging our mix of hands-on experience and the field’s best practices.