The most fundamental challenge in ML: building a model that generalizes to new data rather than memorizing training data.

Overfitting

Model memorizes training data — performs great on training set, poorly on test set. Too complex for the data.

Underfitting

Model too simple to capture patterns — performs poorly on both training and test sets.

Solutions for Overfitting

• More training data
• Regularization (L1/L2)
• Dropout (neural networks)
• Simpler model
• Cross-validation

The Bias-Variance Tradeoff

High bias = underfitting. High variance = overfitting. The goal is the sweet spot of low bias and low variance.

Reference:

Overfitting and generalization

https://developers.google.com/machine-learning/crash-course/generalization/peril-of-overfitting