Lesson 5 of 10
Evaluate a model before relying on it
Separate examples into training data for fitting, validation data for selecting settings, and test data for a final unbiased check. Keep related records together when a random split would put near-duplicates in both training and test sets. For time-based predictions, train on earlier periods and evaluate on later ones.
Accuracy is the share of predictions that are correct, but it can mislead when classes are imbalanced. Precision asks how many positive predictions were correct; recall asks how many actual positive examples were found. For regression, mean absolute error reports the average absolute prediction error in the target's units.
Compare against a simple baseline, inspect errors by meaningful groups, and choose a threshold based on the cost of false positives and false negatives.
Try it: In a system that flags urgent support requests, which error is more costly: missing a truly urgent request or escalating a normal one? Explain what you would measure.