Beyond R-squared, which primarily indicates explained variance, evaluating regression models effectively requires examining metrics like Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Mean Absolute Percentage Error (MAPE) to understand the magnitude and nature of prediction errors. These metrics offer different perspectives on model performance, crucial for robust assessment.
Here's a breakdown of these key regression metrics:
1. Root Mean Squared Error (RMSE)
Formula: sqrt(sum((y_true - y_pred)^2) / n)
Interpretation: RMSE measures the average magnitude of the errors. Because it squares the errors before averaging, it gives a relatively high weight to large errors. The result is in the same units as the target variable.
Use Case: Use RMSE when large errors are particularly undesirable or costly, as it penalizes them more heavily.
2. Mean Absolute Error (MAE)
Formula: sum(|y_true - y_pred|) / n
Interpretation: MAE measures the average magnitude of the errors without considering their direction. All errors contribute equally to the metric. Like RMSE, it's in the same units as the target variable.
Use Case: MAE is more robust to outliers than RMSE. Use it when you want to treat all errors equally, regardless of their magnitude.
3. Mean Absolute Percentage Error (MAPE)
Formula: (1/n) sum(|(y_true - y_pred) / y_true|) 100%
Interpretation: MAPE expresses the error as a percentage of the actual value. This makes it scale-independent, useful for comparing models across different datasets or target variables with varying scales. Be cautious when y_true values are zero or very close to zero, as MAPE can become undefined or extremely large.
Use Case: When you need a relative error metric, especially for forecasting or when comparing performance across products with different price points.
Metric Comparison:
| Metric | Sensitivity to Outliers | Units | Interpretation | Best Use Case |
| :----- | :---------------------- | :------------- | :---------------------------------------------- | :------------------------------------------------- |
| RMSE | High | Same as target | Penalizes large errors more | When large errors are costly |
| MAE | Low | Same as target | Treats all errors equally | When robustness to outliers is important |
| MAPE | Medium (can be extreme) | Percentage | Relative error, scale-independent | Comparing models across different scales; forecasting |
Guidance for Metric Selection:
Prioritize RMSE if your application incurs disproportionately higher costs or risks from large prediction errors.
Choose MAE when you want a straightforward average error and robustness against outliers is important.
Opt for MAPE when you need a scale-independent metric to compare models across different contexts or when the relative error is more meaningful. Always check for zero or near-zero actual values in your data before relying on MAPE.
* Always report multiple metrics. A significant difference between RMSE and MAE often indicates the presence of outliers or a few predictions with very large errors.
Python Example using sklearn.metrics:
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error, mean_absolute_percentage_error
# Example true and predicted values
y_true = np.array([10, 20, 30, 40, 50])
y_pred = np.array([12, 18, 32, 38, 55])
# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_true, y_pred))
print(f"RMSE: {rmse:.2f}")
# Calculate MAE
mae = mean_absolute_error(y_true, y_pred)
print(f"MAE: {mae:.2f}")
# Calculate MAPE (sklearn's MAPE returns a fraction, multiply by 100 for percentage)
mape = mean_absolute_percentage_error(y_true, y_pred) * 100
print(f"MAPE: {mape:.2f}%")
# Example with an outlier to show RMSE sensitivity
y_true_outlier = np.array([10, 20, 30, 40, 100])
y_pred_outlier = np.array([12, 18, 32, 38, 5]) # Large error here
rmse_outlier = np.sqrt(mean_squared_error(y_true_outlier, y_pred_outlier))
mae_outlier = mean_absolute_error(y_true_outlier, y_pred_outlier)
print(f"\nWith outlier:")
print(f"RMSE (outlier): {rmse_outlier:.2f}")
print(f"MAE (outlier): {mae_outlier:.2f}")