Is multiple linear regression invariant to scaling?
Yes, multiple linear regression is invariant to scaling of the input variables, but not necessarily to the scaling of the output variable. Let's break this down:
-
Scaling Input Variables (Features):
When you scale the input variables (features) in a multiple linear regression model, the model's predictive power remains the same. However, the coefficients associated with those scaled variables will change. This is because the coefficients reflect the change in the dependent variable for a unit change in the independent variable. If you change the scale of the independent variable, the corresponding coefficient must adjust to maintain the same relationship.
Here's why the overall model remains invariant:
- The model will adjust the coefficients to account for the change in scale.
- The predictions made by the model will be identical before and after scaling (provided you appropriately transform the scaled variables back to their original scale if needed for interpretation).
-
Scaling the Output Variable:
If you scale the output variable (dependent variable), the model's coefficients will also change proportionally, and the predictions will be scaled accordingly. For instance, if you multiply the output variable by a factor of 2, all the coefficients will also be multiplied by 2. The fundamental relationship captured by the model remains the same, just expressed on a different scale.
In summary:
- Input Scaling: Coefficients change, predictions remain effectively the same (after inverse transformation).
- Output Scaling: Coefficients and predictions change proportionally. The model's relationships remain consistent, just expressed on a different scale.