Lesson 22 - Effects of Multicolinearity

Load Packages

require(ggplot2)
require(MASS)
require(RColorBrewer)
require(dplyr)
require(reshape)

Create Sets of Predictors

We begin by creating several six different sets of predictors. Each set contains two predictors, X1 and X2, with varying degrees of correlation.

Statistics for our Predictor Sets

Generating Fitted Models

Assume we have a variable \(Y\) that is related to \(X1\) and \(X2\) according to the following hypothetical mode:

\[Y = 4 + 2 \cdot X1 + 3 \cdot X2 + e, \hspace{2 em} e \sim N(0, 2.5)\]

For each set of predictors, we will do the following:

1. Create 100 sets of observed \(Y\) values according to the model above.
2. For each set of observed \(Y\) values, we will create a fitted model.
3. For each fitted model, we will store the coefficient estimates and the residual standard error for the model.

All told, we will create 6 models; 100 for each set of predictors. A summary of the statistics from our fitted models is shown below, grouped by predictor set.

Distribution of Parameter Estimates

We see that on average, the estimates of the coefficients are approximately equal to the values, regardless of the strength of the correlation in our predictors. However, the mean of the estimates only tells us part of the story. We should look at how these parameter estimates are distributed.

Relationship between Parameter Estimates

We can see from the plot above, the estimates for \(\beta_0\) and \(\beta_1\) can vary wildly when the predictors are strongly correlated. However, there is some order in the way in which these parameter estimates vary.

Residual Standard Errors

We saw above that the mean of the residual standard error resulting from the fitted models was very close to the true value of \(\sigma\), regardless of the size of \(corr(X1, X2)\). As with the parameter estimates, we should see how the distribution of \(s\) changes with respect to \(corr(X1, X2)\).

Distribution of Predictions

Recall that our hypothetical model was given by \(Y = 3 + 2 \cdot X1 + 2 \cdot X2 + e\), where \(e \sim N(0,2)\). If we plug \(X1 = 1.5, X2 = 1.5\) into this model, we get \(Y = 9 + e\).

We will now take each of the 600 fitted models we created previously, and will use those models to generate predictions for \(X1 = 1.5, X2 = 1.5\). For each prediction, we will also produce a 95% prediction interval, noting the margin of error of the resulting interval.

The table below shows the mean of the predicted values, as well as the mean margin of error for the prediction interval, group by predictor set.

We see that, on average, the predictions and the margin of error are about what they should be, regardless of the value of \(corr(X1, X2)\). As before, we should see how these values are distributed for each set of predictors.

We will start with the predicted values.

We now consider the margin of error for the prediction intervals.

Comparing with a SLR Model

We know that our data was generated using a hypothetical model of the form \(Y = 3 + 2 \cdot X1 + 2 \cdot X2 + e\), where \(e \sim N(0,2)\). However, we will now consider what our results would look like if we considered a simple linear regression model of the form \(Y =\hat\beta_0 + \hat\beta_1 \cdot X1\). More importantly, we will study how the results from that model vary for different values of \(corr(X1, X2)\).

Predictions from SLR Model

As we did with the MRL models, we will now use each of the 600 fitted SLR models to generate predictions for \(X1 = 1.5\). For each prediction, we will also produce a 95% prediction interval, noting the margin of error of the resulting interval.

The table below shows the mean of the predicted values, as well as the mean margin of error for the prediction interval, group by predictor set.

We will plot the distributions of the predictions for each group of predictors.

We will also plot the distributions of the prediction interval margin of errors for each group of predictors.