Lesson 10 - Hypothesis Tests for Parameters

Linear Regression Assumptions (Gauss-Markov Assumptions)

In the last lesson, we introduced a set of assumptions that we will commonly make regarding the error term in our hypothetical model \(Y = \beta_0 + \beta_1 X + e\). These assumptions are as follows:

1. The relationship between X and Y is determined by a linear model of the form \(Y = \beta_0 + \beta_1 X + e\).

2. Seperate observations of \(e\) are independent from one another.

3. The error term \(e\) is normally distributed with a mean of 0 and standard deviation \(\sigma\).

4. The error term \(e\) is independent from \(X\). In particular, the \(\mathrm{e} = \sigma^2\) does not depend on \(X\) (homoskedasticity).

Note that these assumptions are not required in order to find or even apply the OLS regression line. They are, however, necessary if we wish to use the model for inferential purposes.

Distribution of Parameter Estimates

Assume that we fix several \(X\) values, \(x_1, ..., x_n\) and that we use these \(X\) values to generate \(Y\) values \(y_1, ..., y_n\) according to a hypothetical model \(Y = \beta_0 + \beta_1 X + e\). Suppose we then calculate parameter estimates \(\hat\beta_0\) and \(\hat\beta_1\).

If we repeat this process, we will get different values of \(y_1, ..., y_n\) each time, since the error terms \(e_i\) are assumed to be random. Since our \(Y\) values will vary each time, so too will our parameter estimates \(\hat\beta_0\) and \(\hat\beta_1\). In this way, our parameter estimates are random variables whose distribution depends on that of \(e\).

---

It can be shown that if the Gauss-Markov assumptions hold, then the distibutions of \(\hat\beta_0\) and \(\hat\beta_1\) are as follows:

• \(\hat\beta_0\) follows a normal distribution with:

  • \(\mu_{\hat\beta_0} = \mathrm{E}[\hat\beta_0] = \beta_0\)

  • \(\sigma_{\hat\beta_0}^2 = \mathrm{Var}[\hat\beta_0] = \sigma^2 \left[\frac{1}{n} + \frac{\bar x^2}{SXX} \right]\)

• \(\hat\beta_1\) follows a normal distribution with:

  • \(\mu_{\hat\beta_1} = \mathrm{E}[\hat\beta_1] = \beta_1\)

  • \(\sigma_{\hat\beta_1}^2 = \mathrm{Var}[\hat\beta_1] = \frac{\sigma^2}{SXX}\)

---

Unfortunately, we don’t typically know \(\mathrm{Var}[e] = \sigma^2\). If we do not know \(\sigma^2\), then we will need to approximate it using \(s^2 = \frac{SSE}{n-2}\). Replacing \(\sigma^2\) with \(s^2\) in the variance formulas above, and then taking square roots results in the appoximations for \(\mathrm{SD}[\hat\beta_0]\) and \(\mathrm{SD}[\hat\beta_1]\) shown below. These approximations are called the standard errors of the parameter estimates.

• \(\mathrm{SD}[\hat\beta_0] \approx \mathrm{SE}[\hat\beta_0] = s \sqrt{\frac{1}{n} + \frac{\bar x^2}{SXX} }\)

• \(\mathrm{SD}[\hat\beta_1] \approx \mathrm{SE}[\hat\beta_1] = \frac{s^2}{\sqrt{SXX}}\)

Hypothesis Tests for Parameters

It is often useful to be able to test the hypothesis that a particular parameter is equal to a specific number. In other words, we want to be able to conduct hypothesis tests of the following form:

\(\hspace{30pt} H_0: \beta_i = k\)

\(\hspace{30pt} H_A: \beta_i \neq k\)

We can perform such a test using the following test statistic:

\(\large\hspace{30pt} t = \frac{\hat\beta_{i} - k}{\mathrm{SE}[\hat\beta_i]}\)

This test statistic will follow a t distribution with \(n-2\) degrees of freedom.

Although we can test either regression parameter for any value of \(k\), the test that we will be most interested in is the following:

\(\hspace{30pt} H_0: \beta_1 = 0\)

\(\hspace{30pt} H_A: \beta_1 \neq 0\)

If we are unable to reject the null hypothesis in this test, then we can not be confident that the parameter \(\beta_1\) is non-zero. If \(\beta_1 = 0\), then our hypothetical model reduces to \(Y = \beta_0 + e\), and there is no relationship between \(X\) and \(Y\).

Hypothesis Tests in R

Fortunately, R will test for the regression parameters being zero when you call the lm() function. The results of these tests are displayed in the summary output for the model. To demonstrate this, we will return to the Pearson dataset.

myData <- read.table("father_son.txt", sep="\t", header=TRUE)
mod <- lm(sheight ~ fheight, myData)
summary(mod)

Call:
lm(formula = sheight ~ fheight, data = myData)

Residuals:
    Min      1Q  Median      3Q     Max 
-8.8772 -1.5144 -0.0079  1.6285  8.9685 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 33.88660    1.83235   18.49   <2e-16 ***
fheight      0.51409    0.02705   19.01   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 2.437 on 1076 degrees of freedom
Multiple R-squared:  0.2513,    Adjusted R-squared:  0.2506 
F-statistic: 361.2 on 1 and 1076 DF,  p-value: < 2.2e-16

Before putting too much stock into the results of these hypothesis tests, we should perform an analysis of our residuals.

plot(myData$sheight ~ myData$fheight, pch=21, bg="cyan", col="black", cex=1,
     main="Scatter Plot", xlab="Father Height", ylab="Son Height")
abline(mod$coefficients, col="darkred", lwd=2)

plot(mod$residuals ~ myData$fheight, pch=21, bg="cyan", col="black", cex=1,
     main="Residual Plot", xlab="Father Height", ylab="Residuals")
abline(h=0, col="darkred", lwd=2)

qqnorm(mod$residuals)
qqline(mod$residuals)

Simulated Examples

n = 100
x <- runif(n, 0, 10)
y <- 7 + 1.3 * x + rnorm(n, 0, 15)
m <- lm(y ~ x)
summary(m)

Call:
lm(formula = y ~ x)

Residuals:
    Min      1Q  Median      3Q     Max 
-31.020  -9.456  -1.969  11.941  32.565 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  11.4469     2.9146   3.927  0.00016 ***
x             0.7086     0.5112   1.386  0.16890    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 15.83 on 98 degrees of freedom
Multiple R-squared:  0.01922,   Adjusted R-squared:  0.009217 
F-statistic: 1.921 on 1 and 98 DF,  p-value: 0.1689

plot(y ~ x, pch=21, bg="cyan", col="black", cex=1)
abline(m$coefficients, col="darkred", lwd=2)

abline(7, 1.3, col="darkblue", lwd=2, lty=2)