Lesson 05 - Hypothetical and Fitted Models

Deterministic vs. Stochastic Models

Let \(X\) and \(Y\) be two variables that are related in some way. Assume that we wish to build a model that allows us to make predictions about the value of \(Y\) based on a supplied value for \(X\).

• A deterministic model is one in which there is no randomness. Given a specific value for the input variable \(X\), a deterministic model will return a specific value for \(Y\). When working with deterministic models, we tend to use the following terms to refer to the variables:

  • The input variable \(X\) is called the independent variable.
  • The output variable \(Y\) is called the dependent variable.
    
    

• A stochastic model is one that incorporates a random element into its predictions. Given a specific value of \(X\), a stochastic model provides a probability distribution for the variable \(Y\). The model does not tell us the exact value of \(Y\), but it allows us to make probabilistic statements concerning its possible values. When working with stochastic models, we tend to use the following terms to refer to the variables:

  • The input variable \(X\) is called the predictor variable.
  • The output variable \(Y\) is called the response variable.

Hypothetical Model

In simple linear regression, we assume that the predictor variable \(X\) and the response variable \(Y\) are related by a linear stochastic relationship of the following form:

\[Y = \beta_0 + \beta_1 X + e\]

where \(e\) is a random variable. We call the equation above the hypothetical model or population model for linear regression.

• The random variable \(e\) is called the error term or the noise term.

• The error term \(e\) is intended to account for the (hopefully small) effects that have not been explained by the deterministic portion of our model.

• Given a specific paired observation \(\left(x_i, y_i \right)\), we write \(y_i = \beta_0 + \beta_1 x_i+ e_i\).

• When applying this model, we will often assume that \(e\) follows a normal distribution with a mean of 0 and some standard deviation \(\sigma\). The size of \(\sigma\) reflects that amoung of uncertainty in our model.

• Note that if \(E[e]=0\), then \(E[Y|X] = \beta_0 + \beta_1 X\).

Fitted Model

We generally begin a linear regression problem by assuming that a hypothetical model of the form \(Y = \beta_0 + \beta_1 X + e\) exists and describes the relationship between our variables. We do not generally have direct access to such a model, however. Our goal in a regression problem is to approximate the hypothetical model by collecting a sample of paired observations of the form \(\left(x_i, y_i \right)\), and then using those to attempt to reconstruct our model. An approximate model generated from sample data is referred to a fitted model.

For simple linear regression, creating a fitted model involves estimating the values of the parameters \(\beta_0\) and \(\beta_1\), as well as proposing a reasonable distribution for \(e\).

• We denote the estimates of \(\beta_0\) and \(\beta_1\) by \(\hat\beta_0\) and \(\hat\beta_1\) respectively.

• Given parameter estimates \(\hat\beta_0\) and \(\hat\beta_1\), we define a new variable \(\hat Y = \hat \beta_0 + \hat\beta_1 X\).

• The variable \(\hat Y\) is called the fitted value of \(Y\). It is the point estimate that the model generates for \(Y\) given the value of \(X\).

• For a given paired observation \(\left(x_i, y_i \right)\), we write \(\hat y_i = \hat \beta_0 + \hat \beta_1 x_i\).

• For a given paired observation \(\left(x_i, y_i \right)\), we have that \(\hat e_i = y_i - \left(\hat \beta_0 + \hat \beta_1 x_i \right)\), or in other words \(\hat e_i = y_i - \hat y_i\).

• The values \(\hat e_i\) calculated from the paired observations in our sample are called residuals. We can use the residuals to approximate the distribution of the error term \(e\).

• We can write our fitted model as either \(\hat Y = \hat \beta_0 + \hat\beta_1 X\) or \(Y = \hat \beta_0 + \hat\beta_1 X + \hat e\).

Summary

We assume that the relationship between our variables is determined by a hypothetical model of the form: \[Y = \beta_0 + \beta_1 X + e\]

The fitted model is an approximation of the hypothetical model, and can be written in either of the following forms:

\[\hat Y = \hat \beta_0 + \hat \beta_1 X\] \[Y = \hat \beta_0 + \hat \beta_1 X + \hat e\] Given a paired observation \(\left(x_i, y_i \right)\), the fitted value of y given \(x_i\) is given by: \[\hat y_i = \hat\beta_0 + \hat\beta_1 x_i\]

The residual associated with the observation is given by:

\[\hat e_i = y_i - \hat y_i\]

Finding the Fitted Model

We have not yet discussed how to find the fitted model \(\hat Y = \hat \beta_0 + \hat \beta_1 X\). Technically, any proposed values of the parameter estimates \(\hat\beta_0\) and \(\hat\beta_1\) would result in a fitted model. However, some values of these estimates will obviously result in a better estimate of the hypothetical model than others. In the next lecture, we will discuss how to find the best possible parameter estimates (at least for one common interpretation of what word “best” means in this context).