Lesson 04 - Covariance and Correlation

Population Covariance and Correlation

Covariance and correlation are values that measure the degree to which two variables are linearly related. The definitions of population covariance and correlation are as follows:

---

Definition. Assume that \(X\) and \(Y\) are random variables defined for individuals in a population of size \(N\). Let the paired values of \(X\) and \(Y\) for individuals in the population be denoted by \((x_1, y_1), (x_2,y_2), ..., (x_N,y_N)\).

• The population covariance of \(X\) and \(Y\) is denoted by \(Cov[X,Y]\) and is defined by \(Cov[X,Y] = \frac{1}{N} \sum\limits_{i=1}^N (x_i - \mu_X)(y_i - \mu_Y)\).

• The population correlation is denoted by \(Corr[X,Y]\) and is defined by \(Corr[X,Y] = \frac{Cov[X,Y]}{\sigma_X\sigma_Y}\).

---

Properties of Covariance and Correlation

There are several important properties of Covariance and Correlation that should be noted.

1. \(Cov[X,Y]\) is measured in units of \(X\) times units of \(Y\).

2. \(Corr[X,Y]\) is unitless.

3. \(-1 \leq Corr[X,Y] \leq 1\)

4. \(Corr[X,Y]\) is a measure of the linear relationship between \(X\) and \(Y\).

5. If \(X\) and \(Y\) are independent, then \(Cov[X,Y] = Corr[X,Y] = 0\)

6. \(Cov[X,X] = Var[X]\)

Sample Covariance and Correlation

We also have notions of covariance and correlation as calculated from a sample as opposed to a population. The definitions of these quantities are provided below.

---

Definition. Assume that \(X\) and \(Y\) are random variables defined for individuals in a population. Assume a sample of size \(n\) is drawn from the population, and the paired observations of \(X\) and \(Y\) for individuals in the sample are denoted by \((x_1, y_1), (x_2,y_2), ..., (x_n,y_n)\).

• The sample covariance of \(X\) and \(Y\) is denoted by \(cov[X,Y]\) and is defined by \(cov[X,Y] = \frac{1}{n-1} \sum\limits_{i=1}^n (x_i - \bar x)(y_i - \bar y)\).

• The sample correlation is denoted by \(corr[X,Y]\) or \(\rho_{X,Y}\) and is defined by \(corr[X,Y] = \rho_{X,Y} = \frac{cov[X,Y]}{s_X s_Y}\).

---

Algebraic Properties of Mean, Variance, and Covariance

We conclude this lesson by stating important algebraic properties of mean, variance, and covariance. Each of these properties is stated for the population parameters, but also hold for sample statistics.

Theorem. Let \(X\), \(Y\), and \(Z\) be random variables and let \(a\) and \(b\) be constants. Then:

1. \(\mathrm{E}[X + Y] = \mathrm{E}[X] + \mathrm{E}[Y]\)

2. \(\mathrm{Var}[X + Y] = \mathrm{Var}[X] + \mathrm{Var}[Y] +2 \mathrm{Cov}[X,Y]\)

3. \(\mathrm{Cov}[X, Y] = \mathrm{Cov}[Y, X]\)

4. \(\mathrm{Cov}[a X, b Y] = a b \mathrm{Cov}[X, Y]\)

5. \(\mathrm{Cov}[X + Y, Z] = \mathrm{Cov}[X, Z] + \mathrm{Cov}[Y, Z]\)

We will provide proofs of Property 1 and Property 2 in the case where \(X\) is a random variable defined on a population of size \(N\).

---

Proof of Property 1. Let \((x_1,y_1), (x_2,y_2), ..., (x_N,y_N)\) denote the paired values of \(X\) and \(Y\) for individuals within the population. Then:

\[\mathrm{E}[X + Y] = \frac{1}{N} \sum_{i=1}^N (x_i + y_i) = \frac{1}{N} \left(\sum_{i=1}^N x_i + \sum_{i=1}^N y_i \right ) = \frac{1}{N} \sum_{i=1}^N x_i + \frac{1}{N}\sum_{i=1}^N = \mathrm{E}[X] + \mathrm{E}[Y]\]

---

Proof of Property 2. Let \((x_1,y_1), (x_2,y_2), ..., (x_N,y_N)\) denote the paired values of \(X\) and \(Y\) for individuals within the population. Then:

\[\mathrm{Var}[X + Y] = \frac{1}{N} \sum_{i=1}^N \left[(x_i + y_i) - E[X + Y] \right ]^2 \] \[= \frac{1}{N} \sum_{i=1}^N \left[(x_i + y_i) - \mu_X + \mu_Y \right ]^2 \] \[= \frac{1}{N} \sum_{i=1}^N \left[(x_i - \mu_X) + (y_i - \mu_Y)\right ]^2 \] \[= \frac{1}{N} \sum_{i=1}^N \left[(x_i - \mu_X)^2 + 2(x_i - \mu_X)(y_i - \mu_Y) + (y_i - \mu_Y)^2\right ] \] \[= \frac{1}{N} \left[ \sum_{i=1}^N (x_i - \mu_X)^2 + 2\sum_{i=1}^N(x_i - \mu_X)(y_i - \mu_Y) + \sum_{i=1}^N(y_i - \mu_Y)^2\right ] \] \[= \frac{1}{N} \sum_{i=1}^N (x_i - \mu_X)^2 + 2\frac{1}{N}\sum_{i=1}^N(x_i - \mu_X)(y_i - \mu_Y) + \frac{1}{N}\sum_{i=1}^N(y_i - \mu_Y)^2 \] \[= \mathrm{Var}[X] + 2\mathrm{Cov}[X,Y] + \mathrm{Var}[Y] \] \[= \mathrm{Var}[X] + \mathrm{Var}[Y] + 2\mathrm{Cov}[X,Y] \]

---