Lesson 03 - Minimizing SSE

Estimating the Center of a Collection of Values

The mean is only one way of indicating the center of a collection of values. Other so-called measures of central tendency include the median and the midrange. In this lecture, we illustrate one way in which the mean is an “optimal” choice for indicating the center of set of observations.

We will begin our discussion by assuming that the collection of values represents a population, but everything we do in the lecture applies to samples as well as populations.

Consider a population consisting of \(N\) individuals. Let \(X\) be a random variable defined on this population, and let \(x_1, x_2, ..., x_N\) denote the realizations of \(X\) for the individuals within the population. We wish to estimate the location of the “center” of \(X\) for this population using a single number \(m\).

Sum of Squared Errors

There are many reasonable values that could be suggested for \(m\). We need to introduce a way of selecting the “best” possible value. To do this, we will introduce a method of scoring potential values of \(m\). This method will be called “Sum of Squared Errors”, and will be provided as a function of \(m\), denoted by \(SSE(m)\).

Assume that a value of \(m\) has been suggested. We can define the error in this estimate with respect to the ith observation \(x_i\) as follows:

\[e_i = x_i - m\]

Notice that \(e_i\) could be either positive, negative, or zero. To prevent large negative errors from cancelling out large positive errors (thus resulting in a small overall error), we will square each of these error terms. We will then sum the resulting (non-negative) squared errors across the entire population. In summary, we define \(SSE(m)\) as follows:

\[SSE(m) = \sum_{i=1}^N e_i^2 = \sum_{i=1}^N (x_i - m)^2\]

Minimizing SSE

The function \(SSE(m)\) provides a measure of how much the observations in our population vary from a proposed center \(m\). Our goal is to select the value of \(m\) that minimizes the function \(SSE(m)\). The next theorem states that this function is minimized by \(m = \mu\), where \(mu\) is the population mean.

---

Theorem 1. Let \(x_1, x_2, ..., x_N\) be the values of a random variable \(X\) defined on a population of size \(N\). Let \(SSE(m) = \sum_{i=1}^N (x_i - m)^2\). The function \(SSE(m)\) has a unique minimum at \(m = \mu\), where \(\mu = \sum_{i=1}^N x_i\).

---

This result still holds true if our observations \(x_i\) constitute a sample rather than an entire population. This is stated in the following theorem.

---

Theorem 2. Let \(X\) denote a random variable, and let \(x_1, x_2, ..., x_n\) denote a sample of \(n\) observations of \(X\). Let \(SSE(m) = \sum_{i=1}^n (x_i - m)^2\). The function \(SSE(m)\) has a unique minimum at \(m = \bar x\), where \(\bar x = \sum_{i=1}^n x_i\).

---

We will prove Theorem 1 soon, but before doing so, let’s see an example illustrating Theorem 2 with a small sample.

Example: Minimizing SSE for a Sample

Consider the sample given by \(x_1=2, x_2=3, x_3=5, x_4=8\).

We wish to estimate the center of this data with a number \(m\) such that the sum of squared errors is minimized. The steps required to find such an \(m\) are shown below.

• \(SSE(m) = (2-m)^2 + (3-m)^2 + (5 - m)^2 + (8 - m)^2\)

• \(SSE(m) = (4 - 4 m + m^2) + (9 - 6 m +m^2) + (25 - 10 m + m^2) + (64 - 16 m + m^2)\)

• \(SSE(m) = 4 m^2 - 36 m + 102\)

• \(SSE'(m) = 8 m - 36\)

• \(8 m - 36 = 0\)

• \(8 m = 36\)

• \(m = 4.5\)

Note that \(\bar x = \frac{1}{4}(2 + 3 + 5 + 8) = 4.5\)

---

---

We will now prove Theorem 1. The proof of Theorem 2 is practically identical.

Theorem 1. Let \(x_1, x_2, ..., x_N\) be the values of a random variable \(X\) defined on a population of size \(N\). Let \(SSE(m) = \sum_{i=1}^N (x_i - m)^2\). The function \(SSE(m)\) has a unique minimum at \(m = \mu\), where \(\mu = \sum_{i=1}^N x_i\).

Proof. We begin by rewriting the expression for \(SSE(m)\).

\[SSE(m) = \sum_{i=1}^N (x_i^2 - 2 x_i m +m^2)\] \[SSE(m) = \sum_{i=1}^N (m^2 - 2 x_i m +x_i^2)\]

\[SSE(m) = \sum_{i=1}^N (m^2) + \sum_{i=1}^N( - 2 x_i m) + \sum_{i=1}^N (x_i^2)\]

\[SSE(m) = N m^2 -2 m \sum_{i=1}^N x_i + \sum_{i=1}^N x_i^2\]

Notice that for a given population, \(N\), \(-2\sum_{i=1}^N x_i\), \(\sum_{i=1}^N x_i^2\) are all known constants. As such, the formula for \(SSE(m)\) is a second-degree polynomial in \(m\). Since the leading coefficient (\(N\)) is positive, the graph of \(SSE(m)\) is a parabola opening upward, and thus has a unique minimum. Let \(m^*\) be this minimum. Then:

\[m^* = -\frac{-2\sum_{i=1}^N x_i}{2 N} = \frac{1}{N}\sum_{i=1}^N x_i = \mu\]

---