MLR: Properties of Residuals and the F-test

STAT 306 - Module 10

Learning objectives

Show that the random residuals can be written as \(\mathbf{E}=(\mathbf{I}_n-\mathbf{P})\pmb{\varepsilon}\), where \(\mathbf{P}\) is the hat matrix, and therefore do not depend on \(\pmb{\beta}\).
Prove that the sum of the residuals is zero for a fitted regression model.
Derive the expectation and covariance matrix of the vector of residuals in multiple regression.
Show that the residual mean square is an unbiased estimator of the variance about the regression plane in multiple regression.
Show that, under normal errors, the residuals are independent of \(\widehat{\pmb{\beta}}\), and explain how this leads to the \(t\) and \(F\) distributions.
Show that the model mean square \(MSS/p\) is unbiased for \(\sigma^2\) under \(H_0:\beta_1=\dotsc=\beta_p=0\).
Carry out and interpret the F-test for the overall regression, and for nested models, by comparing residual sums of squares.

Residual mean square

In Module 09, we estimated \(\sigma^2\), the variance of the errors, with the residual mean square (also denoted \(MSE\)) \[ \hat\sigma^2 := MSE = \frac{\mathbf{e^\top e}}{n-(p+1)} = \frac{RSS}{n-(p+1)} \;, \] and promised to justify this choice.
Today, we’ll work towards understanding why this is the case.

Linear model and residuals

Recall our linear model \[ \mathbf{Y} = \mathbf{X}\pmb{\beta} + \pmb{\varepsilon} \;, \] where \(\pmb{\varepsilon}\) is a vector containing the error terms \(\varepsilon_1,\dotsc,\varepsilon_n\), which are independent \(N(0,\sigma^2)\). In vector form, \(\pmb{\varepsilon}\sim N_n(\mathbf{0},\sigma^2\mathbf{I}_n)\), so \[ \mathbb{E}[\pmb{\varepsilon}] = \mathbf{0} := \begin{bmatrix} 0 \\ \vdots \\ 0 \end{bmatrix} \;, \qquad \mathrm{Cov}(\pmb{\varepsilon}) = \mathbb{E}[\pmb{\varepsilon}\pmb{\varepsilon}^\top] = \sigma^2\mathbf{I}_n \;. \] (Recall that \(\mathbf{I}_n\) is the \(n\times n\) identity matrix.)
Throughout, the first column of \(\mathbf{X}\) is a column of ones (the intercept), \(\mathbf{X}\) has full column rank \(p+1\) (so \(\mathbf{X^\top X}\) is invertible), and \(n > p+1\).

Our least squares estimates of the \(p+1\) model coefficients \(\pmb{\beta}\) are given by \[ \widehat{\pmb{\beta}} = (\mathbf{X^\top X})^{-1}\mathbf{X^\top y} \;, \] and our observed residuals are \[ \mathbf{e} = \mathbf{y} - \mathbf{X}\widehat{\pmb{\beta}} = \mathbf{y} - \hat{\mathbf{y}} \;. \]
The residual sum of squares is \[RSS = \mathbf{e^\top e} = \sum_{i=1}^n(y_i-\hat{y}_i)^2 \;. \]
Let’s now consider the residuals as a random vector \(\mathbf{E}=\mathbf{Y}-\mathbf{X}\widehat{\pmb{\beta}} = \mathbf{Y}-\widehat{\mathbf{Y}}\), where now \(\widehat{\pmb{\beta}} = (\mathbf{X^\top X})^{-1}\mathbf{X^\top Y}\) is the (random) estimator.

iClicker 1

Which of the following correctly matches the variables to their definitions?

	\(\quad\mathbf{y}-\hat{\mathbf{y}}\quad\)	\(\quad\mathbf{Y} - \mathbf{X}\pmb{\beta}\quad\)	\(\quad\mathbf{Y}-\mathbf{X}\widehat{\pmb{\beta}}\quad\)
(A)	\(\pmb{\varepsilon}\)	\(\mathbf{E}\)	\(\mathbf{e}\)
(B)	\(\pmb{\varepsilon}\)	\(\mathbf{e}\)	\(\mathbf{E}\)
(C)	\(\mathbf{E}\)	\(\pmb{\varepsilon}\)	\(\mathbf{e}\)
(D)	\(\mathbf{e}\)	\(\pmb{\varepsilon}\)	\(\mathbf{E}\)
(E)	\(\mathbf{e}\)	\(\mathbf{E}\)	\(\pmb{\varepsilon}\)

iClicker 1: Solution

Which of the following correctly matches the variables to their definitions?

	\(\quad\mathbf{y}-\hat{\mathbf{y}}\quad\)	\(\quad\mathbf{Y} - \mathbf{X}\pmb{\beta}\quad\)	\(\quad\mathbf{Y}-\mathbf{X}\widehat{\pmb{\beta}}\quad\)
(A)	\(\pmb{\varepsilon}\)	\(\mathbf{E}\)	\(\mathbf{e}\)
(B)	\(\pmb{\varepsilon}\)	\(\mathbf{e}\)	\(\mathbf{E}\)
(C)	\(\mathbf{E}\)	\(\pmb{\varepsilon}\)	\(\mathbf{e}\)
(D)	\(\mathbf{e}\)	\(\pmb{\varepsilon}\)	\(\mathbf{E}\)
(E)	\(\mathbf{e}\)	\(\mathbf{E}\)	\(\pmb{\varepsilon}\)

\(\mathbf{e} = \mathbf{y} - \hat{\mathbf{y}}\) are the observed residuals, \(\pmb{\varepsilon} = \mathbf{Y} - \mathbf{X}\pmb{\beta}\) are the (unobservable) errors, and \(\mathbf{E} = \mathbf{Y} - \mathbf{X}\widehat{\pmb{\beta}}\) are the random residuals.

The Hat Matrix

The fitted values can be written as

\[ \begin{align} \hat{\mathbf{y}} &= \mathbf{X}\widehat{\pmb{\beta}}\\ &= \mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X^\top y}\\ &= \mathbf{P}\mathbf{y} \end{align} \]

The matrix \(\mathbf{P} := \mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X^\top}\) transforms the observed response vector \(\mathbf{y}\) into the fitted values \(\hat{\mathbf{y}}\).
- It “puts a hat on” \(\mathbf{y}\), i.e., \(\hat{\mathbf{y}} = \mathbf{P}\mathbf{y}\).
- For this reason, we call it the hat matrix.
- It is also called a projection matrix because it projects \(\mathbf{y}\) onto the column space of \(\mathbf{X}\).

Properties of the hat matrix

Note that the hat matrix \(\mathbf{P}\) is idempotent, i.e., \(\mathbf{P}\mathbf{P}=\mathbf{P}\):

\[ \begin{align} \mathbf{P}\mathbf{P} &= \mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X^\top X}(\mathbf{X^\top X})^{-1}\mathbf{X^\top} \\ &= \mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X^\top}\\ &= \mathbf{P} \;. \end{align} \]

The hat matrix \(\mathbf{P}\) is also symmetric, i.e., \(\mathbf{P}^\top=\mathbf{P}\):
- Note: the transpose of the inverse of a matrix is the inverse of the transpose of that matrix,
  i.e., \((\mathbf{A}^{-1})^\top = (\mathbf{A}^\top)^{-1}\), for any invertible matrix \(\mathbf{A}\).

\[ \begin{align} \mathbf{P}^\top &= \left(\mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X^\top}\right)^\top \\ &= \mathbf{X}\left[(\mathbf{X^\top X})^{-1}\right]^\top\mathbf{X^\top} \\ &= \mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X^\top}\\ &= \mathbf{P} \;. \end{align} \]

Properties of residuals

Expanding the vector \(\mathbf{E}\) of random residuals, we get

\[ \begin{align} \mathbf{E} &= \mathbf{Y} - \widehat{\mathbf{Y}} \\ &= \mathbf{Y} - \mathbf{P}\mathbf{Y} \\ &= (\mathbf{I}_n - \mathbf{P})\mathbf{Y} \\ &= (\mathbf{I}_n - \mathbf{P})(\mathbf{X}\pmb{\beta}+\pmb{\varepsilon}) \\ &= (\mathbf{I}_n - \mathbf{P})\mathbf{X}\pmb{\beta} + (\mathbf{I}_n - \mathbf{P})\pmb{\varepsilon} \;. \end{align} \]

Exercise 1

Show that the residuals do not depend on \(\pmb{\beta}\).
- This is a key property of the residuals.
- It means that the distribution of the residuals is the same no matter what the true values of the regression coefficients are.

Hint: Show that \((\mathbf{I}_n - \mathbf{P})\mathbf{X}\pmb{\beta} = \mathbf{0}\)

Exercise 1: Solution

\[ \begin{align} (\mathbf{I}_n - \mathbf{P})\mathbf{X} &= (\mathbf{I}_n - \mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X^\top})\mathbf{X} \\ &= \mathbf{X} - \mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X}^\top\mathbf{X}\\ &= \mathbf{X} - \mathbf{X}\\ &= \mathbf{0} \;. \end{align} \]

Therefore, \((\mathbf{I}_n - \mathbf{P})\mathbf{X}\pmb{\beta} = \mathbf{0}\pmb{\beta} = \mathbf{0}\).
Hence, \(\mathbf{E} = (\mathbf{I}_n - \mathbf{P})\pmb{\varepsilon}\).

Exercise 2

Show that \(\mathbf{I}_n-\mathbf{P}\) is:

symmetric, i.e., \((\mathbf{I}_n-\mathbf{P})^\top=\mathbf{I}_n-\mathbf{P}\).
idempotent, i.e., \((\mathbf{I}_n-\mathbf{P})(\mathbf{I}_n-\mathbf{P})=\mathbf{I}_n-\mathbf{P}\).

iClicker 2

Which of the following is equal to \((\mathbf{I}_n-\mathbf{P})(\mathbf{I}_n-\mathbf{P})^\top\)?

\(\mathbf{I}_n-\mathbf{P}\)
\(\mathbf{P}\)
\((\mathbf{X^\top X})^{-1}\mathbf{P}^\top\)
\((\mathbf{X^\top X})^{-1}\mathbf{P}\)
\(\mathbf{I}_n-2\mathbf{P}\)

iClicker 2: Solution

Which of the following is equal to \((\mathbf{I}_n-\mathbf{P})(\mathbf{I}_n-\mathbf{P})^\top\)?

\(\mathbf{I}_n-\mathbf{P}\)
\(\mathbf{P}\)
\((\mathbf{X^\top X})^{-1}\mathbf{P}^\top\)
\((\mathbf{X^\top X})^{-1}\mathbf{P}\)
\(\mathbf{I}_n-2\mathbf{P}\)

Exercise 2: Solution

Symmetry: \[ \begin{align} (\mathbf{I}_n-\mathbf{P})^\top &= \mathbf{I}_n^\top - \mathbf{P}^\top \\ &= \mathbf{I}_n - \mathbf{P} \end{align} \]
Idempotency: \[ \begin{align} (\mathbf{I}_n-\mathbf{P})(\mathbf{I}_n-\mathbf{P}) &= \mathbf{I}_n - \mathbf{P} - \mathbf{P} + \mathbf{P}\mathbf{P} \\ &= \mathbf{I}_n - \mathbf{P} - \mathbf{P} + \mathbf{P} \\ &= \mathbf{I}_n - \mathbf{P} \end{align} \]
Therefore, \[ \begin{align} (\mathbf{I}_n-\mathbf{P})(\mathbf{I}_n-\mathbf{P})^\top &= (\mathbf{I}_n-\mathbf{P})(\mathbf{I}_n-\mathbf{P}) \\ &= \mathbf{I}_n-\mathbf{P} \end{align} \]

Exercise 3

Show that the sum of the residuals is zero, i.e., \(\sum_{i=1}^n E_i = 0\).

Hint: Show that \(\mathbf{X}^\top\mathbf{E} = \mathbf{0}\), and recall that the first column of \(\mathbf{X}\) is a column of ones (due to the intercept).

Exercise 3: Solution

Using \(\mathbf{E} = (\mathbf{I}_n-\mathbf{P})\mathbf{Y}\) and the results from Exercises 1 and 2, we have that: \[ \begin{align} \mathbf{X}^\top\mathbf{E} &= \mathbf{X}^\top(\mathbf{I}_n-\mathbf{P})\mathbf{Y} \\ &= \left((\mathbf{I}_n-\mathbf{P})^\top\mathbf{X}\right)^\top\mathbf{Y} \\ &= \left((\mathbf{I}_n-\mathbf{P})\mathbf{X}\right)^\top\mathbf{Y} \qquad \text{(Exercise 2: } \mathbf{I}_n-\mathbf{P} \text{ is symmetric)} \\ &= \mathbf{0}^\top\mathbf{Y} = \mathbf{0} \qquad \text{(Exercise 1: } (\mathbf{I}_n-\mathbf{P})\mathbf{X}=\mathbf{0}\text{)} \end{align} \]

The first column of \(\mathbf{X}\) is a column of ones (due to the intercept), so the first entry of \(\mathbf{X}^\top\mathbf{E}\) is \[ \begin{bmatrix} 1 & 1 & \cdots & 1 \end{bmatrix}\mathbf{E} = \sum_{i=1}^n E_i = 0 \;. \]

Exercise 4

Compute the expectation of the residuals, \(\mathbb{E}[\mathbf{E}]\).

iClicker 3

What is the expectation of \(\mathbf{E}\)?

\(\mathbf{I}_n - \mathbf{P}\)
\(\sigma^2(\mathbf{I}_n - \mathbf{P})\)
\(\sigma^2\mathbf{I}_n\)
\(\mathbf{1}\)
\(\mathbf{0}\)

iClicker 3: Solution

What is the expectation of \(\mathbf{E}\)?

\(\mathbf{I}_n - \mathbf{P}\)
\(\sigma^2(\mathbf{I}_n - \mathbf{P})\)
\(\sigma^2\mathbf{I}_n\)
\(\mathbf{1}\)
\(\mathbf{0}\)

Exercise 4: Solution

Compute the expectation of the residuals, \(\mathbb{E}[\mathbf{E}]\).

Solution:

There are different ways to show this: \[\mathbb{E}[\mathbf{E}] = \mathbb{E}[\mathbf{Y}-\mathbf{X}\widehat{\pmb{\beta}}] = \mathbb{E}[\mathbf{Y}] - \mathbf{X}\mathbb{E}[\widehat{\pmb{\beta}}] = \mathbf{X}\pmb{\beta} - \mathbf{X}\pmb{\beta} = \mathbf{0}\]
Another way: \[\mathbb{E}[\mathbf{E}] = \mathbb{E}[(\mathbf{I}_n - \mathbf{P})\pmb{\varepsilon}] = (\mathbf{I}_n - \mathbf{P})\mathbb{E}[\pmb{\varepsilon}] = (\mathbf{I}_n - \mathbf{P})\mathbf{0} = \mathbf{0}\]

Exercise 5

Compute the covariance matrix of the residuals, \(\mathrm{Cov}(\mathbf{E})\).

iClicker 4

Which of the following is the covariance matrix of \(\mathbf{E}\)?

\(\sigma^2\mathbf{I}_{p+1}\)
\(\sigma^2\mathbf{I}_n\)
\(\sigma^2(\mathbf{I}_{p+1}-\mathbf{P})\)
\(\sigma^2(\mathbf{I}_n - \mathbf{P})\)
\(\sigma^2(\mathbf{X^\top X})^{-1}\)

iClicker 4: Solution

Which of the following is the covariance matrix of \(\mathbf{E}\)?

\(\sigma^2\mathbf{I}_{p+1}\)
\(\sigma^2\mathbf{I}_n\)
\(\sigma^2(\mathbf{I}_{p+1}-\mathbf{P})\)
\(\sigma^2(\mathbf{I}_n - \mathbf{P})\)
\(\sigma^2(\mathbf{X^\top X})^{-1}\)

Exercise 5: Solution

Compute the covariance matrix of the residuals, \(\mathrm{Cov}(\mathbf{E})\).

Solution:

The covariance matrix of \(\mathbf{E}\) is: \[ \begin{align} \mathrm{Cov}(\mathbf{E}) &= \mathrm{Cov}[(\mathbf{I}_n - \mathbf{P})\pmb{\varepsilon}]\\ &= (\mathbf{I}_n - \mathbf{P})\mathrm{Cov}(\pmb{\varepsilon})(\mathbf{I}_n - \mathbf{P})^\top\\ &= (\mathbf{I}_n - \mathbf{P})\sigma^2\mathbf{I}_n(\mathbf{I}_n - \mathbf{P})^\top\\ &= \sigma^2(\mathbf{I}_n - \mathbf{P})(\mathbf{I}_n - \mathbf{P})^\top\\ &= \sigma^2(\mathbf{I}_n - \mathbf{P})\\ \end{align} \]

The covariance of the residuals

Unlike the errors, the residuals do not, in general, have the same variance: \[\mathrm{Var}(E_i) = \sigma^2(1-P_{ii})\] where \(P_{ii}\) is the \(i\)-th diagonal entry of \(\mathbf{P}\);

Also, the residuals are, in general, correlated. For \(i \neq j\), \[\mathrm{Cov}(E_i, E_j) = -\sigma^2 P_{ij}\]
We will come back to this when we discuss leverage and standardized residuals.

Residual Mean Square as an unbiased estimator of \(\sigma^2\)

Exercise 6

The Residual Mean Square is given by \[ \hat{\sigma}^2 = \frac{\mathbf{E^\top E}}{n-(p+1)} \]

Compute \(\mathbb{E}\left[\hat{\sigma}^2\right]\).

Hints:

For a vector \(\mathbf{x}\) and a matrix \(\mathbf{A}\), \(\mathbf{x}^\top \mathbf{A} \mathbf{x}\) is the matrix notation for a quadratic form, and \(\mathbf{x}^\top \mathbf{A} \mathbf{x} = \sum_{i} \sum_j x_ix_j A_{ij}\).

\(\sum_i A_{ii}=\mathrm{tr}(\mathbf{A})\)

For any matrices \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{C}\) of appropriate dimensions, we have \(\mathrm{tr}(\mathbf{ABC}) = \mathrm{tr}(\mathbf{BCA}) = \mathrm{tr}(\mathbf{CAB})\).

For any matrices \(\mathbf{A}\) and \(\mathbf{B}\) of appropriate dimensions, we have \(\mathrm{tr}(\mathbf{A+B}) = \mathrm{tr}(\mathbf{A}) + \mathrm{tr}(\mathbf{B})\) and \(\mathrm{tr}(\mathbf{A-B}) = \mathrm{tr}(\mathbf{A}) - \mathrm{tr}(\mathbf{B})\).

iClicker 5

Which of the following is equal to \(\mathbb{E}\left[\mathbf{E^\top E}\right]\)?

\(\sigma^2\mathbf{I}_n\)
\(n\sigma^2\)
\((n-p)\sigma^2\)
\(\sigma^2\mathrm{tr}(\mathbf{I}_n-\mathbf{P})\)
\(\sigma^2(\mathbf{I}_n-\mathbf{P})\)

iClicker 5: Solution

Which of the following is equal to \(\mathbb{E}\left[\mathbf{E^\top E}\right]\)?

\(\sigma^2\mathbf{I}_n\)
\(n\sigma^2\)
\((n-p)\sigma^2\)
\(\sigma^2\mathrm{tr}(\mathbf{I}_n-\mathbf{P})\)
\(\sigma^2(\mathbf{I}_n-\mathbf{P})\)

Exercise 6: Solution

Note that: \[ \begin{align*} \mathbb{E}\left[\sum_{i=1}^nE_i^2\right] &= \sum_{i=1}^n \mathbb{E}\left[E_i^2\right] \\ &= \sum_{i=1}^n Var(E_i) \quad \text{(since } \mathbb{E}[E_i] = 0) \\ &= \sum_{i=1}^n \sigma^2(1-P_{ii}) \\ &= \sigma^2\left(n-\sum_{i=1}^nP_{ii}\right) \\ &= \sigma^2\left(n-\text{tr}\left(\mathbf{P}\right)\right)\\ &= \sigma^2\left(n-\text{tr}\left(\mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X}^\top\right)\right) \\ &= \sigma^2\left(n-\text{tr}\left((\mathbf{X^\top X})^{-1}\mathbf{X}^\top\mathbf{X}\right)\right) \\ &= \sigma^2\left(n-\text{tr}\left(\mathbf{I}_{p+1}\right)\right) \\ &= \sigma^2\left(n-(p+1)\right) \end{align*} \]
Another way of showing this is to use the properties of the trace operator and the fact that \(\mathbf{E} = (\mathbf{I}_n-\mathbf{P})\pmb{\varepsilon}\): \[ \begin{align} \mathbb{E}\left[\mathbf{E^\top E}\right] &= \mathbb{E}\left[\left(\left(\mathbf{I}_n-\mathbf{P}\right)\pmb{\varepsilon}\right)^\top\left(\mathbf{I}_n-\mathbf{P}\right)\pmb{\varepsilon}\right] \\ &= \mathbb{E}\left[\pmb{\varepsilon}^\top\left(\mathbf{I}_n-\mathbf{P}\right)^\top\left(\mathbf{I}_n-\mathbf{P}\right)\pmb{\varepsilon}\right] \\ &= \mathbb{E}\left[\pmb{\varepsilon}^\top\left(\mathbf{I}_n-\mathbf{P}\right)\pmb{\varepsilon}\right]\quad \scriptsize{\text{call } \mathbf{M}=\mathbf{I}_n-\mathbf{P}}\\ &= \mathbb{E}\left[\pmb{\varepsilon}^\top\mathbf{M}\pmb{\varepsilon}\right] \\ &= \mathbb{E}\left[\sum_{i=1}^n\sum_{j=1}^n\varepsilon_i\varepsilon_j M_{ij}\right] \\ &= \sum_{i=1}^n\sum_{j=1}^nE\left[\varepsilon_i\varepsilon_j\right] M_{ij} \scriptsize{\qquad \text{since } \mathbb{E}[\varepsilon_i\varepsilon_j]=\mathrm{Cov}(\varepsilon_i,\varepsilon_j)=0 \text{ for } i\neq j}\\ &= \sum_{i=1}^nE\left[\varepsilon_i^2\right] M_{ii} \\ &= \sum_{i=1}^n\sigma^2 M_{ii} \\ &= \sigma^2\sum_{i=1}^n M_{ii} \\ &= \sigma^2 \mathrm{tr}\left(\mathbf{M}\right) \\ &= \sigma^2 \mathrm{tr}\left(\mathbf{I}_n-\mathbf{P}\right)\\ &= \sigma^2\left(\mathrm{tr}\left(\mathbf{I}_n\right) - \mathrm{tr}\left(\mathbf{P}\right)\right) \\ &= \sigma^2\left(\mathrm{tr}\left(\mathbf{I}_n\right) - \mathrm{tr}\left(\mathbf{X}(\mathbf{X^\top X})^{-1}\mathbf{X}^\top\right)\right) \\ &= \sigma^2\left(n - \mathrm{tr}\left((\mathbf{X^\top X})^{-1}\mathbf{X}^\top\mathbf{X}\right)\right) \\ &= \sigma^2\left(n - \mathrm{tr}\left(\mathbf{I}_{p+1}\right)\right) \\ &= \sigma^2\left(n-(p+1)\right) \end{align} \]

Therefore, \[ \mathbb{E}\left[\hat{\sigma}^2\right] = \frac{\mathbb{E}\left[\mathbf{E^\top E}\right]}{n-(p+1)} = \frac{\sigma^2\left(n-(p+1)\right)}{n-(p+1)} = \sigma^2 \;. \]

This shows that the residual mean square is an unbiased estimator of \(\sigma^2\).