STAT 306 - Module 07
So far, we’ve only considered the simple linear regression with one covariate. In practice, we often have more than one covariate of interest.
When we have more than one covariate, we call the model a multiple linear regression.
The theory underlying the simple linear model naturally generalizes to multiple linear regression.
However, in multiple linear regression, it’s more convenient to express our linear model in terms of vectors and matrices.
Before tackling multiple covariates, let’s first see how to express the familiar simple linear regression model in matrix form!
We have formulated the simple linear model equation as: \[ Y_i = \beta_0 + \beta_1x_i + \varepsilon_i \;, \qquad i=1,\dotsc,n \;, \] with error assumptions \(\mathbb{E}[\varepsilon_i] = 0\), \(\operatorname{Var}(\varepsilon_i) = \sigma^2\), and \(\operatorname{Cov}(\varepsilon_i, \varepsilon_j) = 0\) for \(i \ne j\).
How can we write this in terms of vectors and matrices?
\[ \begin{align} \begin{bmatrix} Y_1 \\ Y_2 \\ \vdots \\ Y_n \end{bmatrix} &= \begin{bmatrix} \beta_0 + \beta_1x_1 + \varepsilon_1 \\ \beta_0 + \beta_1x_2 + \varepsilon_2 \\ \vdots \\ \beta_0 + \beta_1x_n + \varepsilon_n \end{bmatrix}\\ \ \\ &= \begin{bmatrix} \beta_0 + \beta_1x_1\\ \beta_0 + \beta_1x_2\\ \vdots \\ \beta_0 + \beta_1x_n\end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \vdots \\ \varepsilon_n \end{bmatrix}\\ \ \\ &=\begin{bmatrix} 1 & x_1\\ 1 & x_2\\ \vdots & \vdots \\ 1 & x_n\end{bmatrix}\begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix} + \begin{bmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \vdots \\ \varepsilon_n \end{bmatrix} \end{align} \]
\[ \mathbf{Y} = \begin{bmatrix} Y_1 \\ Y_2 \\ \vdots \\ Y_n \end{bmatrix}_{n \times 1} \; \mathbf{X} = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix}_{n \times 2} \; \pmb{\beta} = \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix}_{2 \times 1} \; \pmb{\varepsilon} = \begin{bmatrix} \varepsilon_1 \\ \varepsilon_2 \\ \vdots \\ \varepsilon_n \end{bmatrix}_{n \times 1} \; \]
Note that the first column of \(\mathbf{X}\) is a column of \(1\)’s.
The matrix \(\mathbf{X}\) is sometimes referred to as the design matrix.
The error assumptions in vector/matrix form are: \[\mathbb{E}[\pmb{\varepsilon}] = \mathbf{0} \qquad \text{and} \qquad \operatorname{Cov}(\pmb{\varepsilon}) = \sigma^2\mathbf{I}_n \;,\] where \(\mathbf{I}_n\) is the \(n\times n\) identity matrix.
\[ \begin{align} n\hat{\beta}_0 + \hat{\beta}_1\sum_{i=1}^nx_i &= \sum_{i=1}^ny_i \;, \\ \hat{\beta}_0\sum_{i=1}^nx_i+\hat{\beta}_1\sum_{i=1}^nx_i^2 &= \sum_{i=1}^nx_iy_i \;. \end{align} \]
Which of the following matrix equations represents the normal equations?
\[\begin{align}n\hat\beta_0 + \hat\beta_1\sum_{i=1}^n x_i &= \sum_{i=1}^n Y_i \;,\\ \hat\beta_0\sum_{i=1}^n x_i + \hat\beta_1\sum_{i=1}^n x_i^2 &= \sum_{i=1}^n x_iY_i \;.\end{align}\]
Which of the following matrix equations represents the normal equations?
\[\begin{align}n\hat\beta_0 + \hat\beta_1\sum_{i=1}^n x_i &= \sum_{i=1}^n Y_i \;,\\ \hat\beta_0\sum_{i=1}^n x_i + \hat\beta_1\sum_{i=1}^n x_i^2 &= \sum_{i=1}^n x_iY_i \;.\end{align}\]
\[ \begin{align} \begin{bmatrix} n\hat{\beta}_0 + \hat{\beta}_1\sum_{i=1}^nx_i\\ \hat{\beta}_0\sum_{i=1}^nx_i+\hat{\beta}_1\sum_{i=1}^nx_i^2 \end{bmatrix} &= \begin{bmatrix} \sum_{i=1}^n y_i \\ \sum_{i=1}^n x_iy_i \end{bmatrix} \\ \ \\ \begin{bmatrix} n & \sum_{i=1}^nx_i \\ \sum_{i=1}^nx_i & \sum_{i=1}^nx_i^2 \end{bmatrix} \begin{bmatrix} \hat\beta_0 \\ \hat\beta_1 \end{bmatrix} &= \begin{bmatrix} 1 & 1 & \ldots & 1 \\ x_1 & x_2 & \ldots & x_n \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}\\ \ \\ \begin{bmatrix} 1 & 1 & \ldots & 1 \\ x_1 & x_2 & \ldots & x_n \end{bmatrix} \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix} \begin{bmatrix} \hat\beta_0 \\ \hat\beta_1 \end{bmatrix} &= \begin{bmatrix} 1 & 1 & \ldots & 1 \\ x_1 & x_2 & \ldots & x_n \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}\\ \mathbf{X}^\top\mathbf{X}\hat\beta &= \mathbf{X}^\top\mathbf{y} \;. \end{align} \]
If \(\mathbf{X}^\top\mathbf{X}\) is invertible, then by the normal equations \[ \begin{align} \mathbf{X}^\top\mathbf{X}\hat\beta &= \mathbf{X}^\top\mathbf{y} \Leftrightarrow\\ (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{X}\hat\beta &= (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y} \Leftrightarrow\\ \hat\beta &= (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y} \end{align} \]
In other words, our least squares estimate is given by: \[\hat\beta =(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}\]
When is \(\mathbf{X}^\top\mathbf{X}\) not invertible?
\[\mathbf{X}^\top\mathbf{X} = \begin{bmatrix}n & \sum_{i=1}^n x_i\\\sum_{i=1}^n x_i & \sum_{i=1}^n x_i^2\end{bmatrix}\]
Hint: A \(2\times 2\) matrix \(\mathbf{A}=\begin{bmatrix}a & b\\ c & d\end{bmatrix}\) has determinant \(\det(\mathbf{A}) = ad-bc\), and \(\mathbf{A}^{-1} = \dfrac{1}{\det(\mathbf{A})}\begin{bmatrix}d & -b\\ -c & a\end{bmatrix}\).
When is \(\mathbf{X}^\top\mathbf{X}\) not invertible?
\[\mathbf{X}^\top\mathbf{X} = \begin{bmatrix}n & \sum_{i=1}^n x_i\\\sum_{i=1}^n x_i & \sum_{i=1}^n x_i^2\end{bmatrix}\]
Hint: A \(2\times 2\) matrix \(\mathbf{A}=\begin{bmatrix}a & b\\ c & d\end{bmatrix}\) has determinant \(\det(\mathbf{A}) = ad-bc\), and \(\mathbf{A}^{-1} = \dfrac{1}{\det(\mathbf{A})}\begin{bmatrix}d & -b\\ -c & a\end{bmatrix}\).
In other words, \(\mathbf{X}^\top\mathbf{X}\) is not invertible when \(x_1=x_2=\dotsc=x_n\).
Intuitively, this makes sense: if all the covariate values in our data are the same, then it’s not possible to observe a trend between the covariate and the response. See Exercise 4.
For simple linear regression, this means the two columns of \(\mathbf{X}\) must be linearly independent, which requires having at least two distinct covariate values in our data (and \(n \ge 2\)).
For which of the following design matrices is \(\mathbf{X}^\top\mathbf{X}\) invertible? (Assume \(n \ge 2\) and \(c \ne 0\))
\[\mathbf{X}_A = \begin{bmatrix}1 & c\\1 & 0\\\vdots & \vdots\\1 & 0\end{bmatrix}\qquad \mathbf{X}_B = \begin{bmatrix}1 & c\\1 & c\\\vdots & \vdots\\1 & c\end{bmatrix}\]
\[\mathbf{X}_C = \begin{bmatrix}1 & 0\\1 & 0\\\vdots & \vdots\\1 & 0\end{bmatrix}\qquad \mathbf{X}_D = \begin{bmatrix}1 & 1/c\\1 & 1/c\\\vdots & \vdots\\1 & 1/c\end{bmatrix}\]
For which of the following design matrices is \(\mathbf{X}^\top\mathbf{X}\) invertible? (Assume \(n \ge 2\) and \(c \ne 0\))
\[\mathbf{X}_A = \begin{bmatrix}1 & c\\1 & 0\\\vdots & \vdots\\1 & 0\end{bmatrix}\qquad \mathbf{X}_B = \begin{bmatrix}1 & c\\1 & c\\\vdots & \vdots\\1 & c\end{bmatrix}\]
\[\mathbf{X}_C = \begin{bmatrix}1 & 0\\1 & 0\\\vdots & \vdots\\1 & 0\end{bmatrix}\qquad \mathbf{X}_D = \begin{bmatrix}1 & 1/c\\1 & 1/c\\\vdots & \vdots\\1 & 1/c\end{bmatrix}\]
© 2025 Rodolfo Lourenzutti – Material Licensed under CC By-SA 4.0