Practice Exercises

import { d3_createScatterPlot } from "./scripts/d3plots.js"
  import { d3_createScatterPlotWithLine } from "./scripts/d3plots.js"
  import { sumOfSquaredResiduals } from './scripts/d3plots.js'
  import { d3_createNormalDensityPlot } from './scripts/d3plots.js'

1. Formulae Reminder

Recall the following formulae from introductory statistics:

Sample Mean: $\bar{X} = \frac{1}{n} \displaystyle\sum_{i=1}^{n} X_i$
Sample Variance: $\displaystyle S^2_X = \frac{1}{n-1} \sum_{i=1}^{n} (X_i - \bar{X})^2$
Sample Standard Deviation: $\displaystyle S_X = \sqrt{S^2_X}$
Sample Covariance: $\displaystyle \text{cov}(X, Y) = S_{XY} = \frac{1}{n-1} \sum_{i=1}^{n} (X_i - \bar{X})(Y_i - \bar{Y})$
Sample Correlation: $\displaystyle \text{cor}(X, Y) = r_{XY} = \frac{S_{XY}}{S_X S_Y}$

Exercise 1: Some Useful Identities

Prove the following equations:

(a) Prove that $(n-1)S^2_X = \displaystyle\sum_{i=1}^{n} X_i^2 - n\bar{X}^2$.

Hint

Expand the squared term $(X_i - \bar{X})^2 = X_i^2 - 2X_i\bar{X} + \bar{X}^2$. Then distribute the summation, noting that $\bar{X}$ is a constant with respect to the summation index $i$, so $\sum_{i=1}^n \bar{X}^2 = n\bar{X}^2$ and $\sum_{i=1}^n X_i = n\bar{X}$.

Solution

We start with the definition of sample variance:

\[ \begin{aligned} (n-1)S_X^2 &= \sum_{i=1}^{n} (X_i - \bar{X})^2 \\ &= \sum_{i=1}^{n} (X_i^2 - 2X_i\bar{X} + \bar{X}^2) \\ &= \sum_{i=1}^{n} X_i^2 - \sum_{i=1}^{n} 2X_i\bar{X} + \sum_{i=1}^{n} \bar{X}^2 \\ &= \sum_{i=1}^{n} X_i^2 - 2\bar{X}\sum_{i=1}^{n} X_i + n\bar{X}^2 \end{aligned} \]

We know that $\bar{X} = \frac{\sum_{i=1}^{n} X_i}{n} \implies \sum_{i=1}^{n} X_i = n\bar{X}$. Substituting this into the equation:

\[ \begin{aligned} (n-1)S_X^2 &= \sum_{i=1}^{n} X_i^2 - 2\bar{X}\sum_{i=1}^{n} X_i + n\bar{X}^2 \\ &= \sum_{i=1}^{n} X_i^2 - 2\bar{X}(n\bar{X}) + n\bar{X}^2 \\ &= \sum_{i=1}^{n} X_i^2 - 2n\bar{X}^2 + n\bar{X}^2 \\ &= \sum_{i=1}^{n} X_i^2 - n\bar{X}^2 \quad \blacksquare \end{aligned} \]

(b) Prove that $S_{XY} = \frac{1}{n-1} \left( \displaystyle\sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y} \right)$, or equivalently, $(n-1)S_{XY} = \displaystyle\sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y}$.

Hint

Expand the product $(X_i - \bar{X})(Y_i - \bar{Y}) = X_i Y_i - X_i\bar{Y} - \bar{X}Y_i + \bar{X}\bar{Y}$. Distribute the summation over each term, factor out constants $\bar{X}$ and $\bar{Y}$, and substitute $\sum_{i=1}^n X_i = n\bar{X}$ and $\sum_{i=1}^n Y_i = n\bar{Y}$.

Solution

We start with the definition of sample covariance:

\[ \begin{aligned} S_{XY} &= \frac{1}{n-1}\sum_{i=1}^{n} (X_i - \bar{X})(Y_i - \bar{Y}) \\ &= \frac{1}{n-1}\sum_{i=1}^{n} (X_i Y_i - X_i\bar{Y} - \bar{X}Y_i + \bar{X}\bar{Y}) \\ &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - \sum_{i=1}^{n} X_i\bar{Y} - \sum_{i=1}^{n} \bar{X}Y_i + \sum_{i=1}^{n} \bar{X}\bar{Y}\right) \\ &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - \bar{Y}\sum_{i=1}^{n} X_i - \bar{X}\sum_{i=1}^{n} Y_i + n\bar{X}\bar{Y}\right) \end{aligned} \]

Substituting $\sum_{i=1}^{n} X_i = n\bar{X}$ and $\sum_{i=1}^{n} Y_i = n\bar{Y}$:

\[ \begin{aligned} S_{XY} &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - \bar{Y}(n\bar{X}) - \bar{X}(n\bar{Y}) + n\bar{X}\bar{Y}\right) \\ &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y} - n\bar{X}\bar{Y} + n\bar{X}\bar{Y}\right) \\ &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y}\right) \end{aligned} \]

Multiplying both sides by $n - 1$ gives the unscaled form:

\[(n-1)S_{XY} = \sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y} \quad \blacksquare\]

2. Simple Linear Regression

Exercise 2.1 - Least Squares Estimation

In this exercise you will find the line that best fits a data set. Suppose we have $n$ points in $\mathbb{R}^2$ given by $(x_1, Y_1), (x_2, Y_2), \ldots, (x_n, Y_n)$.

d3_createScatterPlot({
    elementId: 'just-points',
    xName: 'x',
    yName: 'y',
    data: data_test, 
    title: `Plot` , 
    xlab: 'Explanatory Variable', 
    ylab: "Response Variable", 
    titleFontSize: "24px", 
    labelFontSize: "20px", 
    tickFontSize: '16px', 
    pointSize: 3, 
    pointColor: 'steelblue', 
    margin: {top: 80, right: 20, bottom: 50, left: 80}
  });

Any straight line can be defined by the equation $Y = b_0 + b_1x$, where $b_0$ is the $y$-intercept and $b_1$ is the slope. The challenge is to find the optimal line, i.e., the optimal values of $b_0$ and $b_1$.

To do this, we first need a way to measure how “wrong” a particular line is. For any single data point $(x_i, Y_i)$, our line predicts the value $\widehat{Y}_i = b_0 + b_1 x_i$. The error for that point, called residual, is the difference between the actual observed value $Y_i$ and the predicted value $\widehat{Y}_i$, i.e., the residual of a point $(x_i, Y_i)$ is given by $e_i = Y_i - \widehat{Y}_i = Y_i - \left(b_0 + b_1x_i\right)$. The residual is the vertical difference between the point and the line.

We want to make the total error as small as possible. If we just added up all the residuals, the positive errors (points above the line) and negative errors (points below the line) would cancel each other out, which isn’t helpful.

To solve this, we square each residual before adding them up. This has two great benefits:

All the error terms become positive.
It penalizes larger errors more heavily (well, this can also be a drawback – more on this later).

This gives us the Residual Sum of Squares (RSS): \[RSS(b_0, b_1) = \sum_{i=1}^{n} (Y_i - (b_0 + b_1 x_i))^2\]

The “best” line is the one that makes this total squared error as small as possible. This is the method of least squares.

In this case, what values of $b_0$ and $b_1$ minimize RSS? Use the sliders below to choose these values.

viewof intercept = {
    let input = Inputs.range([-1, 13], 
                             {
                                value: 8,
                                step: .01, 
                                label: "Intercept: ",
                                width: 300
                             });
         
    d3.select(input).select("label")._groups[0][0].innerHTML = 'β<sub>0</sub>: ';
    
    return input
}

viewof slope = {
    let input = Inputs.range([-5, 5], 
                             {value: 0,
                             step: .01, 
                             label: "Slope: ", width: 300});
    
    d3.select(input).select("label")._groups[0][0].innerHTML = 'β<sub>1</sub>:';
    
    return input
}

Naturally, we don’t want to do this manually. Instead, we can use calculus to find the optimal values of $b_0$ and $b_1$. In this exercise we will find the values of $b_0$ and $b_1$ that minimize $RSS$.

Exercise 2.1.1

Find the partial derivatives of $RSS$ with respect to $b_0$ and $b_1$.

Hint

Use the chain rule.

Solution

The $RSS$ is given by:

\[RSS(b_0, b_1) = \sum_{i=1}^{n} (Y_i - b_0 - b_1 x_i)^2\]

The partial derivative with respect to $b_0$ is:

\[ \begin{aligned} \frac{\partial RSS}{\partial b_0} &= \sum_{i=1}^{n} 2(Y_i - b_0 - b_1 x_i)(-1) \\ &= -2\sum_{i=1}^{n} (Y_i - b_0 - b_1 x_i) \end{aligned} \]

The partial derivative with respect to $b_1$ is:

\[ \begin{aligned} \frac{\partial RSS}{\partial b_1} &= \sum_{i=1}^{n} 2(Y_i - b_0 - b_1 x_i)(-x_i) \\ &= -2\sum_{i=1}^{n} x_i(Y_i - b_0 - b_1 x_i) \quad \blacksquare \end{aligned} \]

Exercise 2.1.2

Set the partial derivatives equal to zero and solve $\frac{\partial RSS}{\partial b_0} = 0$ for $\widehat{\beta}_0$ in terms of $\widehat{\beta}_1$.

Hint

When you set the partial derivatives equal to zero, the solutions are the estimators $\widehat{\beta}_0$ and $\widehat{\beta}_1$, so replace $b_0$ with $\widehat{\beta}_0$ and $b_1$ with $\widehat{\beta}_1$. Then divide both sides by $-2$ and split the sum into three separate summations: $\sum Y_i$, $\sum \widehat{\beta}_0$, and $\sum \widehat{\beta}_1 x_i$.

Solution

Set $\frac{\partial RSS}{\partial b_0} = 0$:

\[ \begin{aligned} -2\sum_{i=1}^{n} (Y_i - \widehat{\beta}_0 - \widehat{\beta}_1 x_i) &= 0 \\ \sum_{i=1}^{n} Y_i - \sum_{i=1}^{n} \widehat{\beta}_0 - \sum_{i=1}^{n} \widehat{\beta}_1 x_i &= 0 \\ n\bar{Y} - n\widehat{\beta}_0 - \widehat{\beta}_1(n\bar{x}) &= 0 \\ \bar{Y} - \widehat{\beta}_0 - \widehat{\beta}_1\bar{x} &= 0 \end{aligned} \]

Solving for $\widehat{\beta}_0$:

\[\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1\bar{x} \quad \blacksquare\]

Rearranging this result gives:

\[\bar{Y} = \widehat{\beta}_0 + \widehat{\beta}_1\bar{x}\]

This establishes a fundamental geometric property of ordinary least squares regression: the fitted regression line always passes through the point of averages (the centroid), $(\bar{x}, \bar{Y})$.

Exercise 2.1.3

For the equation $\frac{\partial RSS}{\partial b_1} = 0$, replace the value of $\widehat{\beta}_0$ from Exercise 2.1.2 into the equation and solve for $\widehat{\beta}_1$.

Hint

Divide by $-2$ and expand the summand: $x_i(Y_i - \widehat{\beta}_0 - \widehat{\beta}_1 x_i) = x_i Y_i - x_i\widehat{\beta}_0 - \widehat{\beta}_1 x_i^2$. Replace $\widehat{\beta}_0$ with $(\bar{Y} - \widehat{\beta}_1\bar{x})$, collect all terms involving $\widehat{\beta}_1$ on one side of the equation, factor out $\widehat{\beta}_1$, and divide.

Solution

Set $\frac{\partial RSS}{\partial b_1} = 0$:

\[ \begin{aligned} -2\sum_{i=1}^{n} x_i(Y_i - \widehat{\beta}_0 - \widehat{\beta}_1 x_i) &= 0 \\ \sum_{i=1}^{n} (x_i Y_i - x_i\widehat{\beta}_0 - \widehat{\beta}_1 x_i^2) &= 0 \\ \sum_{i=1}^{n} x_i Y_i - \widehat{\beta}_0\sum_{i=1}^{n} x_i - \widehat{\beta}_1\sum_{i=1}^{n} x_i^2 &= 0 \end{aligned} \]

Substitute $\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1\bar{x}$:

\[ \begin{aligned} \sum_{i=1}^{n} x_i Y_i - (\bar{Y} - \widehat{\beta}_1\bar{x})\sum_{i=1}^{n} x_i - \widehat{\beta}_1\sum_{i=1}^{n} x_i^2 &= 0 \\ \sum_{i=1}^{n} x_i Y_i - \bar{Y}(n\bar{x}) + \widehat{\beta}_1\bar{x}(n\bar{x}) - \widehat{\beta}_1\sum_{i=1}^{n} x_i^2 &= 0 \\ \sum_{i=1}^{n} x_i Y_i - n\bar{x}\bar{Y} &= \widehat{\beta}_1\left(\sum_{i=1}^{n} x_i^2 - n\bar{x}^2\right) \end{aligned} \]

Solving for $\widehat{\beta}_1$:

\[\widehat{\beta}_1 = \frac{\sum_{i=1}^{n} x_i Y_i - n\bar{x}\bar{Y}}{\sum_{i=1}^{n} x_i^2 - n\bar{x}^2} \quad \blacksquare\]

(Verification of Global Minimum): Setting the first derivatives to zero finds the stationary point $(\widehat{\beta}_0, \widehat{\beta}_1)$. To verify that this point minimizes $RSS$, consider the Hessian matrix of second-order partial derivatives:

\[ H = \begin{pmatrix} \frac{\partial^2 RSS}{\partial b_0^2} & \frac{\partial^2 RSS}{\partial b_0 \partial b_1} \\ \frac{\partial^2 RSS}{\partial b_1 \partial b_0} & \frac{\partial^2 RSS}{\partial b_1^2} \end{pmatrix} = \begin{pmatrix} 2n & 2\sum_{i=1}^{n} x_i \\ 2\sum_{i=1}^{n} x_i & 2\sum_{i=1}^{n} x_i^2 \end{pmatrix} \]

The first leading principal minor is $2n > 0$. The determinant of $H$ is:

\[ \det(H) = (2n)\left(2\sum_{i=1}^{n} x_i^2\right) - \left(2\sum_{i=1}^{n} x_i\right)^2 = 4n\sum_{i=1}^{n} x_i^2 - 4(n\bar{x})^2 = 4n\left(\sum_{i=1}^{n} x_i^2 - n\bar{x}^2\right) = 4n(n-1)S_x^2 \]

Whenever not all $x_i$ are identical ($S_x^2 > 0$), $\det(H) > 0$. Because both the leading principal minor and the determinant are strictly positive, $H$ is strictly positive definite everywhere. Thus, $RSS(b_0, b_1)$ is strictly convex, and $(\widehat{\beta}_0, \widehat{\beta}_1)$ is the unique global minimum.

Exercise 2.1.4

Using the equalities from Exercise 1, show that you can write $\widehat{\beta}_1$ as $\widehat{\beta}_1 = r_{XY} \frac{S_Y}{S_X}$.

Hint

Notice that from Exercise 1, the numerator of $\widehat{\beta}_1$ is $(n-1)S_{XY}$ and the denominator is $(n-1)S_X^2$. Recall the definition of the correlation coefficient $r_{XY} = \frac{S_{XY}}{S_X S_Y}$, so $S_{XY} = r_{XY} S_X S_Y$.

Solution

From Exercise 1, we have:

$(n-1)S_X^2 = \sum_{i=1}^{n} X_i^2 - n\bar{X}^2$
$(n-1)S_{XY} = \sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y}$

Substituting these into the expression for $\widehat{\beta}_1$:

\[ \widehat{\beta}_1 = \frac{(n-1)S_{XY}}{(n-1)S_X^2} = \frac{S_{XY}}{S_X^2} \]

The correlation coefficient is defined as $r_{XY} = \frac{S_{XY}}{S_X S_Y}$. Therefore, $S_{XY} = r_{XY} S_X S_Y$. Substituting this into the expression for $\widehat{\beta}_1$:

\[ \widehat{\beta}_1 = \frac{r_{XY} S_X S_Y}{S_X^2} = r_{XY}\frac{S_Y}{S_X} \quad \blacksquare \]

Here are some different (but equivalent) ways to write $\widehat{\beta}_1$:

\[ \widehat{\beta}_1 = \frac{\sum_{i=1}^{n} x_i Y_i - n\bar{x}\bar{Y}}{\sum_{i=1}^{n} x_i^2 - n\bar{x}^2} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})Y_i}{\sum_{i=1}^{n} (x_i - \bar{x})^2} = r_{XY}\frac{S_Y}{S_X} \]

Why can we drop $\bar{Y}$ in the numerator? Notice that:

\[\sum_{i=1}^{n} (x_i - \bar{x})(Y_i - \bar{Y}) = \sum_{i=1}^{n} (x_i - \bar{x})Y_i - \bar{Y}\sum_{i=1}^{n} (x_i - \bar{x})\]

Since $\sum_{i=1}^{n} (x_i - \bar{x}) = \sum x_i - n\bar{x} = 0$, the second term drops out entirely.

Writing $\widehat{\beta}_1$ as $\frac{\sum_{i=1}^{n}(x_i - \bar{x})Y_i}{\sum_{i=1}^{n}(x_i - \bar{x})^2}$ shows that $\widehat{\beta}_1$ is a linear combination of the response observations, $\widehat{\beta}_1 = \sum_{i=1}^{n} w_i Y_i$, with deterministic weights $w_i = \frac{x_i - \bar{x}}{\sum (x_i - \bar{x})^2}$. This linear form is the cornerstone of variance calculations and the Gauss-Markov theorem later in the course.

Exercise 2.1.5

The data in the plot can be loaded into R using the data frame provided in the code block below.

Use this data to compute the numerical values of $\widehat{\beta}_0$ and $\widehat{\beta}_1$. Once you calculate both values, plug them into the sliders in the plot above to find the corresponding RSS!

📋 Solution

Method 1: Using the derived summary-statistic formulas

beta_1_hat <- cor(data$x, data$y) * (sd(data$y) / sd(data$x))
beta_0_hat <- mean(data$y) - beta_1_hat * mean(data$x)

print(beta_1_hat)
# [1] 1.630748

print(beta_0_hat)
# [1] 4.791133

Method 2: Verifying with R’s built-in lm() function

model <- lm(y ~ x, data = data)
coef(model)
# (Intercept)           x 
#    4.791133    1.630748

Exercise 2.1.6

This time, compute the Residual Sum of Squares (RSS) yourself in R. Does your computed value match the one in the interactive plot?

📋 Solution

y_hat <- beta_0_hat + beta_1_hat * data$x
rss <- sum((data$y - y_hat)^2)

cat("Exact RSS:   ", rss, "\n")
cat("Rounded RSS: ", round(rss, 3), "\n")
# Exact RSS:    67.69049 
# Rounded RSS:  67.691

Notice that the exact minimum RSS is $\approx 67.6905$. In the interactive plot, the sliders step by $0.01$, rounding the parameters to $\beta_0 = 4.79$ and $\beta_1 = 1.63$. Evaluating the RSS at those rounded values gives:

\[RSS(4.79, 1.63) \approx 67.6907 \approx 67.691\]

which matches the title in the interactive plot.

Exercise 2.2 - Distribution of $Y_i$

Consider the following linear regression model:

\[ Y_i = \beta_0 + \beta_1 x_i + \epsilon_i \tag{1}\]

where $\epsilon_i \sim N(0, \sigma^2)$ are independent and identically distributed normal random variables.

Exercise 2.2.1

Calculate the expectation and variance of $Y_i$.

Hint

Remember, given $X_i = x_i$, the values $x_i$, $\beta_0$, and $\beta_1$ are deterministic (fixed constants).

Solution

The conditional expected value of $Y_i$ given $X_i = x_i$ is:

\[ E[Y_i] = E[\beta_0 + \beta_1 x_i + \epsilon_i] = E[\beta_0] + E[\beta_1 x_i] + E[\epsilon_i] = \beta_0 + \beta_1 x_i + 0 = \beta_0 + \beta_1 x_i \]

The conditional variance of $Y_i$ given $X_i = x_i$ is:

\[ \text{Var}(Y_i) = \text{Var}(\beta_0 + \beta_1 x_i + \epsilon_i) = \text{Var}(\epsilon_i) = \sigma^2 \quad \blacksquare \]

Exercise 2.2.2

What is the distribution of $Y_i$?

Hint

Recall that any linear transformation of a normally distributed random variable is also normally distributed.

Solution

Since $\epsilon_i \sim N(0, \sigma^2)$ and $Y_i$ is a linear transformation of $\epsilon_i$, $Y_i$ is also normally distributed:

\[ Y_i \sim N(\beta_0 + \beta_1 x_i, \sigma^2) \quad \blacksquare \]

Exercise 2.3 - LSE: Infinite Solutions

Let $(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)$ be a sample of size $n$ of $X$ and $Y$. Suppose $x_i = c$ for all $i = 1, \ldots, n$.

Exercise 2.3.1

Derive the least squares estimators of $\beta_0$ and $\beta_1$ in this case considering the model Equation 1.

Hint

Substitute $x_i = c$ into the $RSS(b_0, b_1)$ objective function. Differentiate with respect to $b_0$ and $b_1$, set the partial derivatives to zero, and examine the resulting system of equations. Do you obtain two independent equations, or two copies of the same relationship?

Solution

The Residual Sum of Squares (RSS) is given by:

\[ RSS(b_0, b_1) = \sum_{i=1}^{n} (y_i - b_0 - b_1 x_i)^2 \]

Since $x_i = c$ for all $i$, we have:

\[ RSS(b_0, b_1) = \sum_{i=1}^{n} (y_i - b_0 - b_1 c)^2 \]

To find the least squares estimators, we take the partial derivatives with respect to $b_0$ and $b_1$ and set them to zero:

\[ \begin{aligned} \frac{\partial RSS}{\partial b_0} = 0 \implies -2\sum_{i=1}^{n} (y_i - \widehat{\beta}_0 - \widehat{\beta}_1 c) &= 0 \\ \sum_{i=1}^{n} y_i - n\widehat{\beta}_0 - nc\widehat{\beta}_1 &= 0 \\ n\bar{y} - n\widehat{\beta}_0 - nc\widehat{\beta}_1 &= 0 \\ \bar{y} - \widehat{\beta}_0 - c\widehat{\beta}_1 &= 0 \end{aligned} \]

which leads to $\widehat{\beta}_0 + c\widehat{\beta}_1 = \bar{y}$.

Next, taking the partial derivative with respect to $b_1$:

\[ \begin{aligned} \frac{\partial RSS}{\partial b_1} = 0 \implies -2c\sum_{i=1}^{n} (y_i - \widehat{\beta}_0 - \widehat{\beta}_1 c) &= 0 \\ \bar{y} - \widehat{\beta}_0 - c\widehat{\beta}_1 &= 0 \end{aligned} \]

which gives the exact same equation: $\widehat{\beta}_0 + c\widehat{\beta}_1 = \bar{y}$.

Since both partial derivatives lead to the same equation, we have a single equation with two unknowns. This means there is no unique solution for $\widehat{\beta}_0$ and $\widehat{\beta}_1$. Any pair of estimators $(\widehat{\beta}_0, \widehat{\beta}_1)$ that satisfies the condition:

\[\widehat{\beta}_0 + c\widehat{\beta}_1 = \bar{y}\]

will minimize the RSS.

This result is intuitive: when all $x_i$ are the same, the data contains zero variability in $X$ ($S_X^2 = 0$), providing no information to estimate a slope. Any straight line passing through the point of averages $(c, \bar{Y})$ is a valid least squares regression line and achieves the exact same minimal RSS. $\blacksquare$

Exercise 2.3.2

Suppose $c=1$ and $\bar{Y} = 10$. Draw the graph of the regression line using $\widehat{\beta}_1 = 10$ and another graph using $\widehat{\beta}_1 = 0$. What do you notice?

Hint

Plug $c=1$ and $\bar{Y}=10$ into the constraint $\widehat{\beta}_0 + c\widehat{\beta}_1 = \bar{Y}$. For each candidate slope $\widehat{\beta}_1$, solve for the corresponding intercept $\widehat{\beta}_0$, write the line equation $y = \widehat{\beta}_0 + \widehat{\beta}_1 x$, and check where both lines intersect.

Solution

If $c = 1$ and $\bar{Y} = 10$, the condition is:

\[\widehat{\beta}_0 + \widehat{\beta}_1 = 10\]

If we choose $\widehat{\beta}_1 = 10$, then $\widehat{\beta}_0 = 10 - 10 = 0$, and the regression line is $y = 10x$.
If we choose $\widehat{\beta}_1 = 0$, then $\widehat{\beta}_0 = 10 - 0 = 10$, and the regression line is $y = 10$.
If we choose $\widehat{\beta}_1 = -10$, then $\widehat{\beta}_0 = 10 - (-10) = 20$, and the regression line is $y = 20 - 10x$.

Both lines (and indeed infinitely many other lines satisfying the constraint) pass through the point $(1, 10)$, which is $(c, \bar{Y})$. When all $x_i$ are identical, the data provides no information about how $Y$ changes as $X$ varies. Any line pivoting through the centroid $(c, \bar{Y})$ fits the data equally well and achieves the same minimum RSS.

Scatter plot of Y vs X when X is constant (c = 1, Ȳ = 10) with three alternative least squares regression lines.

The plot below shows the surface of the Residual Sum of Squares (RSS) as a function of $\beta_0$ and $\beta_1$. This is what the objective function you minimized in Exercise 2.1 looks like when $c=1$ and $\bar{Y} = 10$. The red line indicates the theoretical minimum of the RSS, which occurs when $\beta_0 = \bar{Y} - c\beta_1$. As you can see, the RSS does not have a unique minimum because $X$ is constant. Instead, there is a valley of minima along which the RSS is minimized.

Figure 1: Surface plot of the Sum of Squared Residuals (SSR) as a function of b_0 and b_1.

Exercise 2.4 - LSE: Dummy Variables

Let $Y$ be a numerical quantitative variable being measured across two groups. For example, $Y$ could be the number of red blood cells in a blood sample measured from patients with leukemia and healthy patients. If we assign $X=1$ for patients with leukemia and $X=0$ for healthy patients, then we have two groups defined by the binary variable $X$.

Let $(X_1, Y_1), (X_2, Y_2), \ldots, (X_n, Y_n)$ be a sample of size $n$ of $X$ and $Y$. Suppose $X$ is a dummy variable, i.e., $X_i \in \{0, 1\}$ for $i=1, \ldots, n$.

Show that the least squares estimator of $\beta_1$ is given by $\widehat{\beta}_1 = \bar{Y}_1 - \bar{Y}_0$, where $\bar{Y}_1$ is the mean of $Y$ for observations where $X=1$ and $\bar{Y}_0$ is the mean of $Y$ for observations where $X=0$.

Hint

Let $n_1$ be the number of observations with $X_i = 1$ and $n_0$ be the number of observations with $X_i = 0$, so $n = n_0 + n_1$. Evaluate the sums appearing in the normal equations: \[\sum_{i=1}^n X_i, \quad \sum_{i=1}^n X_i^2, \quad \sum_{i=1}^n Y_i, \quad \text{and} \quad \sum_{i=1}^n X_i Y_i\] in terms of $n_0, n_1, \bar{Y}_0$, and $\bar{Y}_1$. Then solve the two normal equations for $\widehat{\beta}_0$ and $\widehat{\beta}_1$.

Solution

The Residual Sum of Squares is given by:

\[ RSS(b_0, b_1) = \sum_{i=1}^{n} (y_i - b_0 - b_1 x_i)^2 \]

Setting the partial derivatives to zero yields the standard normal equations:

\[ \begin{aligned} n\widehat{\beta}_0 + \widehat{\beta}_1 \sum_{i=1}^{n} x_i &= \sum_{i=1}^{n} y_i \\ \widehat{\beta}_0 \sum_{i=1}^{n} x_i + \widehat{\beta}_1 \sum_{i=1}^{n} x_i^2 &= \sum_{i=1}^{n} x_i y_i \end{aligned} \]

Let $n_1$ be the number of observations with $x_i = 1$, and $n_0$ be the number of observations with $x_i = 0$. Since $x_i \in \{0, 1\}$:

$n = n_0 + n_1$
$\sum_{i=1}^{n} x_i = n_1$ (since $0$ contributes nothing and $1$ contributes $1$)
$\sum_{i=1}^{n} x_i^2 = n_1$ (since $0^2 = 0$ and $1^2 = 1$)
$\sum_{i=1}^{n} y_i = \sum_{i: x_i=0} y_i + \sum_{i: x_i=1} y_i = n_0 \bar{Y}_0 + n_1 \bar{Y}_1$
$\sum_{i=1}^{n} x_i y_i = \sum_{i: x_i=1} y_i = n_1 \bar{Y}_1$

From the second normal equation:

\[ \widehat{\beta}_0 n_1 + \widehat{\beta}_1 n_1 = n_1 \bar{Y}_1 \implies \widehat{\beta}_0 + \widehat{\beta}_1 = \bar{Y}_1 \implies \widehat{\beta}_0 = \bar{Y}_1 - \widehat{\beta}_1 \]

From the first normal equation:

\[ (n_0 + n_1)\widehat{\beta}_0 + n_1 \widehat{\beta}_1 = n_0 \bar{Y}_0 + n_1 \bar{Y}_1 \]

Substitute $\widehat{\beta}_0 = \bar{Y}_1 - \widehat{\beta}_1$:

\[ \begin{aligned} (n_0 + n_1)(\bar{Y}_1 - \widehat{\beta}_1) + n_1 \widehat{\beta}_1 &= n_0 \bar{Y}_0 + n_1 \bar{Y}_1 \\ n_0 \bar{Y}_1 - n_0 \widehat{\beta}_1 + n_1 \bar{Y}_1 - n_1 \widehat{\beta}_1 + n_1 \widehat{\beta}_1 &= n_0 \bar{Y}_0 + n_1 \bar{Y}_1 \\ n_0 \bar{Y}_1 - n_0 \widehat{\beta}_1 &= n_0 \bar{Y}_0 \\ \bar{Y}_1 - \widehat{\beta}_1 &= \bar{Y}_0 \\ \widehat{\beta}_1 &= \bar{Y}_1 - \bar{Y}_0 \quad \blacksquare \end{aligned} \]

Also, solving for the intercept:

\[ \widehat{\beta}_0 = \bar{Y}_1 - (\bar{Y}_1 - \bar{Y}_0) = \bar{Y}_0 \quad \blacksquare \]

Interpretation: In simple linear regression with a single binary dummy predictor $X \in \{0, 1\}$, the intercept $\widehat{\beta}_0 = \bar{Y}_0$ is the sample mean of the baseline reference group ($X = 0$), and the slope $\widehat{\beta}_1 = \bar{Y}_1 - \bar{Y}_0$ represents the difference between the two group sample means. Linear regression with a binary indicator is mathematically identical to a two-sample comparison of means.

Exercise 2.5 - LSE: Properties

Consider the simple linear regression model:

\[ Y_i = \beta_0 + \beta_1 x_i + \epsilon_i, \qquad i = 1, \dots, n \]

where the predictor values $x_1, \dots, x_n$ are fixed constants, and the errors $\epsilon_i$ satisfy $E[\epsilon_i] = 0$, $\text{Var}(\epsilon_i) = \sigma^2$, and $\text{Cov}(\epsilon_i, \epsilon_j) = 0$ for $i \neq j$.

Exercise 2.5.1

What is $E[\widehat{\beta}_1]$? Is $\widehat{\beta}_1$ an unbiased estimator of $\beta_1$?

Hint

The least squares estimator for the slope: \[ \widehat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{i=1}^n (x_i - \bar{x})^2} \] Remember that $x_i$ are constants and $E[Y_i] = \beta_0 + \beta_1 x_i$. Apply the linearity of expectation.

Solution

Recall the least squares estimator for the slope:

\[ \widehat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{i=1}^n (x_i - \bar{x})^2} \]

Working directly from here:

\[ \begin{aligned} E[\widehat{\beta}_1] &= E\left[\frac{\sum_{i=1}^n (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{i=1}^n (x_i - \bar{x})^2}\right]\\ &=\frac{\sum_{i=1}^n (x_i - \bar{x})\left(E[Y_i] - E[\bar{Y}]\right)}{\sum_{i=1}^n (x_i - \bar{x})^2} \\ &= \frac{\sum_{i=1}^n (x_i - \bar{x})\left[(\beta_0 + \beta_1 x_i) - (\beta_0 + \beta_1 \bar{x})\right]}{\sum_{i=1}^n (x_i - \bar{x})^2} \\ &= \frac{\beta_1 \sum_{i=1}^n (x_i - \bar{x})^2}{\sum_{i=1}^n (x_i - \bar{x})^2} = \beta_1 \end{aligned} \]

Since $E[\widehat{\beta}_1] = \beta_1$, $\widehat{\beta}_1$ is an unbiased estimator of $\beta_1$. $\blacksquare$

Exercise 2.5.2

Show that you can write $\widehat{\beta}_1 = \sum_{i=1}^n w_i Y_i$, where $w_i = \frac{x_i - \bar{x}}{\sum_{j=1}^n (x_j - \bar{x})^2}$. Moreover, show that $\sum_{i=1}^n w_i = 0$ and $\sum_{i=1}^n w_i x_i = 1$.

Hint

Expand the product in the numerator of $\widehat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{j=1}^n (x_j - \bar{x})^2}$. Compute the value of the sum of deviations around the mean: $\sum_{i=1}^n (x_i - \bar{x})$.

Also, remember that $\sum_{i=1}^n x_i^2 - n\bar{x}^2 = \sum_{i=1}^n (x_i - \bar{x})^2$.

Solution

1. Expressing $\widehat{\beta}_1 = \sum_{i=1}^n w_i Y_i$:

Recall the least squares estimator for $\beta_1$: \[ \begin{align*} \widehat{\beta}_1 &= \frac{\sum_{i=1}^n (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{j=1}^n (x_j - \bar{x})^2}\\ &= \frac{\sum_{i=1}^n\left[ (x_i - \bar{x})Y_i - (x_i - \bar{x})\bar{Y}\right]}{\sum_{j=1}^n (x_j - \bar{x})^2}\\ &= \frac{\sum_{i=1}^n (x_i - \bar{x})Y_i - \bar{Y}\sum_{i=1}^n (x_i - \bar{x})}{\sum_{j=1}^n (x_j - \bar{x})^2}\\ &= \frac{\sum_{i=1}^n (x_i - \bar{x})Y_i}{\sum_{j=1}^n (x_j - \bar{x})^2}\\ &= \sum_{i=1}^n \left( \frac{x_i - \bar{x}}{\sum_{j=1}^n (x_j - \bar{x})^2} \right) Y_i\\ &= \sum_{i=1}^n w_i Y_i \end{align*} \] Therefore, $\widehat{\beta}_1$ can be written as a linear combination of the $Y_i$ with weights $w_i=\frac{x_i - \bar{x}}{\sum_{j=1}^n (x_j - \bar{x})^2}$.

Also, \[ \sum_{i=1}^n w_i = \sum_{i=1}^n \frac{x_i - \bar{x}}{\sum_{j=1}^n (x_j - \bar{x})^2} = \frac{1}{\sum_{j=1}^n (x_j - \bar{x})^2}\sum_{i=1}^n (x_i - \bar{x}) = \frac{1}{\sum_{j=1}^n (x_j - \bar{x})^2} (n\bar{x} - n\bar{x}) = 0 \] and \[ \begin{align*} \sum_{i=1}^n w_i x_i &= \sum_{i=1}^n \frac{(x_i - \bar{x})x_i}{\sum_{j=1}^n (x_j - \bar{x})^2} \\ &= \frac{1}{\sum_{j=1}^n (x_j - \bar{x})^2} \sum_{i=1}^n (x_i - \bar{x})x_i\\ &= \frac{1}{\sum_{j=1}^n (x_j - \bar{x})^2} \sum_{i=1}^n (x_i^2 - \bar{x}x_i)\\ &= \frac{1}{\sum_{j=1}^n (x_j - \bar{x})^2} \left(\sum_{i=1}^n x_i^2 - n\bar{x}^2\right)\\ &= \frac{1}{\sum_{j=1}^n (x_j - \bar{x})^2} \sum_{i=1}^n (x_i - \bar{x})^2\\ &= 1. \end{align*} \]

Exercise 2.5.3

What is the $\text{Var}[\widehat{\beta}_1]$?

Hint

Use the linear combination representation $\widehat{\beta}_1 = \sum_{i=1}^n w_i Y_i$, where $w_i = \frac{x_i - \bar{x}}{\sum_{j=1}^n (x_j - \bar{x})^2}$ established in Exercise 2.5.2. Recall that the observations $Y_1, \dots, Y_n$ are independent with constant variance $\text{Var}(Y_i) = \sigma^2$. For independent random variables, $\text{Var}\left(\sum a_i Y_i\right) = \sum a_i^2 \text{Var}(Y_i)$.

Solution

Using the representation $\widehat{\beta}_1 = \sum_{i=1}^n w_i Y_i$ where $w_i = \frac{x_i - \bar{x}}{\sum_{j=1}^n (x_j - \bar{x})^2}$:

\[ \text{Var}(\widehat{\beta}_1) = \text{Var}\left( \sum_{i=1}^n w_i Y_i \right) \]

Because $Y_1, \dots, Y_n$ are independent observations with constant variance $\text{Var}(Y_i) = \sigma^2$, the variance of the linear combination is:

\[ \begin{aligned} \text{Var}(\widehat{\beta}_1) &= \sum_{i=1}^n w_i^2 \text{Var}(Y_i) \\ &= \sigma^2 \sum_{i=1}^n w_i^2 \end{aligned} \]

Now compute the sum of the squared weights $\sum_{i=1}^n w_i^2$:

\[ \sum_{i=1}^n w_i^2 = \sum_{i=1}^n \left( \frac{x_i - \bar{x}}{\sum_{j=1}^n (x_j - \bar{x})^2} \right)^2 = \frac{\sum_{i=1}^n (x_i - \bar{x})^2}{\left(\sum_{j=1}^n (x_j - \bar{x})^2\right)^2} = \frac{1}{\sum_{i=1}^n (x_i - \bar{x})^2} \]

Therefore:

\[ \text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \quad \blacksquare \]

Exercise 2.5.4

What would be the sampling distribution of $\widehat{\beta}_1$ if the errors $\epsilon_i$ are normally distributed? Justify your answer.

Hint

Recall from Exercise 2.5.2 that $\widehat{\beta}_1$ is a linear combination of $Y_1, \dots, Y_n$. What is the distribution of each $Y_i$? What is the distribution of a linear combination of independent normal random variables?

Solution

If the errors are independent and identically distributed as $\epsilon_i \sim N(0, \sigma^2)$, then each response variable:

\[ Y_i = \beta_0 + \beta_1 x_i + \epsilon_i \]

is a linear transformation of a normal random variable, which means:

\[ Y_i \overset{\text{ind}}{\sim} N(\beta_0 + \beta_1 x_i, \; \sigma^2) \]

From Exercise 2.5.2, $\widehat{\beta}_1$ is written as a linear combination of $Y_1, \dots, Y_n$:

\[ \widehat{\beta}_1 = \sum_{i=1}^n w_i Y_i, \quad \text{where } w_i = \frac{x_i - \bar{x}}{\sum_{j=1}^n (x_j - \bar{x})^2} \text{ are fixed constants.} \]

A fundamental property of the normal distribution is that any linear combination of independent normal random variables is itself normally distributed. Thus, $\widehat{\beta}_1$ follows a normal distribution.

Using the expectation $E[\widehat{\beta}_1] = \beta_1$ from Exercise 2.5.1 and the variance $\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$ from Exercise 2.5.3:

\[ \widehat{\beta}_1 \sim N\left(\beta_1, \; \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\right) \quad \blacksquare \]

Exercise 2.5.5

What is $E[\widehat{\beta}_0]$? Is $\widehat{\beta}_0$ an unbiased estimator of $\beta_0$?

Hint

Start with $\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x}$. Apply the linearity of expectation and use the result from Exercise 2.5.1.

Solution

The least squares estimator for the intercept is:

\[ \widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x} \]

Taking the expectation of both sides:

\[ E[\widehat{\beta}_0] = E[\bar{Y} - \widehat{\beta}_1 \bar{x}] = E[\bar{Y}] - \bar{x} E[\widehat{\beta}_1] \]

First, find $E[\bar{Y}]$:

\[ E[\bar{Y}] = E\left[ \frac{1}{n} \sum_{i=1}^n Y_i \right] = \frac{1}{n} \sum_{i=1}^n E[Y_i] = \frac{1}{n} \sum_{i=1}^n (\beta_0 + \beta_1 x_i) = \beta_0 + \beta_1 \bar{x} \]

From Exercise 2.5.1, we have $E[\widehat{\beta}_1] = \beta_1$. Substituting these expressions:

\[ \begin{aligned} E[\widehat{\beta}_0] &= (\beta_0 + \beta_1 \bar{x}) - \bar{x} \beta_1 \\ &= \beta_0 \end{aligned} \]

Since $E[\widehat{\beta}_0] = \beta_0$, $\widehat{\beta}_0$ is an unbiased estimator of $\beta_0$. $\blacksquare$

Exercise 2.5.6

Calculate $\text{Cov}(\bar{Y}, \widehat{\beta}_1)$.

Hint

Write $\bar{Y}$ and $\widehat{\beta}_1$ as linear combinations of $Y_1, \dots, Y_n$. Use the bilinearity of covariance and the fact that $Y_1, \dots, Y_n$ are independent observations.

Solution

Write $\bar{Y} = \frac{1}{n} \sum_{i=1}^n Y_i$ and $\widehat{\beta}_1 = \sum_{j=1}^n w_j Y_j$, where $w_j = \frac{x_j - \bar{x}}{\sum_{k=1}^n (x_k - \bar{x})^2}$.

Using the bilinearity of covariance:

\[ \begin{aligned} \text{Cov}(\bar{Y}, \widehat{\beta}_1) &= \text{Cov}\left( \frac{1}{n}\sum_{i=1}^n Y_i, \; \sum_{j=1}^n w_j Y_j \right) \\ &= \frac{1}{n} \sum_{i=1}^n \sum_{j=1}^n w_j \text{Cov}(Y_i, Y_j) \end{aligned} \]

Since $Y_1, \dots, Y_n$ are independent, $\text{Cov}(Y_i, Y_j) = 0$ whenever $i \neq j$, and $\text{Cov}(Y_i, Y_i) = \text{Var}(Y_i) = \sigma^2$. The double sum collapses to the diagonal terms where $i = j$:

\[ \begin{aligned} \text{Cov}(\bar{Y}, \widehat{\beta}_1) &= \frac{1}{n} \sum_{i=1}^n w_i \sigma^2 \\ &= \frac{\sigma^2}{n} \sum_{i=1}^n w_i \\ &= \frac{\sigma^2}{n} \frac{\sum_{i=1}^n (x_i - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \end{aligned} \]

Because $\sum_{i=1}^n (x_i - \bar{x}) = 0$:

\[ \text{Cov}(\bar{Y}, \widehat{\beta}_1) = \frac{\sigma^2}{n} \cdot 0 = 0 \quad \blacksquare \]

Remark: Under normal errors, zero covariance implies that the sample mean $\bar{Y}$ and the estimated slope $\widehat{\beta}_1$ are statistically independent.

Exercise 2.5.7

What is the $\text{Var}[\widehat{\beta}_0]$?

Hint

Start with $\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x}$ and apply the variance of a linear combination of two random variables. Use the results from Exercises 2.5.3 and 2.5.6.

Solution

Starting from the relation $\widehat{\beta}_0 = \bar{Y} - \bar{x}\widehat{\beta}_1$:

\[ \begin{aligned} \text{Var}(\widehat{\beta}_0) &= \text{Var}(\bar{Y} - \bar{x}\widehat{\beta}_1) \\ &= \text{Var}(\bar{Y}) + \bar{x}^2 \text{Var}(\widehat{\beta}_1) - 2\bar{x}\text{Cov}(\bar{Y}, \widehat{\beta}_1) \end{aligned} \]

Evaluating each component:

$\displaystyle \text{Var}(\bar{Y}) = \text{Var}\left(\frac{1}{n}\sum_{i=1}^n Y_i\right) = \frac{1}{n^2}\sum_{i=1}^n \text{Var}(Y_i) = \frac{n\sigma^2}{n^2} = \frac{\sigma^2}{n}$
From Exercise 2.5.3: $\displaystyle \text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$
From Exercise 2.5.6: $\displaystyle \text{Cov}(\bar{Y}, \widehat{\beta}_1) = 0$

Substituting these three results back into the equation:

\[ \begin{aligned} \text{Var}(\widehat{\beta}_0) &= \frac{\sigma^2}{n} + \bar{x}^2 \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} - 2\bar{x}(0) \\ &= \sigma^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right) \quad \blacksquare \end{aligned} \]

Exercise 2.5.8

What would be the sampling distribution of $\widehat{\beta}_0$ if the errors $\epsilon_i$ are normally distributed? Justify your answer.

Hint

Notice that $\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x}$ can also be written as a linear combination of $Y_1, \dots, Y_n$. What is the distribution of a linear combination of independent normal random variables? Use the mean and variance obtained in Exercises 2.5.5 and 2.5.7.

Solution

We can express $\widehat{\beta}_0$ directly in terms of $Y_1, \dots, Y_n$:

\[ \widehat{\beta}_0 = \bar{Y} - \bar{x}\widehat{\beta}_1 = \frac{1}{n}\sum_{i=1}^n Y_i - \bar{x}\sum_{i=1}^n w_i Y_i = \sum_{i=1}^n \left( \frac{1}{n} - \bar{x} w_i \right) Y_i = \sum_{i=1}^n c_i Y_i \]

where $c_i = \frac{1}{n} - \frac{\bar{x}(x_i - \bar{x})}{\sum_{j=1}^n (x_j - \bar{x})^2}$ are fixed, non-random constants.

Because each observation $Y_i \overset{\text{ind}}{\sim} N(\beta_0 + \beta_1 x_i, \sigma^2)$ is normally distributed, any linear combination $\sum_{i=1}^n c_i Y_i$ is also normally distributed. Hence, $\widehat{\beta}_0$ follows a normal distribution.

Using the expectation $E[\widehat{\beta}_0] = \beta_0$ from Exercise 2.5.5 and the variance $\text{Var}(\widehat{\beta}_0) = \sigma^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)$ from Exercise 2.5.7:

\[ \widehat{\beta}_0 \sim N\left( \beta_0, \; \sigma^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right) \right) \quad \blacksquare \]

Exercise 2.5.9

Show that $\text{Cov}(e_i, \widehat{\beta}_1) = 0$. Use this result to justify that $\text{RSS}$ and $\widehat{\beta}_1$ are independent.

Hint

Express the residual $e_i = Y_i - \widehat{Y}_i$ in terms of $(Y_i - \bar{Y})$ and $(x_i - \bar{x})\widehat{\beta}_1$. Then evaluate the covariance with $\widehat{\beta}_1$ using results from previous exercises.

For independence, recall what zero covariance implies for normally distributed random variables, and note that $\text{RSS} = \sum_{i=1}^n e_i^2$ is a function of the residuals $e_1, \dots, e_n$.

Solution

The fitted value for observation $i$ is $\widehat{Y}_i = \widehat{\beta}_0 + \widehat{\beta}_1 x_i$. Since $\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x}$, we have:

\[ \widehat{Y}_i = \bar{Y} + \widehat{\beta}_1(x_i - \bar{x}) \]

The $i$-th residual $e_i$ is therefore:

\[ e_i = Y_i - \widehat{Y}_i = (Y_i - \bar{Y}) - (x_i - \bar{x})\widehat{\beta}_1 \]

Now compute the covariance between $e_i$ and $\widehat{\beta}_1$:

\[ \begin{aligned} \text{Cov}(e_i, \widehat{\beta}_1) &= \text{Cov}\Big( (Y_i - \bar{Y}) - (x_i - \bar{x})\widehat{\beta}_1, \; \widehat{\beta}_1 \Big) \\ &= \text{Cov}(Y_i, \widehat{\beta}_1) - \text{Cov}(\bar{Y}, \widehat{\beta}_1) - (x_i - \bar{x})\text{Var}(\widehat{\beta}_1) \end{aligned} \]

We evaluate each term:

Writing $\widehat{\beta}_1 = \sum_{j=1}^n w_j Y_j$ where $w_j = \frac{x_j - \bar{x}}{\sum_{k=1}^n (x_k - \bar{x})^2}$ and using $\text{Cov}(Y_i, Y_j) = 0$ for $i \neq j$: \[ \text{Cov}(Y_i, \widehat{\beta}_1) = \sum_{j=1}^n w_j \text{Cov}(Y_i, Y_j) = w_i \text{Var}(Y_i) = \frac{x_i - \bar{x}}{\sum_{k=1}^n (x_k - \bar{x})^2}\sigma^2 \]
From Exercise 2.5.6, $\text{Cov}(\bar{Y}, \widehat{\beta}_1) = 0$.
From Exercise 2.5.3, $\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{k=1}^n (x_k - \bar{x})^2}$, so: \[ (x_i - \bar{x})\text{Var}(\widehat{\beta}_1) = \frac{x_i - \bar{x}}{\sum_{k=1}^n (x_k - \bar{x})^2}\sigma^2 \]

Substituting these back:

\[ \text{Cov}(e_i, \widehat{\beta}_1) = \frac{x_i - \bar{x}}{\sum_{k=1}^n (x_k - \bar{x})^2}\sigma^2 - 0 - \frac{x_i - \bar{x}}{\sum_{k=1}^n (x_k - \bar{x})^2}\sigma^2 = 0 \quad \blacksquare \]

Independence of $\text{RSS}$ and $\widehat{\beta}_1$:

Because $Y_1, \dots, Y_n$ are normally distributed, linear combinations of them are also normally distributed. For normally distributed variables, zero covariance implies statistical independence.

Since $\text{Cov}(e_i, \widehat{\beta}_1) = 0$ for all $i = 1, \dots, n$, each residual $e_i$ is independent of $\widehat{\beta}_1$. Because $\text{RSS} = \sum_{i=1}^n e_i^2$ is a function of these residuals $e_1, \dots, e_n$, it follows that $\text{RSS}$ and $\widehat{\beta}_1$ are independent. $\blacksquare$

Exercise 2.5.10

Show that $\text{Cov}(e_i, \widehat{\beta}_0) = 0$. Use this result to justify that $\text{RSS}$ and $\widehat{\beta}_0$ are independent.

Hint

Substitute $\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x}$ into $\text{Cov}(e_i, \widehat{\beta}_0)$ and apply linearity of covariance. You can reuse the result from Exercise 2.5.9.

For independence, follow the same reasoning as in Exercise 2.5.9.

Solution

Express $\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x}$. Using linearity of covariance:

\[ \text{Cov}(e_i, \widehat{\beta}_0) = \text{Cov}(e_i, \bar{Y} - \bar{x}\widehat{\beta}_1) = \text{Cov}(e_i, \bar{Y}) - \bar{x}\text{Cov}(e_i, \widehat{\beta}_1) \]

From Exercise 2.5.9, $\text{Cov}(e_i, \widehat{\beta}_1) = 0$. Now consider $\text{Cov}(e_i, \bar{Y})$. Substituting $e_i = (Y_i - \bar{Y}) - (x_i - \bar{x})\widehat{\beta}_1$:

\[ \text{Cov}(e_i, \bar{Y}) = \text{Cov}(Y_i - \bar{Y}, \bar{Y}) - (x_i - \bar{x})\text{Cov}(\widehat{\beta}_1, \bar{Y}) \]

From Exercise 2.5.6, $\text{Cov}(\widehat{\beta}_1, \bar{Y}) = 0$. For the first term:

\[ \begin{aligned} \text{Cov}(Y_i - \bar{Y}, \bar{Y}) &= \text{Cov}(Y_i, \bar{Y}) - \text{Var}(\bar{Y}) \\ &= \text{Cov}\left(Y_i, \frac{1}{n}\sum_{j=1}^n Y_j\right) - \frac{\sigma^2}{n} \\ &= \frac{1}{n}\text{Var}(Y_i) - \frac{\sigma^2}{n} \\ &= \frac{1}{n}\sigma^2 - \frac{\sigma^2}{n} = 0 \end{aligned} \]

Therefore:

\[ \text{Cov}(e_i, \bar{Y}) = 0 - (x_i - \bar{x})(0) = 0 \]

Putting this together:

\[ \text{Cov}(e_i, \widehat{\beta}_0) = 0 - \bar{x}(0) = 0 \quad \blacksquare \]

Independence of $\text{RSS}$ and $\widehat{\beta}_0$:

Because $Y_1, \dots, Y_n$ are normally distributed, zero covariance between linear combinations of them implies statistical independence.

Since $\text{Cov}(e_i, \widehat{\beta}_0) = 0$ for all $i = 1, \dots, n$, each residual $e_i$ is independent of $\widehat{\beta}_0$. Because $\text{RSS} = \sum_{i=1}^n e_i^2$ is a function of these residuals $e_1, \dots, e_n$, it follows that $\text{RSS}$ and $\widehat{\beta}_0$ are independent. $\blacksquare$

Exercise 2.5.11

Show that $\frac{\widehat{\beta}_1-\beta_1}{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_1)}}$ follows a $t$-distribution with $n-2$ degrees of freedom. You can use the fact that $\text{RSS}/\sigma^2 \sim \chi^2_{n-2}$ and that $\widehat{\beta}_1$ is independent of $\text{RSS}$ as shown in Exercise 2.5.9.

Hint

Recall the definition of a Student’s $t$-distribution: \[ T = \frac{Z}{\sqrt{U / d}} \sim t_d \] where $Z \sim N(0, 1)$ and $U \sim \chi^2_d$ are independent.

Identify $Z$ by standardizing $\widehat{\beta}_1$, let $U = \text{RSS}/\sigma^2$, and verify that the ratio simplifies algebraically to the desired expression.

Solution

Definition of Student’s $t$-distribution: A random variable $T$ follows a Student’s $t$-distribution with $d$ degrees of freedom, written $T \sim t_d$, if it can be represented as:

\[ T = \frac{Z}{\sqrt{U / d}} \]

where:

$Z \sim N(0, 1)$ is standard normal,
$U \sim \chi^2_d$ is chi-square distributed with $d$ degrees of freedom,
$Z$ and $U$ are statistically independent.

We verify each condition:

1. The numerator ($Z$): From Exercise 2.5.4, $\widehat{\beta}_1 \sim N\left(\beta_1, \text{Var}(\widehat{\beta}_1)\right)$, where $\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$. Standardizing gives:

\[ Z = \frac{\widehat{\beta}_1 - \beta_1}{\sqrt{\text{Var}(\widehat{\beta}_1)}} = \frac{\widehat{\beta}_1 - \beta_1}{\sqrt{\frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}}} \sim N(0, 1) \]

2. The denominator ($U$): We are given that:

\[ U = \frac{\text{RSS}}{\sigma^2} \sim \chi^2_{n-2} \]

so the degrees of freedom parameter is $d = n - 2$.

3. Independence of $Z$ and $U$: From Exercise 2.5.9, $\widehat{\beta}_1$ is independent of $\text{RSS}$. Since $Z$ is a function only of $\widehat{\beta}_1$, and $U$ is a function only of $\text{RSS}$, $Z$ and $U$ are independent.

4. Algebraic simplification: Construct the $t$-ratio:

\[ T = \frac{Z}{\sqrt{U / (n-2)}} = \frac{\frac{\widehat{\beta}_1 - \beta_1}{\sqrt{\text{Var}(\widehat{\beta}_1)}}}{\sqrt{\frac{\text{RSS}/\sigma^2}{n - 2}}} \]

Recall that the unbiased estimator of the error variance $\sigma^2$ is:

\[ \hat{\sigma}^2 = s^2 = \frac{\text{RSS}}{n - 2} \]

Substitute $\hat{\sigma}^2$ into the denominator:

\[ \frac{\text{RSS} / \sigma^2}{n - 2} = \frac{\hat{\sigma}^2}{\sigma^2} \]

Notice that the ratio of the estimated variance to the true variance of $\widehat{\beta}_1$ is:

\[ \frac{\widehat{\text{Var}}(\widehat{\beta}_1)}{\text{Var}(\widehat{\beta}_1)} = \frac{\frac{\hat{\sigma}^2}{\sum_{i=1}^n (x_i - \bar{x})^2}}{\frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}} = \frac{\hat{\sigma}^2}{\sigma^2} \]

Therefore:

\[ \sqrt{\frac{U}{n - 2}} = \sqrt{\frac{\widehat{\text{Var}}(\widehat{\beta}_1)}{\text{Var}(\widehat{\beta}_1)}} = \frac{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_1)}}{\sqrt{\text{Var}(\widehat{\beta}_1)}} \]

Substituting this into the ratio $T$:

\[ T = \frac{\frac{\widehat{\beta}_1 - \beta_1}{\sqrt{\text{Var}(\widehat{\beta}_1)}}}{\frac{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_1)}}{\sqrt{\text{Var}(\widehat{\beta}_1)}}} = \frac{\widehat{\beta}_1 - \beta_1}{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_1)}} \]

Since all conditions of the definition are satisfied:

\[ \frac{\widehat{\beta}_1 - \beta_1}{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_1)}} = \frac{\widehat{\beta}_1 - \beta_1}{\sqrt{\frac{\hat{\sigma}^2}{\sum_{i=1}^n (x_i - \bar{x})^2}}} \sim t_{n-2} \quad \blacksquare \]

Exercise 2.5.12

Show that $\frac{\widehat{\beta}_0-\beta_0}{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_0)}}$ follows a $t$-distribution with $n-2$ degrees of freedom. You can use the fact that $\text{RSS}/\sigma^2 \sim \chi^2_{n-2}$ and that $\widehat{\beta}_0$ is independent of $\text{RSS}$ as shown in Exercise 2.5.10.

Hint

Recall the definition of a Student’s $t$-distribution: \[ T = \frac{Z}{\sqrt{U / d}} \sim t_d \] where $Z \sim N(0, 1)$ and $U \sim \chi^2_d$ are independent.

Identify $Z$ by standardizing $\widehat{\beta}_0$ using its sampling distribution from Exercise 2.5.8, let $U = \text{RSS}/\sigma^2$, and verify that the ratio simplifies algebraically to the desired expression.

Solution

Definition of Student’s $t$-distribution: A random variable $T$ follows a Student’s $t$-distribution with $d$ degrees of freedom, written $T \sim t_d$, if it can be represented as:

\[ T = \frac{Z}{\sqrt{U / d}} \]

where:

$Z \sim N(0, 1)$ is standard normal,
$U \sim \chi^2_d$ is chi-square distributed with $d$ degrees of freedom,
$Z$ and $U$ are statistically independent.

We verify each condition:

1. The numerator ($Z$): From Exercise 2.5.8, $\widehat{\beta}_0 \sim N\left(\beta_0, \text{Var}(\widehat{\beta}_0)\right)$, where $\text{Var}(\widehat{\beta}_0) = \sigma^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)$. Standardizing gives:

\[ Z = \frac{\widehat{\beta}_0 - \beta_0}{\sqrt{\text{Var}(\widehat{\beta}_0)}} = \frac{\widehat{\beta}_0 - \beta_0}{\sqrt{\sigma^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)}} \sim N(0, 1) \]

2. The denominator ($U$): We are given that:

\[ U = \frac{\text{RSS}}{\sigma^2} \sim \chi^2_{n-2} \]

so the degrees of freedom parameter is $d = n - 2$.

3. Independence of $Z$ and $U$: From Exercise 2.5.10, $\widehat{\beta}_0$ is independent of $\text{RSS}$. Since $Z$ is a function only of $\widehat{\beta}_0$, and $U$ is a function only of $\text{RSS}$, $Z$ and $U$ are independent.

4. Algebraic simplification: Construct the $t$-ratio:

\[ T = \frac{Z}{\sqrt{U / (n-2)}} = \frac{\frac{\widehat{\beta}_0 - \beta_0}{\sqrt{\text{Var}(\widehat{\beta}_0)}}}{\sqrt{\frac{\text{RSS}/\sigma^2}{n - 2}}} \]

Recall that the unbiased estimator of the error variance $\sigma^2$ is:

\[ \hat{\sigma}^2 = s^2 = \frac{\text{RSS}}{n - 2} \]

Substitute $\hat{\sigma}^2$ into the denominator:

\[ \frac{\text{RSS} / \sigma^2}{n - 2} = \frac{\hat{\sigma}^2}{\sigma^2} \]

Notice that the ratio of the estimated variance to the true variance of $\widehat{\beta}_0$ is:

\[ \frac{\widehat{\text{Var}}(\widehat{\beta}_0)}{\text{Var}(\widehat{\beta}_0)} = \frac{\hat{\sigma}^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)}{\sigma^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)} = \frac{\hat{\sigma}^2}{\sigma^2} \]

Therefore:

\[ \sqrt{\frac{U}{n - 2}} = \sqrt{\frac{\widehat{\text{Var}}(\widehat{\beta}_0)}{\text{Var}(\widehat{\beta}_0)}} = \frac{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_0)}}{\sqrt{\text{Var}(\widehat{\beta}_0)}} \]

Substituting this into the ratio $T$:

\[ T = \frac{\frac{\widehat{\beta}_0 - \beta_0}{\sqrt{\text{Var}(\widehat{\beta}_0)}}}{\frac{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_0)}}{\sqrt{\text{Var}(\widehat{\beta}_0)}}} = \frac{\widehat{\beta}_0 - \beta_0}{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_0)}} \]

Since all conditions of the definition are satisfied:

\[ \frac{\widehat{\beta}_0 - \beta_0}{\sqrt{\widehat{\text{Var}}(\widehat{\beta}_0)}} = \frac{\widehat{\beta}_0 - \beta_0}{\sqrt{\hat{\sigma}^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right)}} \sim t_{n-2} \quad \blacksquare \]

Exercise 2.5.13

Compute $\text{Cov}(\widehat{\beta}_0, \widehat{\beta}_1)$.

Hint

Express $\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x}$ and use the linearity of covariance: \[ \text{Cov}(\bar{Y} - \bar{x}\widehat{\beta}_1, \; \widehat{\beta}_1) \] Apply the results from Exercise 2.5.6 ($\text{Cov}(\bar{Y}, \widehat{\beta}_1) = 0$) and Exercise 2.5.3 ($\text{Var}(\widehat{\beta}_1)$).

Solution

Express $\widehat{\beta}_0$ in terms of $\bar{Y}$ and $\widehat{\beta}_1$:

\[ \widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x} \]

Using the linearity of covariance:

\[ \begin{aligned} \text{Cov}(\widehat{\beta}_0, \widehat{\beta}_1) &= \text{Cov}(\bar{Y} - \bar{x}\widehat{\beta}_1, \; \widehat{\beta}_1) \\ &= \text{Cov}(\bar{Y}, \widehat{\beta}_1) - \bar{x}\text{Var}(\widehat{\beta}_1) \end{aligned} \]

From Exercise 2.5.6, $\text{Cov}(\bar{Y}, \widehat{\beta}_1) = 0$. From Exercise 2.5.3, $\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$.

Substituting these results:

\[ \text{Cov}(\widehat{\beta}_0, \widehat{\beta}_1) = 0 - \bar{x}\left(\frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\right) = -\frac{\bar{x}\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \quad \blacksquare \]

Notice that if the predictor values are centered so that $\bar{x} = 0$, then $\text{Cov}(\widehat{\beta}_0, \widehat{\beta}_1) = 0$. When $\bar{x} > 0$, the covariance is negative, meaning that an overestimate of the slope $\widehat{\beta}_1$ is associated with an underestimate of the intercept $\widehat{\beta}_0$.

Exercise 2.5.14

In Exercise 2.5.11, you showed that \[ T = \frac{\widehat{\beta}_1-\beta_1}{\widehat{\text{SE}}(\widehat{\beta}_1)} \sim t_{n-2}, \qquad \text{where } \widehat{\text{SE}}(\widehat{\beta}_1) = \frac{\widehat{\sigma}}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2}}. \]

Let $t^*_{n-2,1-\alpha/2}$ be the $1-\alpha/2$ quantile of the $t_{n-2}$ distribution.

(a) Starting from $\mathbb{P}\left(-t^*_{n-2,1-\alpha/2} \leq T \leq t^*_{n-2,1-\alpha/2}\right) = 1-\alpha$, derive the $(1-\alpha)100\%$ confidence interval for $\beta_1$.

Hint (a)

Multiply all sides of the inequalities by $\widehat{\text{SE}}(\widehat{\beta}_1) > 0$, then isolate $\beta_1$ in the middle. Be careful with the direction of the inequalities when you multiply by $-1$.

Solution (a)

To simplify the notation, write $t^* = t^*_{n-2,1-\alpha/2}$ and $\widehat{\text{SE}} = \widehat{\text{SE}}(\widehat{\beta}_1)$. Then,

\[ \begin{aligned} 1-\alpha &= \mathbb{P}\left(-t^* \leq \frac{\widehat{\beta}_1-\beta_1}{\widehat{\text{SE}}} \leq t^*\right) \\ &= \mathbb{P}\left(-t^*\,\widehat{\text{SE}} \leq \widehat{\beta}_1-\beta_1 \leq t^*\,\widehat{\text{SE}}\right) \\ &= \mathbb{P}\left(-\widehat{\beta}_1 - t^*\,\widehat{\text{SE}} \leq -\beta_1 \leq -\widehat{\beta}_1 + t^*\,\widehat{\text{SE}}\right) \\ &= \mathbb{P}\left(\widehat{\beta}_1 - t^*\,\widehat{\text{SE}} \leq \beta_1 \leq \widehat{\beta}_1 + t^*\,\widehat{\text{SE}}\right) \end{aligned} \]

where in the last step we multiplied all sides by $-1$, which flips the inequalities. Therefore, the $(1-\alpha)100\%$ confidence interval for $\beta_1$ is

\[ CI(\beta_1, 1-\alpha) = \widehat{\beta}_1 \pm t^*_{n-2,1-\alpha/2}\,\frac{\widehat{\sigma}}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2}} \quad \blacksquare \]

(b) In the probability statement you obtained, which quantities are random and which are fixed? Use your answer to explain why, after we compute the interval with our data, we cannot say that “$\beta_1$ is in the interval with probability $1-\alpha$”.

Hint (b)

Remember that $\beta_1$ is a parameter, while $\widehat{\beta}_1$ and $\widehat{\sigma}$ are statistics.

Solution (b)

In the last probability statement, $\beta_1$ is a fixed (unknown) constant. The random quantities are the endpoints $\widehat{\beta}_1 \pm t^*\,\widehat{\text{SE}}$, since both $\widehat{\beta}_1$ and $\widehat{\sigma}$ depend on the random sample. So the statement is about the procedure: before we collect the data, the random interval will capture $\beta_1$ with probability $1-\alpha$.

Once we plug in the observed data, the endpoints become numbers, say $[47.1, 53.2]$. Now nothing is random: $\beta_1$ is either in $[47.1, 53.2]$ or it is not. This is why we say we are $(1-\alpha)100\%$ confident: approximately $(1-\alpha)100\%$ of the intervals computed from repeated samples (at the same $x$ values) would contain $\beta_1$.

Exercise 2.5.15

Using the result of Exercise 2.5.12, derive the $(1-\alpha)100\%$ confidence interval for $\beta_0$.

Hint

The algebra is the same as in Exercise 2.5.14. Only the standard error changes.

Solution

From Exercise 2.5.12,

\[ T = \frac{\widehat{\beta}_0-\beta_0}{\widehat{\text{SE}}(\widehat{\beta}_0)} \sim t_{n-2}, \qquad \text{where } \widehat{\text{SE}}(\widehat{\beta}_0) = \widehat{\sigma}\sqrt{\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2}}. \]

Write $t^* = t^*_{n-2,1-\alpha/2}$. Following the same steps as in Exercise 2.5.14:

\[ \begin{aligned} 1-\alpha &= \mathbb{P}\left(-t^* \leq \frac{\widehat{\beta}_0-\beta_0}{\widehat{\text{SE}}(\widehat{\beta}_0)} \leq t^*\right) \\ &= \mathbb{P}\left(-t^*\,\widehat{\text{SE}}(\widehat{\beta}_0) \leq \widehat{\beta}_0-\beta_0 \leq t^*\,\widehat{\text{SE}}(\widehat{\beta}_0)\right) \\ &= \mathbb{P}\left(\widehat{\beta}_0 - t^*\,\widehat{\text{SE}}(\widehat{\beta}_0) \leq \beta_0 \leq \widehat{\beta}_0 + t^*\,\widehat{\text{SE}}(\widehat{\beta}_0)\right) \end{aligned} \]

Therefore,

\[ CI(\beta_0, 1-\alpha) = \widehat{\beta}_0 \pm t^*_{n-2,1-\alpha/2}\,\widehat{\sigma}\sqrt{\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2}} \quad \blacksquare \]

Exercise 2.5.16

Consider the hypotheses

\[ H_0: \beta_1 = \beta_{1,0} \quad \text{vs} \quad H_A: \beta_1 \neq \beta_{1,0}, \]

where $\beta_{1,0}$ is a fixed number (e.g., $\beta_{1,0} = 0$). Define the test statistic $T_0 = \dfrac{\widehat{\beta}_1-\beta_{1,0}}{\widehat{\text{SE}}(\widehat{\beta}_1)}$.

mcqFeedback = (correct, text) => html`<div style="padding: 0.6em 1em; margin-top: 0.4em; margin-left: 2em; border-left: 5px solid ${correct ? "#2e7d32" : "#c62828"}; background: ${correct ? "#e8f5e9" : "#ffebee"}; color: #222;"><strong>${correct ? "Correct!" : "Not quite."}</strong> ${text}</div>`

(a) What is the distribution of $T_0$?

viewof ex616a = {
  const input = Inputs.radio(
  new Map([
    ["N(0, 1)", "A"],
    ["t with n−2 degrees of freedom", "B"],
    ["t with n−1 degrees of freedom", "C"],
    ["Not possible to know", "D"]
  ]),
  {value: null}
  );
  input.style.marginLeft = "2em";
  return input;
}

{
  const fb = {
    A: [false, "Two problems. First, we use the estimated standard error (σ is replaced by its estimate), which gives heavier tails than a normal. Second, the numerator is centered at β₁ − β₁,₀, which is 0 only if H₀ is true, and we have not said that it is."],
    B: [false, "The t with n−2 degrees of freedom is the right family only when β₁ = β₁,₀ (i.e., when H₀ is true). Here we are not told that H₀ holds. If H₀ is false, the numerator is shifted away from 0 and T₀ is no longer a t distribution."],
    C: [false, "Two problems.  The numerator is centered at β₁ − β₁,₀, which is 0 only if H₀ is true. But even if that was given (it wasn't), the degrees of freedom would be n−2, not n−1 (we estimate two parameters, β₀ and β₁)."],
    D: [true, "The distribution of T₀ depends on whether β₁ = β₁,₀. If H₀ is true, T₀ ~ t(n−2); if not, T₀ is shifted away from 0. Since we don't know which is the case, we can't say what the distribution is."]
  };
  return ex616a ? mcqFeedback(...fb[ex616a]) : html``;
}

(b) What is the distribution of $T_0$ when $H_0$ is true?

viewof ex616b = {
  const input = Inputs.radio(
  new Map([
    ["N(0, 1)", "A"],
    ["t with n−2 degrees of freedom", "B"],
    ["t with n−1 degrees of freedom", "C"],
    ["Not possible to know", "D"]
  ]),
  {value: null}
  );
  input.style.marginLeft = "2em";
  return input;
}

{
  const fb = {
    A: [false, "A normal would be right if σ were known, so that the standard error was exact. But we plug in the estimate σ̂, which adds extra variability and makes the tails heavier: the result is a t distribution."],
    B: [true, "When H₀ is true, β₁ = β₁,₀ and T₀ is exactly the standardized slope from Exercise 2.5.11, so T₀ ~ t(n−2). We lose 2 degrees of freedom because we estimate two parameters, β₀ and β₁."],
    C: [false, "Right family, wrong degrees of freedom. n−1 would be correct if we only estimated one parameter (e.g., a single mean). Here we estimate both β₀ and β₁, so the degrees of freedom are n−2."],
    D: [false, "We can know it. Once H₀ is true, β₁ = β₁,₀ and T₀ is the same quantity as in Exercise 2.5.11, whose distribution we already derived."]
  };
  return ex616b ? mcqFeedback(...fb[ex616b]) : html``;
}

(c) Consider the rule “reject $H_0$ if $|T_0| > t^*_{n-2,1-\alpha/2}$”. Show that the probability of rejecting $H_0$ when $H_0$ is true (Type I error) is exactly $\alpha$.

Hint

Use the symmetry of the $t$ distribution: $\mathbb{P}(T < -t^*) = \mathbb{P}(T > t^*) = \alpha/2$.

Solution

Write $t^* = t^*_{n-2,1-\alpha/2}$. Under $H_0$, $T_0 \sim t_{n-2}$, so

\[ \begin{aligned} \mathbb{P}(\text{reject } H_0 \mid H_0 \text{ true}) &= \mathbb{P}(|T_0| > t^*) \\ &= \mathbb{P}(T_0 < -t^*) + \mathbb{P}(T_0 > t^*) \\ &= \frac{\alpha}{2} + \frac{\alpha}{2} = \alpha \quad \blacksquare \end{aligned} \]

Exercise 2.5.17

Show that the two-sided test of Exercise 2.5.16 at significance level $\alpha$ rejects $H_0: \beta_1 = \beta_{1,0}$ if and only if $\beta_{1,0}$ is not in the $(1-\alpha)100\%$ confidence interval for $\beta_1$ from Exercise 2.5.14.

Hint

Start from the event “do not reject $H_0$”, i.e., $|T_0| \leq t^*_{n-2,1-\alpha/2}$, and rewrite it as a statement about $\beta_{1,0}$.

Solution

Write $t^* = t^*_{n-2,1-\alpha/2}$ and $\widehat{\text{SE}} = \widehat{\text{SE}}(\widehat{\beta}_1)$. We do not reject $H_0$ if and only if

\[ \begin{aligned} \left|\frac{\widehat{\beta}_1 - \beta_{1,0}}{\widehat{\text{SE}}}\right| \leq t^* &\iff -t^*\,\widehat{\text{SE}} \leq \widehat{\beta}_1 - \beta_{1,0} \leq t^*\,\widehat{\text{SE}} \\ &\iff \widehat{\beta}_1 - t^*\,\widehat{\text{SE}} \leq \beta_{1,0} \leq \widehat{\beta}_1 + t^*\,\widehat{\text{SE}} \\ &\iff \beta_{1,0} \in CI(\beta_1, 1-\alpha). \end{aligned} \]

Negating both sides, we reject $H_0$ if and only if $\beta_{1,0} \notin CI(\beta_1, 1-\alpha)$. $\blacksquare$

In other words, the $(1-\alpha)100\%$ confidence interval is the set of all values $\beta_{1,0}$ that would not be rejected by a two-sided test at level $\alpha$. This is why we can read a confidence interval as a range of plausible values for $\beta_1$. Note that this correspondence requires a two-sided test and matching levels ($\alpha$ for the test, $1-\alpha$ for the interval).

Exercise 2.5.18

In Module 04, we fitted the model body_mass_g ~ flipper_length_mm to the penguins data ($n = 333$) and obtained the 95% confidence interval $(47.12, 53.18)$ g/mm for $\beta_1$.

Without computing anything else, answer the following questions. Justify your answers.

(a) At $\alpha = 0.05$, would you reject $H_0: \beta_1 = 50$ in favour of $H_A: \beta_1 \neq 50$?

Hint (a)

Take a look at the result of Exercise 2.5.17.

Solution (a)

$50 \in (47.12, 53.18)$, so we do not reject $H_0$. The p-value is larger than $0.05$.

(b) At $\alpha = 0.05$, would you reject $H_0: \beta_1 = 45$ in favour of $H_A: \beta_1 \neq 45$?

Hint (b)

Same idea as in (a): use the result of Exercise 2.5.17.

Solution (b)

$45 \notin (47.12, 53.18)$, so we reject $H_0$. The p-value is smaller than $0.05$.

(c) Using only the interval and $n = 333$, compute the p-value for testing $H_0: \beta_1 = 0$ vs $H_A: \beta_1 \neq 0$.

Hint (c)

The confidence interval is symmetric around $\widehat{\beta}_1$. So you can recover $\widehat{\beta}_1$ and $\widehat{\text{SE}}(\widehat{\beta}_1)$ from the calculated interval. Then, compute the observed test statistic and the p-value.

Solution (c)

Since $0 \notin (47.12, 53.18)$, we already know the p-value is smaller than $0.05$. But the interval has more information than that. It has the form $\widehat{\beta}_1 \pm t^*_{n-2,0.975}\,\widehat{\text{SE}}(\widehat{\beta}_1)$, so we can recover both $\widehat{\beta}_1$ and $\widehat{\text{SE}}(\widehat{\beta}_1)$.

The estimate is the midpoint of the interval:

\[ \widehat{\beta}_1 = \frac{47.12 + 53.18}{2} = 50.15. \]

The width of the interval is $53.18 - 47.12 = 6.06 = 2\times t^*_{331,0.975}\,\widehat{\text{SE}}(\widehat{\beta}_1)$, so \[ \widehat{\text{SE}}(\widehat{\beta}_1) = \frac{(53.18 - 47.12)/2}{t^*_{331,0.975}} = \frac{3.03}{1.967} \approx 1.540. \]

Then, the observed test statistic is

\[ t_{\text{obs}} = \frac{\widehat{\beta}_1 - 0}{\widehat{\text{SE}}(\widehat{\beta}_1)} = \frac{50.15}{1.540} \approx 32.6, \]

and the p-value is $2\mathbb{P}(T \geq 32.6)$ with $T \sim t_{331}$, which is about $3 \times 10^{-105}$. So, there is overwhelming evidence against $H_0: \beta_1 = 0$.

t_star <- qt(0.975, df = 331)
se <- (53.18 - 47.12) / 2 / t_star
t_obs <- (47.12 + 53.18) / 2 / se
2 * pt(abs(t_obs), df = 331, lower.tail = FALSE)

Exercise 2.5.19

Suppose that, instead of recording $x_i$ and $Y_i$, we record them in different units: $x_i^* = a x_i + b$ and $Y_i^* = c Y_i + d$, where $a, b, c, d$ are constants with $a \neq 0$ and $c \neq 0$. For example:

flipper_length_cm $= 0.1 \times$ flipper_length_mm
body_mass_kg $= 0.001 \times$ body_mass_g

Let $\widehat{\beta}_1^*$ and $\widehat{\sigma}^*$ be the estimates obtained by regressing $Y^*$ on $x^*$.

(a) Express $\widehat{\beta}_1^*$, the fitted values $\widehat{Y}_i^*$, the residuals $e_i^*$, and $\widehat{\sigma}^*$ in terms of the estimates from the original model.

Hint (a)

Show that $x_i^* - \bar{x}^* = a(x_i - \bar{x})$ and $Y_i^* - \bar{Y}^* = c(Y_i - \bar{Y})$.
Then, remember that \[\widehat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{i=1}^n (x_i - \bar{x})^2}\]
Finally, remember that \[\widehat{Y}_i = \bar{Y} + \widehat{\beta}_1 (x_i - \bar{x})\]

Solution (a)

Note that \[ \begin{align*} \left(x^*_i-\bar{x}^*\right) &= \left[\left(ax_i+b\right) - \left(a\bar{x}+b\right)\right]\\ &= (ax_i - a\bar{x})\\ &= a(x_i - \bar{x})\\ \end{align*} \] Analogously, $\left(Y^*_i-\bar{Y}^*\right) = c(Y_i - \bar{Y})$. Then,

\[ \widehat{\beta}_1^* = \frac{\sum_{i=1}^n (x_i^* - \bar{x}^*)(Y_i^* - \bar{Y}^*)}{\sum_{i=1}^n (x_i^* - \bar{x}^*)^2} = \frac{\sum_{i=1}^n a(x_i - \bar{x})\,c(Y_i - \bar{Y})}{\sum_{i=1}^n a^2(x_i - \bar{x})^2} = \frac{c}{a}\widehat{\beta}_1. \]

For the fitted values, using the centered form, \[ \widehat{Y}_i^* = \bar{Y}^* + \widehat{\beta}_1^*(x_i^* - \bar{x}^*) = c\bar{Y} + d + \frac{c}{a}\widehat{\beta}_1\, a(x_i - \bar{x}) = c\left[\bar{Y} + \widehat{\beta}_1(x_i - \bar{x})\right] + d = c\,\widehat{Y}_i + d. \]

Therefore, $e_i^* = Y_i^* - \widehat{Y}_i^* = (cY_i + d) - (c\widehat{Y}_i + d) = c\,e_i$.

Next, \[ \text{RSS}^* = \sum_{i=1}^n (e_i^*)^2 = \sum_{i=1}^n (c e_i)^2 = c^2 \sum_{i=1}^n (e_i)^2 = c^2\,\text{RSS}. \]

And, finally \[ \widehat{\sigma}^* = \sqrt{\frac{\text{RSS}^*}{n-2}} = \sqrt{\frac{c^2\text{RSS}}{n-2}} = |c|\sqrt{\frac{\text{RSS}}{n-2}} = |c|\widehat{\sigma}. \]

$\blacksquare$

(b) Show that $\widehat{\text{SE}}(\widehat{\beta}_1^*) = \left|\frac{c}{a}\right|\widehat{\text{SE}}(\widehat{\beta}_1)$.

Hint (b)

Use the formula for $\widehat{\text{SE}}(\widehat{\beta}_1)$ from Exercise 2.5.11: \[ \widehat{\text{SE}}(\widehat{\beta}_1) = \frac{\widehat{\sigma}}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2}} \]

Solution (b)

\[ \widehat{\text{SE}}(\widehat{\beta}_1^*) = \frac{\widehat{\sigma}^*}{\sqrt{\sum_{i=1}^n (x^*_i - \bar{x}^*)^2}} = \frac{|c|\,\widehat{\sigma}}{\sqrt{a^2 \sum_{i=1}^n (x_i - \bar{x})^2}} = \left|\frac{c}{a}\right|\widehat{\text{SE}}(\widehat{\beta}_1) \quad \blacksquare \]

(c) Show that the test statistic for $H_0: \beta_1 = 0$ changes at most by its sign. What happens to the p-value of the two-sided test?

Solution (c)

\[ T^* = \frac{\widehat{\beta}_1^*}{\widehat{\text{SE}}(\widehat{\beta}_1^*)} = \frac{\frac{c}{a}\widehat{\beta}_1}{\left|\frac{c}{a}\right|\widehat{\text{SE}}(\widehat{\beta}_1)} = \text{sign}\left(\frac{c}{a}\right) T. \]

So, $|T^*| = |T|$ and the two-sided p-value does not change. The evidence about the association between $X$ and $Y$ does not depend on the units we use to measure them.

(d) For the penguins, $\widehat{\beta}_1 \approx 50.15$ g/mm. What is $\widehat{\beta}_1^*$ when using cm and kg?

Solution (d)

With $a = 1/10$ and $c = 1/1000$, $\widehat{\beta}_1^* = \frac{1/1000}{1/10}\widehat{\beta}_1 = \frac{\widehat{\beta}_1}{100} \approx 0.50$ kg/cm. Indeed, $50.15$ g/mm $= 501.5$ g/cm $= 0.5015$ kg/cm.

Exercise 2.5.20 (Challenging!)

An engineer wants to estimate the slope $\beta_1$ as precisely as possible. They can choose the values $x_1, \ldots, x_n$ anywhere in an interval $[a, b]$ (e.g., the temperature of an oven), and $n$ is even.

(a) Given that $\text{Var}(\widehat{\beta}_1)$ is associated with the error in the estimate, what should the engineer try to do when choosing the $x_i$’s?

Solution (a)

Since $\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$, and the engineer cannot control $\sigma^2$, they should choose the $x_i$’s to make $\sum_{i=1}^n (x_i - \bar{x})^2$ as large as possible, i.e., spread the $x$ values out as much as possible.

(b) Show that for any constant $m$, $\sum_{i=1}^n (x_i - m)^2 = \sum_{i=1}^n (x_i - \bar{x})^2 + n(\bar{x} - m)^2$.

Hint (b)

Write $x_i - m = x_i - \bar{x} + \bar{x} - m$ and expand the square.

Solution (b)

\[ \begin{aligned} \sum_{i=1}^n (x_i - m)^2 &= \sum_{i=1}^n \left[(x_i - \bar{x}) + (\bar{x} - m)\right]^2 \\ &= \sum_{i=1}^n (x_i - \bar{x})^2 + 2(\bar{x} - m)\sum_{i=1}^n (x_i - \bar{x}) + n(\bar{x} - m)^2 \\ &= \sum_{i=1}^n (x_i - \bar{x})^2 + n(\bar{x} - m)^2 \quad \text{ since } \sum_{i=1}^n (x_i - \bar{x}) = 0. \end{aligned} \]

(c) Use (b) to show that $\sum_{i=1}^n (x_i - \bar{x})^2 \leq \frac{n(b-a)^2}{4}$ for any choice of $x_i \in [a,b]$. Is this bound attainable? If so, which design attains this bound?

Hint (c)

By (b), $\sum_{i=1}^n (x_i - \bar{x})^2 \leq \sum_{i=1}^n (x_i - m)^2$ for any $m$. Which choice of $m$ makes the maximum possible distance $|x_i - m|$ for $x_i \in [a,b]$ as small as possible?

Solution (c)

Let $m$ be any constant. By Item (b) above,

\[ \sum_{i=1}^n (x_i - \bar{x})^2 = \sum_{i=1}^n (x_i - m)^2 - n(\bar{x} - m)^2 \leq \sum_{i=1}^n (x_i - m)^2, \]

because $n(\bar{x} - m)^2 \geq 0$. Now, since each $x_i$ is in $[a,b]$, the distance $|x_i - m|$ is at most the distance from $m$ to the farthest endpoint $a$ or $b$, so

\[ \sum_{i=1}^n (x_i - \bar{x})^2 \leq \sum_{i=1}^n (x_i - m)^2 \leq n \max\{(m-a)^2, (b-m)^2\}. \]

So, $n \max\{(m-a)^2, (b-m)^2\}$ is an upper bound for $\sum_{i=1}^n (x_i - \bar{x})^2$, and it is valid for every choice of $m$ and of $x_1, \ldots, x_n \in [a,b]$. Different values of $m$ give different valid bounds, so we look for the $m$ that gives the smallest one, i.e., the one that minimizes $\max\{|m-a|, |b-m|\}$.

Every real number $m$ is either the midpoint $\frac{a+b}{2}$, or $\frac{a+b}{2} + \varepsilon$, or $\frac{a+b}{2} - \varepsilon$ for some $\varepsilon > 0$. We compute $\max\{|m-a|, |b-m|\}$ in each case:

If $m = \frac{a+b}{2}$, then $|m-a| = |b-m| = \frac{b-a}{2}$, so the maximum is $\frac{b-a}{2}$.
If $m = \frac{a+b}{2} + \varepsilon$, then $|m - a| = \frac{b-a}{2} + \varepsilon$ and $|b - m| = \left|\frac{b-a}{2} - \varepsilon\right| \leq \frac{b-a}{2} + \varepsilon$. So the maximum is $\frac{b-a}{2} + \varepsilon$.
If $m = \frac{a+b}{2} - \varepsilon$, the same computation with the roles of $a$ and $b$ swapped gives $|b - m| = \frac{b-a}{2} + \varepsilon$ and $|m - a| = \left|\frac{b-a}{2} - \varepsilon\right| \leq \frac{b-a}{2} + \varepsilon$. So the maximum is again $\frac{b-a}{2} + \varepsilon$.

Since $\frac{b-a}{2} < \frac{b-a}{2} + \varepsilon$ for every $\varepsilon > 0$, the midpoint is the only value of $m$ that gives the smallest bound:

\[ \sum_{i=1}^n (x_i - \bar{x})^2 \leq n\left(\frac{b-a}{2}\right)^2 = \frac{n(b-a)^2}{4} \quad \text{for any choice of } x_1, \ldots, x_n \in [a,b]. \quad \blacksquare \]

Next, we show that this best bound can actually be reached.

Suppose $n/2$ observations are at $x = a$ and $n/2$ are at $x = b$ (this is where we need $n$ even). Then $\bar{x} = \frac{a+b}{2}$, and every $x_i$ is exactly $\frac{b-a}{2}$ away from $\bar{x}$. Therefore,

\[ \sum_{i=1}^n (x_i - \bar{x})^2 = n\left(\frac{b-a}{2}\right)^2 = \frac{n(b-a)^2}{4}. \]

(d) What is the main drawback of the design you found in (c)?

Hint (d)

How many distinct $x$ values does the design have? Could you check the linearity assumption?

Solution (d)

This design only has two distinct values of $x$. Any curve passing through the two group means fits the data equally well, so we have no way to check if the relationship is linear: the residual plot would show only two vertical strips of points. If the true relationship is curved, we would not notice it. The engineer could put most points at the extremes and a few in the middle, trading a bit of precision for the ability to check linearity.

Exercise 2.6 - Expected Response & Prediction

Exercise 2.6.1

Compute $\text{Var}(\widehat{Y}_{\text{new}})$ for a new observation $Y_{\text{new}}$ at $x_{\text{new}}$.

Hint

Write $\widehat{Y}_{\text{new}} = \widehat{\beta}_0 + \widehat{\beta}_1 x_{\text{new}}$ either:

In centered form: $\widehat{Y}_{\text{new}} = \bar{Y} + \widehat{\beta}_1(x_{\text{new}} - \bar{x})$, using $\text{Cov}(\bar{Y}, \widehat{\beta}_1) = 0$ from Exercise 2.5.6; or
Directly from $\text{Var}(\widehat{\beta}_0 + x_{\text{new}}\widehat{\beta}_1)$, using $\text{Var}(\widehat{\beta}_0)$ from Exercise 2.5.7, $\text{Var}(\widehat{\beta}_1)$ from Exercise 2.5.3, and $\text{Cov}(\widehat{\beta}_0, \widehat{\beta}_1)$ from Exercise 2.5.13.

Solution

Solution 1:

Since $\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1 \bar{x}$, we rewrite $\widehat{Y}_{\text{new}}$ as:

Recall from previous exercises:

$\text{Var}(\bar{Y}) = \frac{\sigma^2}{n}$
$\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$ (Exercise 2.5.3)
$\text{Cov}(\bar{Y}, \widehat{\beta}_1) = 0$ (Exercise 2.5.6)

Substituting these values:

\[ \begin{align*} \text{Var}\left[\widehat{Y}_{\text{new}}\right] &= \text{Var}\left[\bar{Y} + \widehat{\beta}_1(x_{\text{new}} - \bar{x})\right]\\ &= \text{Var}(\bar{Y}) + (x_{\text{new}} - \bar{x})^2 \text{Var}(\widehat{\beta}_1) + 2(x_{\text{new}} - \bar{x})\text{Cov}(\bar{Y}, \widehat{\beta}_1)\\ &= \frac{\sigma^2}{n} + (x_{\text{new}} - \bar{x})^2 \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} + 0 \\ &= \sigma^2 \left[ \frac{1}{n} + \frac{(x_{\text{new}} - \bar{x})^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right] \quad \blacksquare \end{align*} \]

Solution 2:

Alternatively, expanding directly:

From Exercises 2.5.3, 2.5.7, and 2.5.13:

$\text{Var}(\widehat{\beta}_0) = \sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\right)$
$\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$
$\text{Cov}(\widehat{\beta}_0, \widehat{\beta}_1) = -\frac{\bar{x}\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}$

Substituting these terms:

\[ \begin{align} \text{Var}(\widehat{Y}_{\text{new}}) &= \text{Var}(\widehat{\beta}_0 + x_{\text{new}}\widehat{\beta}_1)\\ &= \text{Var}(\widehat{\beta}_0) + x_{\text{new}}^2\text{Var}(\widehat{\beta}_1) + 2x_{\text{new}}\text{Cov}(\widehat{\beta}_0, \widehat{\beta}_1)\\ &= \sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\right) + x_{\text{new}}^2 \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} - 2x_{\text{new}}\bar{x}\frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \\ &= \frac{\sigma^2}{n} + \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\left( \bar{x}^2 - 2x_{\text{new}}\bar{x} + x_{\text{new}}^2 \right) \\ &= \sigma^2 \left[ \frac{1}{n} + \frac{(x_{\text{new}} - \bar{x})^2}{\sum_{i=1}^n (x_i - \bar{x})^2} \right] \quad \blacksquare \end{align} \]

Notice that $\text{Var}(\widehat{Y}_{\text{new}})$ is minimized at $x_{\text{new}} = \bar{x}$, where it equals $\sigma^2/n$, and increases as $x_{\text{new}}$ moves farther away from the center of the training data $\bar{x}$.

Exercise 2.6.2

Compute $\text{Cov}(\widehat{Y}_i, \widehat{Y}_j)$ for any $i, j \in \{1, \dots, n\}$.

Hint

Express both fitted values in centered form: \[ \widehat{Y}_i = \bar{Y} + \widehat{\beta}_1(x_i - \bar{x}) \quad \text{and} \quad \widehat{Y}_j = \bar{Y} + \widehat{\beta}_1(x_j - \bar{x}) \] Expand $\text{Cov}(\widehat{Y}_i, \widehat{Y}_j)$ using the bilinearity of covariance. Then use the fact that $\text{Cov}(\bar{Y}, \widehat{\beta}_1) = 0$ (Exercise 2.5.6), $\text{Var}(\bar{Y}) = \frac{\sigma^2}{n}$, and the variance formula for $\widehat{\beta}_1$ (Exercise 2.5.3).

Solution

Express both fitted values in centered form:

\[ \widehat{Y}_i = \bar{Y} + \widehat{\beta}_1(x_i - \bar{x}) \quad \text{and} \quad \widehat{Y}_j = \bar{Y} + \widehat{\beta}_1(x_j - \bar{x}) \]

Using the bilinearity of covariance:

\[ \begin{align*} \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) &= \text{Cov}\left(\bar{Y} + \widehat{\beta}_1(x_i - \bar{x}), \; \bar{Y} + \widehat{\beta}_1(x_j - \bar{x})\right) \\ &= \text{Cov}(\bar{Y}, \bar{Y}) + (x_i - \bar{x})\text{Cov}(\widehat{\beta}_1, \bar{Y}) + (x_j - \bar{x})\text{Cov}(\bar{Y}, \widehat{\beta}_1) + (x_i - \bar{x})(x_j - \bar{x})\text{Cov}(\widehat{\beta}_1, \widehat{\beta}_1) \\ &= \text{Var}(\bar{Y}) + (x_i - \bar{x})\text{Cov}(\widehat{\beta}_1, \bar{Y}) + (x_j - \bar{x})\text{Cov}(\bar{Y}, \widehat{\beta}_1) + (x_i - \bar{x})(x_j - \bar{x})\text{Var}(\widehat{\beta}_1) \end{align*} \]

From previous exercises:

$\text{Var}(\bar{Y}) = \frac{\sigma^2}{n}$
$\text{Cov}(\bar{Y}, \widehat{\beta}_1) = 0$ (Exercise 2.5.6)
$\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{k=1}^n (x_k - \bar{x})^2}$ (Exercise 2.5.3)

Substituting these terms:

\[ \begin{align*} \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) &= \frac{\sigma^2}{n} + 0 + 0 + (x_i - \bar{x})(x_j - \bar{x}) \frac{\sigma^2}{\sum_{k=1}^n (x_k - \bar{x})^2} \\ &= \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \quad \blacksquare \end{align*} \]

Notice that when $i = j$:

\[ \text{Cov}(\widehat{Y}_i, \widehat{Y}_i) = \text{Var}(\widehat{Y}_i) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})^2}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \]

which agrees with the variance formula derived in Exercise 2.6.1 evaluated at $x_{\text{new}} = x_i$. In matrix notation, $\widehat{\mathbf{Y}} = \mathbf{H}\mathbf{Y}$ with hat matrix $\mathbf{H}$, so $\text{Cov}(\widehat{Y}_i, \widehat{Y}_j) = \sigma^2 h_{ij}$, where $h_{ij} = \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2}$.

Exercise 2.6.3

Compute $\text{Cov}(Y_i, \widehat{Y}_j)$ for any $i, j \in \{1, \dots, n\}$.

Hint

Express the fitted value as $\widehat{Y}_j = \bar{Y} + \widehat{\beta}_1(x_j - \bar{x})$ and apply linearity of covariance: \[ \text{Cov}\left(Y_i, \; \bar{Y} + (x_j - \bar{x})\widehat{\beta}_1\right) \] Alternatively, express $Y_i = \widehat{Y}_i + e_i$ and use the results from Exercises 2.5.9 and 2.5.10.

Solution

Solution 1:

Express the observed response as $Y_i = \widehat{Y}_i + e_i$:

\[ \begin{align*} \text{Cov}(Y_i, \widehat{Y}_j) &= \text{Cov}(\widehat{Y}_i + e_i, \; \widehat{Y}_j) \\ &= \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) + \text{Cov}(e_i, \widehat{Y}_j) \end{align*} \]

Writing the fitted value as $\widehat{Y}_j = \widehat{\beta}_0 + \widehat{\beta}_1 x_j$:

\[ \text{Cov}(e_i, \widehat{Y}_j) = \text{Cov}(e_i, \widehat{\beta}_0) + x_j \text{Cov}(e_i, \widehat{\beta}_1) \]

From Exercises 2.5.9 and 2.5.10, each residual is uncorrelated with the estimated coefficients: $\text{Cov}(e_i, \widehat{\beta}_0) = 0$ and $\text{Cov}(e_i, \widehat{\beta}_1) = 0$. Therefore, $\text{Cov}(e_i, \widehat{Y}_j) = 0$.

Therefore, from Exercise 2.6.2:

\[ \text{Cov}(Y_i, \widehat{Y}_j) = \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \quad \blacksquare \]

Solution 2:

Alternatively, express $\widehat{Y}_j$ in centered form:

\[ \widehat{Y}_j = \bar{Y} + \widehat{\beta}_1(x_j - \bar{x}) \]

Using the linearity of covariance:

\[ \begin{align*} \text{Cov}\left(Y_i, \widehat{Y}_j\right) &= \text{Cov}\left(Y_i, \bar{Y} + \widehat{\beta}_1(x_j - \bar{x})\right) \\ &= \text{Cov}\left(Y_i, \bar{Y}\right) + (x_j - \bar{x})\text{Cov}(Y_i, \widehat{\beta}_1) \end{align*} \]

Evaluating each component:

Writing $\bar{Y} = \frac{1}{n}\sum_{k=1}^n Y_k$ and using that observations are independent with $\text{Var}(Y_i) = \sigma^2$: \[ \text{Cov}(Y_i, \bar{Y}) = \text{Cov}\left(Y_i, \frac{1}{n}\sum_{k=1}^n Y_k\right) = \frac{1}{n}\text{Var}(Y_i) = \frac{\sigma^2}{n} \]
From the derivation in Exercise 2.5.9: \[ \text{Cov}(Y_i, \widehat{\beta}_1) = \frac{x_i - \bar{x}}{\sum_{k=1}^n (x_k - \bar{x})^2}\sigma^2 \]

Substituting these results into the covariance expression:

\[ \begin{align*} \text{Cov}(Y_i, \widehat{Y}_j) &= \text{Cov}\left(Y_i, \bar{Y}\right) + (x_j - \bar{x})\text{Cov}(Y_i, \widehat{\beta}_1) \\ &= \frac{\sigma^2}{n} + (x_j - \bar{x})\frac{x_i - \bar{x}}{\sum_{k=1}^n (x_k - \bar{x})^2}\sigma^2 \\ &= \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \quad \blacksquare \end{align*} \]

Notice that when $i = j$:

\[ \text{Cov}(Y_i, \widehat{Y}_i) = \text{Var}(\widehat{Y}_i) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})^2}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \]

which again equals the variance formula from Exercise 2.6.1 evaluated at $x_{\text{new}} = x_i$.

Exercise 2.7 - Properties of the residuals

Exercise 2.7.1

Show that $\widehat{\sigma}^2 = \frac{\text{RSS}}{n-2}$ is an unbiased estimator of $\sigma^2$ (i.e., $E[\widehat{\sigma}^2] = \sigma^2$).

Hint

Express $\text{RSS} = \sum_{i=1}^n e_i^2$ where $e_i = Y_i - \widehat{Y}_i$. Show that $E[e_i] = 0$ so that $E[e_i^2] = \text{Var}(e_i)$, and compute $\text{Var}(e_i)$ using the results from Exercises 2.6.2 and 2.6.3.

Solution

Express the residual as $e_i = Y_i - \widehat{Y}_i$. First, compute $\text{Var}(e_i)$:

\[ \begin{align*} \text{Var}(e_i) &= \text{Var}(Y_i - \widehat{Y}_i) \\ &= \text{Var}(Y_i) + \text{Var}(\widehat{Y}_i) - 2\text{Cov}(Y_i, \widehat{Y}_i) \end{align*} \]

Using the model assumption $\text{Var}(Y_i) = \sigma^2$ and the results from Exercises 2.6.2 and 2.6.3:

$\text{Var}(\widehat{Y}_i) = \text{Cov}(\widehat{Y}_i, \widehat{Y}_i) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})^2}{\sum_{k=1}^n (x_k - \bar{x})^2} \right]$ (Exercise 2.6.2 setting $j = i$, or Exercise 2.6.1)
$\text{Cov}(Y_i, \widehat{Y}_i) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})^2}{\sum_{k=1}^n (x_k - \bar{x})^2} \right]$ (Exercise 2.6.3 setting $j = i$)

Since $\text{Cov}(Y_i, \widehat{Y}_i) = \text{Var}(\widehat{Y}_i)$, the variance simplifies directly to:

\[ \begin{align*} \text{Var}(e_i) &= \text{Var}(Y_i) - \text{Var}(\widehat{Y}_i) \\ &= \sigma^2 - \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})^2}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \\ &= \sigma^2 \left( 1 - \frac{1}{n} - \frac{(x_i - \bar{x})^2}{\sum_{k=1}^n (x_k - \bar{x})^2} \right) \end{align*} \]

First, note that $E[e_i] = E[Y_i - \widehat{Y}_i] = E[Y_i] - E[\widehat{Y}_i] = (\beta_0 + \beta_1 x_i) - (\beta_0 + \beta_1 x_i) = 0$ because the OLS estimators are unbiased (Exercises 2.5.1 and 2.5.5).

Since $E[e_i] = 0$, we have $E[e_i^2] = \text{Var}(e_i)$. Taking the expectation of $\text{RSS} = \sum_{i=1}^n e_i^2$:

\[ \begin{align*} E[\text{RSS}] &= E\left[ \sum_{i=1}^n e_i^2 \right] = \sum_{i=1}^n E[e_i^2] = \sum_{i=1}^n \text{Var}(e_i) \\ &= \sigma^2 \sum_{i=1}^n \left( 1 - \frac{1}{n} - \frac{(x_i - \bar{x})^2}{\sum_{k=1}^n (x_k - \bar{x})^2} \right) \\ &= \sigma^2 \left( n - \frac{n}{n} - \frac{\sum_{i=1}^n (x_i - \bar{x})^2}{\sum_{k=1}^n (x_k - \bar{x})^2} \right) \\ &= \sigma^2 (n - 1 - 1) \\ &= \sigma^2 (n - 2) \end{align*} \]

Therefore:

\[ E[\widehat{\sigma}^2] = E\left[ \frac{\text{RSS}}{n - 2} \right] = \frac{E[\text{RSS}]}{n - 2} = \frac{\sigma^2 (n - 2)}{n - 2} = \sigma^2 \quad \blacksquare \]

Exercise 2.7.2

Show that, differently from the errors, the residuals are correlated (i.e., $\text{Cov}(e_i, e_j) \neq 0$ for $i \neq j$). Besides, explain what happens to the correlation between residuals as the sample size $n$ increases.

Hint

Expand $\text{Cov}(e_i, e_j) = \text{Cov}(Y_i - \widehat{Y}_i, \, Y_j - \widehat{Y}_j)$ using the bilinearity of covariance, recalling that $\text{Cov}(Y_i, Y_j) = 0$ for $i \neq j$.
Substitute the covariance results derived in Exercises 2.6.2 and 2.6.3: $\text{Cov}(\widehat{Y}_i, \widehat{Y}_j)$, $\text{Cov}(Y_i, \widehat{Y}_j)$, and $\text{Cov}(\widehat{Y}_i, Y_j)$.

Solution

Express the residuals as $e_i = Y_i - \widehat{Y}_i$ and $e_j = Y_j - \widehat{Y}_j$. Expanding $\text{Cov}(e_i, e_j)$ by the bilinearity of covariance:

\[ \begin{align*} \text{Cov}(e_i, e_j) &= \text{Cov}(Y_i - \widehat{Y}_i, \; Y_j - \widehat{Y}_j) \\ &= \text{Cov}(Y_i, Y_j) - \text{Cov}(Y_i, \widehat{Y}_j) - \text{Cov}(\widehat{Y}_i, Y_j) + \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) \end{align*} \]

Since observations are independent, $\text{Cov}(Y_i, Y_j) = 0$ for $i \neq j$.

Now apply the results from Exercises 2.6.2 and 2.6.3:

From Exercise 2.6.2: \[ \text{Cov}\left(\widehat{Y}_i, \widehat{Y}_j\right) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \]
From Exercise 2.6.3: \[ \text{Cov}\left(Y_i, \widehat{Y}_j\right) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \]
By symmetry of covariance and Exercise 2.6.3: \[ \text{Cov}\left(\widehat{Y}_i, Y_j\right) = \text{Cov}\left(Y_j, \widehat{Y}_i\right) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_j - \bar{x})(x_i - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] = \text{Cov}(Y_i, \widehat{Y}_j) \]

Notice that all three terms are identical to $\text{Cov}(\widehat{Y}_i, \widehat{Y}_j)$:

\[ \text{Cov}(Y_i, \widehat{Y}_j) = \text{Cov}(\widehat{Y}_i, Y_j) = \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) \]

Substituting these back into the covariance expansion:

\[ \begin{align*} \text{Cov}(e_i, e_j) &= 0 - \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) - \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) + \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) \\ &= -\text{Cov}(\widehat{Y}_i, \widehat{Y}_j) \\ &= -\sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \quad \blacksquare \end{align*} \]

Because $\frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \neq 0$ in general, the residuals are correlated (unlike the true errors $\varepsilon_i$, which are uncorrelated).

As the sample size $n$ increases, both $\frac{1}{n} \to 0$ and $\frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \to 0$, so $\text{Cov}(e_i, e_j) \to 0$ and the correlation between residuals approaches zero. This means that as $n$ gets large, the residuals become approximately uncorrelated.

Exercise 2.7.3

A very interesting and counterintuitive property of the residuals is that they are uncorrelated with the fitted values. Even though the residual $e_i = Y_i - \widehat{Y}_i$ and the fitted value $\widehat{Y}_i$ are both computed from the same observed responses, they turn out to be uncorrelated.

Show that $\text{Cov}(e_i, \widehat{Y}_j) = 0$ for all $i, j \in \{1, \dots, n\}$.

Hint

There are two ways to approach this:

Express $\widehat{Y}_j = \widehat{\beta}_0 + x_j \widehat{\beta}_1$ and apply the results from Exercises 2.5.9 and 2.5.10 ($\text{Cov}(e_i, \widehat{\beta}_1) = 0$ and $\text{Cov}(e_i, \widehat{\beta}_0) = 0$).
Alternatively, express $e_i = Y_i - \widehat{Y}_i$ and apply the linearity of covariance: \[ \text{Cov}(e_i, \widehat{Y}_j) = \text{Cov}(Y_i, \widehat{Y}_j) - \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) \] Then substitute the covariance formulas derived in Exercises 2.6.2 and 2.6.3.

Solution

Solution 1 (Using Estimated Coefficients and Exercises 2.5.9 and 2.5.10):

Express the fitted value as $\widehat{Y}_j = \widehat{\beta}_0 + x_j \widehat{\beta}_1$. Using the linearity of covariance:

\[ \begin{align*} \text{Cov}(e_i, \widehat{Y}_j) &= \text{Cov}(e_i, \; \widehat{\beta}_0 + x_j \widehat{\beta}_1) \\ &= \text{Cov}(e_i, \widehat{\beta}_0) + x_j \text{Cov}(e_i, \widehat{\beta}_1) \end{align*} \]

From Exercises 2.5.9 and 2.5.10, each residual is uncorrelated with both OLS coefficient estimators:

$\text{Cov}(e_i, \widehat{\beta}_1) = 0$ (Exercise 2.5.9)
$\text{Cov}(e_i, \widehat{\beta}_0) = 0$ (Exercise 2.5.10)

Therefore:

\[ \text{Cov}(e_i, \widehat{Y}_j) = 0 + x_j(0) = 0 \quad \blacksquare \]

Solution 2 (Using Exercises 2.6.2 and 2.6.3):

Express the residual as $e_i = Y_i - \widehat{Y}_i$. Using the linearity of covariance:

\[ \begin{align*} \text{Cov}(e_i, \widehat{Y}_j) &= \text{Cov}(Y_i - \widehat{Y}_i, \; \widehat{Y}_j) \\ &= \text{Cov}(Y_i, \widehat{Y}_j) - \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) \end{align*} \]

From Exercise 2.6.3 (via the direct expansion in Solution 2):

\[ \text{Cov}(Y_i, \widehat{Y}_j) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \]

From Exercise 2.6.2:

\[ \text{Cov}(\widehat{Y}_i, \widehat{Y}_j) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] \]

Subtracting the two terms:

\[ \text{Cov}(e_i, \widehat{Y}_j) = \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] - \sigma^2 \left[ \frac{1}{n} + \frac{(x_i - \bar{x})(x_j - \bar{x})}{\sum_{k=1}^n (x_k - \bar{x})^2} \right] = 0 \quad \blacksquare \]

In particular, when $i = j$, $\text{Cov}(e_i, \widehat{Y}_i) = 0$. Even though each residual $e_i = Y_i - \widehat{Y}_i$ involves $\widehat{Y}_i$, the least-squares fitting procedure decomposes the response $Y_i$ into two uncorrelated components: the fitted value $\widehat{Y}_i$ and the residual $e_i$. Furthermore, because $Y_1, \dots, Y_n$ are normally distributed, zero covariance implies that the residuals and fitted values are statistically independent.

Exercise 2.8 - Application: cargo e-bike energy consumption

A sustainable-delivery cooperative tested a cargo e-bike across $n = 18$ dry days with a standardized payload. On each day $i$, they recorded the outdoor temperature (temp, in $^\circ\text{C}$, ranging from $-4^\circ\text{C}$ to $24^\circ\text{C}$) and the average battery energy consumption (energy, in Wh/km). Because cold air is denser and increases battery internal resistance, energy consumption per kilometre is higher on cold days and decreases as temperature rises. They fitted the simple linear regression model

\[ \text{energy}_i = \beta_0 + \beta_1\,\text{temp}_i + \varepsilon_i, \qquad \varepsilon_i \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^2). \]

Some entries of the R output were accidentally smudged and are denoted [A], [B], [C], and [D]:

> fit <- lm(energy ~ temp, data = ebike)
> summary(fit)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)  42.4204     0.6411    [A]   < 2e-16
temp         -0.8455       [B]  -17.38  8.23e-12

Residual standard error: 1.674 on 16 degrees of freedom
Multiple R-squared:    [C], Adjusted R-squared:  0.9466
F-statistic:   [D] on 1 and 16 DF,  p-value: 8.228e-12

The sample summary statistics are:

$\bar{x} = 10.3889^\circ\text{C}$,
$S_{xx} = \sum_{i=1}^{18}(x_i - \bar{x})^2 = 1184.2778$,
$\bar{y} = 33.6367\text{ Wh/km}$, and
$TSS = \sum_{i=1}^{18}(y_i - \bar{y})^2 = 891.4226$.

Use $t^*_{16, 0.975} = 2.1199$.

Exercise 2.8.1

Reconstruct the four missing numbers [A], [B], [C], and [D]. Show two different ways to obtain [D] (one from $RSS$ and $TSS$, and one from the $t$-statistic of temp).

Hint

$T = \widehat{\beta}_j / \widehat{\text{SE}}(\widehat{\beta}_j)$,
$\widehat{\sigma}^2 = \frac{RSS}{n-2}$,
$R^2 = 1 - RSS/TSS$, and
$F = \frac{MSS/1}{RSS/(n-2)} = T_{\text{temp}}^2$.

Solution

[A]: $t_{\text{Intercept}} = \frac{42.4204}{0.6411} = 66.1681 \approx 66.17$.
[B]: $\widehat{\text{SE}}(\widehat{\beta}_1) = \frac{-0.8455}{-17.38} = 0.0486$ (or from $\frac{\widehat{\sigma}}{\sqrt{S_{xx}}} = \frac{1.674}{\sqrt{1184.2778}} = 0.04864$).
[C]: $RSS = (n-2)\widehat{\sigma}^2 = 16 \times 1.674^2 = 44.8364$ (R’s unrounded value is $44.8349$). Thus $R^2 = 1 - \frac{44.8364}{891.4226} = 0.9497$.
[D]:
- Method 1 (sums of squares): $MSS = TSS - RSS = 891.4226 - 44.8364 = 846.5862$, so $F = \frac{846.5862 / 1}{44.8364 / 16} = \mathbf{302.1}$.
- Method 2 (slope $t$-statistic): In SLR, \[F = T_{\text{temp}}^2 = \left(\frac{-0.8455}{\frac{1.674}{\sqrt{1184.2778}}}\right)^2 = (-17.38141)^2 \approx \mathbf{302.1134}\]

Exercise 2.8.2

Write the fitted regression line and interpret both $\widehat{\beta}_0$ and $\widehat{\beta}_1$ in context. Is the intercept $\beta_0$ meaningful here? Can we conclude from this study alone that warmer temperature causes the change in battery energy consumption? Explain briefly.

Solution

The fitted line is $\widehat{\text{energy}} = 42.4204 - 0.8455\,\text{temp}$.

$\widehat{\beta}_0 = 42.4204$: At $0^\circ\text{C}$ (freezing point), the expected e-bike battery energy consumption is $42.4204\text{ Wh/km}$. Because $0^\circ\text{C}$ lies well within the observed temperature range $[-4^\circ\text{C}, 24^\circ\text{C}]$, evaluating the model at $x = 0$ is not an extrapolation and $\beta_0$ is meaningful in this context.
$\widehat{\beta}_1 = -0.8455$: Each $1^\circ\text{C}$ increase in outdoor temperature is associated with an average decrease of $0.8455\text{ Wh/km}$ in battery energy consumption.
No, we cannot conclude causation from this study alone: because temperature was observed across days rather than randomly assigned in a controlled test chamber, unobserved day-level confounders (such as wind speed, traffic stop-and-go patterns, or rider pacing on cold vs. warm days) could also contribute to the observed slope.

Exercise 2.8.3

A partner cooperative in the United States wants the model expressed with energy consumption in watt-hours per mile ($1\text{ mile} = 1.609344\text{ km}$) and temperature in degrees Fahrenheit ($\text{temp}^* = 1.8 \times \text{temp} + 32$).

Without refitting the model, find the new fitted slope $\widehat{\beta}_1^*$, the new fitted intercept $\widehat{\beta}_0^*$, the new residual standard error $\widehat{\sigma}^*$, and the new $R^2$ for regressing $\text{energy}^*$ on $\text{temp}^*$.

Hint

See Exercise 2.5.19.

Solution

Substituting $\text{temp}_i = \frac{\text{temp}_i^* - 32}{1.8}$ into $\widehat{Y}_i^* = c(\widehat{\beta}_0 + \widehat{\beta}_1\,\text{temp}_i)$ gives:

\[ \widehat{Y}_i^* = \underbrace{c\left(\widehat{\beta}_0 - \frac{32}{1.8}\widehat{\beta}_1\right)}_{\widehat{\beta}_0^*} + \underbrace{\left(\frac{c}{1.8}\widehat{\beta}_1\right)}_{\widehat{\beta}_1^*} \text{temp}_i^*. \]

$\widehat{\beta}_1^* = \frac{1.609344}{1.8}(-0.8455) = -0.7559\text{ Wh/mile per }^\circ\text{F}$.
$\widehat{\beta}_0^* = 1.609344\times\left(42.4204 - \frac{32}{1.8}(-0.8455)\right) = 1.609344(57.4515) \approx 92.4592\text{ Wh/mile}$.
$\widehat{\sigma}^* = c\widehat{\sigma} = 1.609344 \times 1.674 = 2.6940\text{ Wh/mile}$.
$R^2 = 0.9497$ (unchanged, as $R^2$ is invariant to linear rescaling and shifting).

Exercise 2.8.4

Compute the 95% confidence intervals for $\beta_0$ and $\beta_1$, and interpret both intervals in context.

Solution

$\text{CI}\left(\beta_0, 0.95\right) = 42.4204 \pm 2.1199 \times 0.6411 = 42.4204 \pm 1.3591 = [41.0613,\; 43.7795]\text{ Wh/km}$
- Interpretation: We are 95% confident that at $0^\circ\text{C}$, the expected (mean) e-bike battery energy consumption is between $41.0613\text{ Wh/km}$ and $43.7795\text{ Wh/km}$.
$\text{CI}\left(\beta_1, 0.95\right) = -0.8455 \pm 2.1199 \times 0.0486 = [-0.9485,\; -0.7425]\text{ Wh/km per }^\circ\text{C}$
- Interpretation: We are 95% confident that each $1^\circ\text{C}$ increase in outdoor temperature is associated with an expected decrease in battery energy consumption between $0.7425\text{ Wh/km}$ and $0.9485\text{ Wh/km}$ (i.e., an expected change between $-0.9485$ and $-0.7425\text{ Wh/km}$).

Exercise 2.8.5

Compute the 95% confidence interval for the mean energy consumption and the 95% prediction interval for the energy consumption of a new trip at $10^\circ\text{C}$, and compare them to the corresponding intervals at $-4^\circ\text{C}$ (where the $95\%$ CI is $[44.0991, 47.5056]$ and the $95\%$ PI is $[41.8661, 49.7386]$).

Why are the intervals narrower at $10^\circ\text{C}$ than at $-4^\circ\text{C}$? Since $10^\circ\text{C}$ is almost equal to $\bar{x} = 10.3889^\circ\text{C}$, why doesn’t the 95% confidence interval at $10^\circ\text{C}$ have width almost zero?

Solution

At $x_0 = 10^\circ\text{C}$, $\widehat{y}_0 = 42.4204 - 0.8455(10) = 33.9654\text{ Wh/km}$, and

The confidence interval: \[ \begin{align} \text{CI}\left(\mu_{Y_0}, 0.95\right) &= \hat{y}_0 \pm t^*_{16, 0.975}\,\widehat{\sigma}\sqrt{\frac{1}{n} + \frac{(x_0 - \bar{x})^2}{S_{xx}}} \\ &= 33.9654 \pm 2.1199 \times 1.674\sqrt{0.05568} \\ &= 33.9654 \pm 0.8374 \\ &= [33.1280,\; 34.8028]\text{ Wh/km} \end{align} \]
The prediction interval: \[ \begin{align} \text{PI}\left(Y_{\text{new}}, 0.95\right) &= \hat{y}_0 \pm t^*_{16, 0.975}\,\widehat{\sigma}\sqrt{1 + \frac{1}{n} + \frac{(x_0 - \bar{x})^2}{S_{xx}}} \\ &= 33.9654 \pm 2.1199 \times 1.674\sqrt{1 + 0.05568} \\ &= 33.9654 \pm 3.6461\\ &= [30.3192,\; 37.6115]\text{ Wh/km} \end{align} \]

When calculating the prediction interval, we need to account for two sources of uncertainty: (i) the irreducible random noise $\varepsilon$ in a new observation, and (ii) the uncertainty in the fitted line $\widehat{Y} = \widehat{\beta}_0 + \widehat{\beta}_1 x$. The confidence interval for the mean response accounts for the second source of uncertainty.

Comparison with $-4^\circ\text{C}$: here $\widehat{y}_0 = 42.4204 - 0.8455(-4) = 45.8024\text{ Wh/km}$ and $(x_0 - \bar{x})^2 = (-4 - 10.3889)^2 = 207.0404$, so \[ \begin{align} \text{CI}\left(\mu_{Y_0}, 0.95\right) &= 45.8024 \pm 2.1199 \times 1.674\sqrt{\frac{1}{18} + \frac{207.0404}{1184.2778}} \\ &= 45.8024 \pm 2.1199 \times 1.674\sqrt{0.23038} \\ &= 45.8024 \pm 1.7032 \\ &= [44.0991,\; 47.5056]\text{ Wh/km} \end{align} \] \[ \begin{align} \text{PI}\left(Y_{\text{new}}, 0.95\right) &= 45.8024 \pm 2.1199 \times 1.674\sqrt{1 + 0.23038} \\ &= 45.8024 \pm 3.9363 \\ &= [41.8661,\; 49.7387]\text{ Wh/km} \end{align} \]

As you can see, the CI at $10^\circ\text{C}$ is about half as wide as at $-4^\circ\text{C}$. The PI shrinks only slightly (about 7%).

Why are the intervals narrower at $10^\circ\text{C}$? The uncertainty in the fitted line depends on the distance from $x_0$ to $\bar{x}$ through the term $\frac{(x_0 - \bar{x})^2}{S_{xx}}$. At $x_0 = -4$, $(x_0 - \bar{x})^2 = (-14.3889)^2 = 207.04$, so $\frac{1}{n} + \frac{(x_0-\bar{x})^2}{S_{xx}} = 0.2304$. At $x_0 = 10$, $(x_0 - \bar{x})^2 = 0.1512$, so the same quantity is $0.0557$. The fitted line pivots around $(\bar{x}, \bar{y})$: the uncertainty in the slope $\widehat{\beta}_1$ moves the line more the farther we are from $\bar{x}$. Hence the estimate of the mean response is most precise at $\bar{x}$ and gets worse as $x_0$ moves away from it.

For the PI, the extra “$1$” inside the square root (the noise $\varepsilon$ of the new observation) does not depend on $x_0$, so it dominates both intervals. This is why the PI narrows only slightly.
Why isn’t the CI at $10^\circ\text{C}$ nearly zero, even though $x_0 \approx \bar{x}$? Even exactly at $x_0 = \bar{x}$, the fitted value is $\widehat{Y}_0 = \bar{Y}$, which is still a random quantity with $\text{Var}(\bar{Y}) = \sigma^2/n > 0$. The term $\frac{1}{n}$ in the square root comes from the uncertainty of the intercept/level of the line (estimated from only $n = 18$ days), and it does not depend on $x_0$. It is the minimum possible value of $\frac{1}{n} + \frac{(x_0 - \bar{x})^2}{S_{xx}}$. So the CI width is at least $2\times 2.1199 \times 1.674 /\sqrt{18} = 1.673$ Wh/km, and it only goes to zero as $n \to \infty$. Moreover, $\sigma$ is unknown and estimated by $\widehat{\sigma} = 1.674$, which also enters the width. In fact, at $10^\circ\text{C}$ the CI width ($1.6748$) is almost equal to this minimum ($1.673$).

3. Multiple Linear Regression (MLR)

Exercise 3.1 - Application: food-truck daily revenue

A food-truck owner recorded, for $n = 30$ days, the daily revenue (revenue, in units of $\$10$), the temperature (temp, from $8.0$ to $27.3\,^\circ\text{C}$), the foot traffic in front of the truck (traffic, in hundreds of passers-by per hour, from $0.5$ to $4.4$), the rainfall (rain, in mm, from $0$ to $14.8$), and the day of the month (day, from $3$ to $31$). They fitted the model

\[ \text{revenue}_i = \beta_0 + \beta_1\,\text{temp}_i + \beta_2\,\text{traffic}_i + \varepsilon_i, \qquad \varepsilon_i \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^2). \]

Some entries of the R output were smudged and are denoted [A], [B], [C], and [D]:

> fit2 <- lm(revenue ~ temp + traffic, data = truck)
> summary(fit2)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   1.6031     4.3274     [A]    0.714
temp          1.6512     0.2282   7.237 8.77e-08 ***
traffic       8.0241        [B]   5.466 8.74e-06 ***

Residual standard error: 6.905 on 27 degrees of freedom
Multiple R-squared:    [C], Adjusted R-squared:  0.8296
F-statistic:   [D] on 2 and 27 DF,  p-value: 1.601e-11

Other summary statistics:

$\bar{y} = 51.04$, $TSS = \sum_{i=1}^{30}(y_i - \bar{y})^2 = 8117.174$, and the sample correlation between temp and traffic is $0.43$;
the simple linear regression revenue ~ temp gives $\widehat{\beta}_0 = 10.6148$, $\widehat{\beta}_1 = 2.1905$ (standard error $0.2932$), and $R^2 = 0.6659$.

The estimated covariance matrix of $\widehat{\pmb{\beta}}$ (the output of vcov(fit2)) is

            (Intercept)      temp   traffic
(Intercept)   18.726657 -0.618545 -2.420703
temp          -0.618545  0.052060 -0.144840
traffic       -2.420703 -0.144840  2.155416

Because the printed numbers are rounded, your answers may differ from the solutions in the last digit. Critical values are given in the exercises where they are needed.

Exercise 3.1.1

Skip for now

Part [D] uses the F-statistic, which we have not covered yet. Skip [D] for now and come back to it after we cover the F-test (after the midterm). Parts [A], [B], and [C] can be done now.

Reconstruct the four missing numbers [A], [B], [C], and [D]. Show two different ways to obtain [C] (one from $RSS$ and $TSS$, and one from the adjusted $R^2$). The residual sum of squares of this model is $RSS = 1287.453$.

Hint

$R^2_{\text{adj}} = 1 - (1-R^2)\,\dfrac{n-1}{n-(p+1)}$.
$F = \dfrac{MSS/p}{RSS/(n-(p+1))}$, with $MSS = TSS - RSS$.

Solution

[A]: $t_{\text{Intercept}} = \frac{1.6031}{4.3274} = 0.370$.
[B]: $\widehat{\text{SE}}(\widehat{\beta}_2) = \frac{8.0241}{5.466} = 1.468$.
[C], way 1: $R^2 = 1 - \frac{RSS}{TSS} = 1 - \frac{1287.453}{8117.174} = 0.8414$.
[C], way 2: $1 - R^2 = (1 - R^2_{\text{adj}})\frac{n-(p+1)}{n-1} = 0.1704\times\frac{27}{29} = 0.1586$, so $R^2 \approx 0.841$ (the same as in way 1, up to the rounding of $R^2_{\text{adj}}$).
[D]: $MSS = 8117.174 - 1287.453 = 6829.721$, so $F = \frac{6829.721/2}{1287.453/27} = \frac{3414.86}{47.68} = 71.62$.

Exercise 3.1.2

Interpret each of the estimated coefficients $\widehat{\beta}_0$, $\widehat{\beta}_1$, and $\widehat{\beta}_2$ in the context of the problem, including the units. Is the intercept meaningful here?

Solution

$\widehat{\beta}_1 = 1.6512$: an increase of $1^\circ\text{C}$ in temperature is associated with an expected revenue that is $1.6512$ units higher (i.e., about $\$16.51$), keeping the foot traffic constant.
$\widehat{\beta}_2 = 8.0241$: an increase of $100$ passers-by per hour in foot traffic is associated with an expected revenue that is $8.0241$ units higher (i.e., about $\$80.24$), keeping the temperature constant.
$\widehat{\beta}_0 = 1.6031$: the expected revenue ($\$16.03$) for days with a temperature of $0\,^\circ\text{C}$ and no passers-by. It is not meaningful: the observed temperatures range from $8$ to $27.3\,^\circ\text{C}$ and the traffic from $0.5$ to $4.4$, so this is an extrapolation far outside the data.

Note the wording: we say “associated with” (not “causes”) because these are observational data, and “expected” (the mean revenue of days with given characteristics) because the coefficients say nothing certain about what will happen on one particular day.

Exercise 3.1.3

The owner reads the output and says: “to increase my revenue by $\$80$ per day, I just need to move the truck to a spot with $100$ more passers-by per hour”. Is this conclusion supported by the analysis? Explain.

Solution

Not really. The coefficient describes how the expected revenue is associated with foot traffic, keeping the temperature constant; it is an association measured in observational data. Foot traffic is not something the owner controlled. For example, busy days may be busy because of weekends or events (which are not in the model) that also increase sales for other reasons. Moving the truck to a different spot changes other things as well (competition, visibility, type of customers), and the model says nothing about those.

This does not mean that the owner is wrong: foot traffic may well have a real effect on revenue. The point is that, as is, this analysis cannot tell us whether the association is causal, so the conclusion is not guaranteed by the analysis.

Exercise 3.1.4

The coefficient of temp is $2.1905$ in the simple linear regression revenue ~ temp, but $1.6512$ in the model that also includes traffic.

Why do the two estimates differ? Say what association each of them describes.
Neither of the two estimates is “wrong”. Which one would you use to answer the question “by how much is an increase of $1\,^\circ\text{C}$ in temperature associated with a change in the expected revenue, if I don’t know anything about the traffic?”

Hint

The sample correlation between temp and traffic is $0.43$. When the temperature changes from one day to another, what tends to happen to the foot traffic?

Solution

The two coefficients answer different questions. In revenue ~ temp, $2.1905$ is the association between temperature and expected revenue without keeping anything else constant; since warmer days also tend to have more foot traffic ($r = 0.43$), and foot traffic is positively associated with revenue, the simple regression slope for temp also reflects part of the association of traffic with revenue. In the model with both covariates, $1.6512$ is the association with temperature keeping the foot traffic constant, so it only reflects the part of the association with revenue that is not already accounted for by differences in traffic. (For the curious: one can show that the difference is $2.1905 - 1.6512 = 0.5393 \approx \widehat{\beta}_2\times\delta$, where $\delta \approx 0.067$ is the slope of the regression of traffic on temp.)
The simple regression slope, $2.1905$: if we do not know the traffic, the association we want is the one between temperature and revenue including the traffic that goes with the temperature. The multiple regression coefficient is the right one if we want the association keeping the traffic constant.

Exercise 3.1.5

Assume normal errors.

Compute and interpret a 95% confidence interval for $\beta_1$.
Compute and interpret a 95% confidence interval for $\beta_2$.

Use $t^*_{27,\,0.975} = 2.0518$.

Hint

The confidence interval is $\widehat{\beta}_j \pm t^*_{n-(p+1),\,0.975}\,\widehat{\text{SE}}(\widehat{\beta}_j)$. Remember that the revenue is in units of $\$10$ and that the traffic is in hundreds of people per hour.

Solution

$1.6512 \pm 2.0518\times0.2282 = 1.6512 \pm 0.4682 = [1.183,\; 2.119]$. We are 95% confident that an increase of $1\,^\circ\text{C}$ in temperature is associated with an expected daily revenue that is between $1.18$ and $2.12$ units higher (i.e., between $\$11.83$ and $\$21.19$ higher), keeping the foot traffic constant. (The confidence level refers to the procedure: if we repeated the study many times, about 95% of the intervals computed in this way would contain the true $\beta_1$.)
$8.0241 \pm 2.0518\times1.468 = 8.0241 \pm 3.0120 = [5.012,\; 11.036]$. We are 95% confident that an increase of $100$ passers-by per hour in foot traffic is associated with an expected daily revenue that is between $\$50.12$ and $\$110.36$ higher, keeping the temperature constant.

Exercise 3.1.6

Assume normal errors. Test $H_0:\beta_1 = 0$ against $H_A:\beta_1\neq 0$ at the 5% level. Write the test statistic and its distribution under $H_0$, and state your conclusion in the context of the problem. Use $t^*_{27,\,0.975} = 2.0518$.

Hint

$T = \dfrac{\widehat{\beta}_j - \beta_{j,0}}{\widehat{\text{SE}}(\widehat{\beta}_j)} \sim t_{n-(p+1)}$ under $H_0:\beta_j=\beta_{j,0}$.

Solution

$H_0:\beta_1=0$ vs $H_A:\beta_1\neq0$. Under $H_0$ (and normal errors), \[ T = \frac{\widehat{\beta}_1 - 0}{\widehat{\text{SE}}(\widehat{\beta}_1)} = \frac{1.6512}{0.2282} = 7.236 \sim t_{27} \quad (7.237 \text{ in the output, which uses unrounded values}). \] Since $|T_{\text{obs}}| \approx 7.24 > 2.0518$ (equivalently, $p$-value $= 8.77\times10^{-8} < 0.05$), we reject $H_0$: there is strong evidence that, after accounting for foot traffic, the temperature is associated with daily revenue.

Exercise 3.1.7

Assume normal errors. A friend of the owner claims that “an increase of $100$ passers-by per hour is associated with an expected revenue that is $\$50$ higher, keeping the temperature constant”.

Test this claim at the 5% level, i.e., $H_0:\beta_2 = 5$ against $H_A:\beta_2\neq 5$.
Check that the confidence interval from Exercise 3.1.5(b) leads to the same decision.
Would the decision change at the 1% level?

Use $t^*_{27,\,0.975} = 2.0518$.

Hint

Remember that the revenue is in units of $\$10$, so $$50 corresponds to $5$ units. A two-sided test at level $\alpha$ is related to the $100(1-\alpha)\%$ confidence interval.

Solution

Part A

$T = \dfrac{8.0241 - 5}{1.468} = 2.060 > 2.0518$, so we reject $H_0:\beta_2=5$ at the 5% level (the $p$-value is $0.049$). There is enough evidence, at the 5% level, that the true slope for $100$ more passers-by per hour is not exactly $\$50$; the estimate suggests it is larger.

The $p$-value is the probability, under $H_0$, of a $t_{27}$ statistic at least as extreme as $|T_{\text{obs}}|$ (in both tails). In R:

t_obs <- (8.0241 - 5) / 1.468
2 * pt(-abs(t_obs), df = 27)

[1] 0.0491556

Part B

The interval $[5.012,\; 11.036]$ does not contain $5$, which leads to the same decision (a two-sided test at level $\alpha$ rejects $H_0:\beta_2=\beta_{2,0}$ if and only if the $100(1-\alpha)\%$ confidence interval does not contain $\beta_{2,0}$).

Part C

The $p$-value does not depend on the significance level, so there is no need to recompute the test statistic: we only compare the $p$-value from (a) with the new level. Since $0.049 > 0.01$, we would not reject $H_0$ at the 1% level: there is not enough evidence against the claim at that level. Note also that $\$50$ is only barely outside the 95% interval, so for practical purposes the friend’s claim is not far from what the data suggest.

Exercise 3.1.8

Skip for now

This exercise uses the F-test, which we have not covered yet. Come back to it after we cover the F-test (after the midterm).

Test the null hypothesis $H_0:\beta_1 = \beta_2 = 0$ at the 5% level using the F-statistic in the output. State the alternative hypothesis, and explain what the conclusion tells us (and what it does not tell us). Use $F^*_{2,\,27;\,0.95} = 3.3541$.

Solution

$H_0:\beta_1=\beta_2=0$ vs $H_A:$ at least one of $\beta_1,\beta_2$ is not zero. $F = 71.62 > 3.3541$ ($p$-value $1.6\times10^{-11}$), so we reject $H_0$. There is strong evidence that temperature and/or foot traffic are linearly associated with daily revenue. This does not tell us which of the two covariates is associated with revenue (we use the $t$-tests for that), nor that the model is adequate or that its predictions will be accurate; it only says that the model with temp and traffic explains more than a model with just an intercept would.

Exercise 3.1.9

The owner is planning next Saturday, when the forecast is $20\,^\circ\text{C}$ and a foot traffic of $2.5$ (i.e., $250$ passers-by per hour) is expected. Compute a 95% confidence interval for the expected revenue of days with these conditions, and interpret it. Use $t^*_{27,\,0.975} = 2.0518$.

Hint

With $\mathbf{x}_0 = (1, 20, 2.5)^\top$, $\widehat{y}_0 = \mathbf{x}_0^\top\widehat{\pmb{\beta}}$ and $\widehat{\text{Var}}(\widehat{y}_0) = \mathbf{x}_0^\top\,\widehat{\text{Cov}}(\widehat{\pmb{\beta}})\,\mathbf{x}_0$ (use the matrix given above). The interval is $\widehat{y}_0 \pm t^*\sqrt{\widehat{\text{Var}}(\widehat{y}_0)}$.

Solution

We have that : $\widehat{y}_0 = 1.6031 + 1.6512(20) + 8.0241(2.5) = 54.69$ units (i.e., $\$546.9$). Also, $\mathbf{x}_0^\top\widehat{\text{Cov}}(\widehat{\pmb{\beta}})\mathbf{x}_0 \approx 1.693$ so $\widehat{\text{SE}}(\widehat{y}_0) = 1.301$ and \[ 54.69 \pm 2.0518\times1.301 = 54.69 \pm 2.67 = [52.02,\; 57.36]. \] We are 95% confident that the average daily revenue, over all days with a temperature of $20\,^\circ\text{C}$ and a foot traffic of $2.5$, is between $\$520.2$ and $\$573.6$.

Exercise 3.1.10

For the same conditions as in Exercise 3.1.9 ($20\,^\circ\text{C}$ and a foot traffic of $2.5$), compute a 95% prediction interval for the revenue of next Saturday, and interpret it. Use $t^*_{27,\,0.975} = 2.0518$.

Hint

The interval is $\widehat{y}_0 \pm t^*\sqrt{\widehat{\sigma}^2 + \widehat{\text{Var}}(\widehat{y}_0)}$, where $\widehat{\sigma}^2 = 6.905^2 = 47.68$ and $\widehat{\text{Var}}(\widehat{y}_0)$ is the value computed in Exercise 3.1.9.

Solution

$\widehat{\text{SE}}(Y_{\text{new}} - \widehat{y}_0) = \sqrt{47.68 + 1.693} = 7.027$, so \[ 54.69 \pm 2.0518\times7.027 = 54.69 \pm 14.42 = [40.27,\; 69.11]. \] There is an approximately 95% probability that the revenue of next Saturday will be between $\$402.7$ and $\$691.1$, assuming that next Saturday indeed has a temperature of $20\,^\circ\text{C}$ and a foot traffic of $2.5$ (the interval does not include the uncertainty of the forecasts).

Note: Unlike a confidence interval, which is about a fixed parameter, a prediction interval is about a random* future value, so a probability statement makes sense. It is only approximate because the interval is computed from estimated quantities ($\widehat{\pmb{\beta}}$ and $\widehat{\sigma}$), and it relies on the model assumptions (linearity, constant variance, normal errors).

Exercise 3.1.11

Why is the prediction interval of Exercise 3.1.10 so much wider than the confidence interval of Exercise 3.1.9? Which interval should the owner use to decide how much cash to prepare for next Saturday?

Solution

The confidence interval only accounts for the uncertainty in estimating the mean revenue (the estimated plane). The prediction interval has, in addition, the variability of a single day around its mean, $\sigma^2$, which is much larger here ($47.68$ vs $1.69$) and does not shrink as we collect more data. To decide how much cash to prepare for next Saturday, the owner needs the prediction interval, because a single day is what matters.

Exercise 3.1.12

The owner also wants a prediction for a day at $3\,^\circ\text{C}$ with a foot traffic of $3.0$. Should we trust the interval given by the same formula as in Exercise 3.1.10? Explain.

Solution

No. The observed temperatures range from $8.0$ to $27.3\,^\circ\text{C}$, so $3\,^\circ\text{C}$ is an extrapolation. The formulas assume that the linear relationship between temperature and revenue continues to hold outside the range of the data, which we cannot check with these data (in cold weather revenue might drop much faster, or the truck may not operate at all). The interval computed with the formula would only be valid if the model were correct there: it does not account for the possibility that the model fails outside the observed range.

Exercise 3.1.13

Interpret $R^2 = 0.8414$, the adjusted $R^2 = 0.8296$, and the residual standard error $\widehat{\sigma} = 6.905$ in the context of the problem.

Hint

$R^2$ is a proportion of the total sum of squares, and $\widehat{\sigma} = \sqrt{RSS/(n-(p+1))}$ is measured in the units of the response (here, $\$10$).

Solution

$R^2 = 0.8414$: $84.1\%$ of the variability (sum of squares) of daily revenue in these $30$ days is accounted for by the linear relationship with temperature and foot traffic; the remaining $15.9\%$ is not explained by the model.
Adjusted $R^2 = 0.8296$: a version of $R^2$ that penalizes the number of covariates (it replaces $RSS/TSS$ by $\frac{RSS/(n-(p+1))}{TSS/(n-1)}$). It is not literally a proportion of explained variability, but it is more suitable to compare models of different sizes.
$\widehat{\sigma} = 6.905$: the typical distance between the observed revenue of a day and the revenue given by the fitted plane is about $6.9$ units, i.e., about $\$69$.

Exercise 3.1.14

Decide whether each statement is correct, and explain briefly.

“The model predicts the revenue of $84\%$ of the days correctly.”
“Temperature and foot traffic explain $84\%$ of the variability in the daily revenues of these 30 days.”
“Since $R^2$ is high, the linear model is appropriate for these data.”
“Temperature and foot traffic cause $84\%$ of the revenue.”
“Adding traffic to the model with only temp ($R^2 = 0.6659$) increases the proportion of variability explained by about $17.6$ percentage points.”

Hint

Think about what each statement would mean for a continuous response, and which of them confuse association with causation or $R^2$ with model adequacy.

Solution

Incorrect. $R^2$ is a proportion of variability, not a percentage of correct predictions: it does not measure how many predictions are exactly right (or within some tolerance).
Correct (as a description of this sample: the linear relationship with temp and traffic accounts for $84\%$ of the total sum of squares of the revenues).
Incorrect. A high $R^2$ does not say that the linear model is appropriate: a straight-line or plane can have a high $R^2$ with a clearly curved relationship, with non-constant variance, etc. We need the diagnostic plots (Module 06) for that.
Incorrect. $R^2$ measures association in the observed data, not causation.
Correct. $0.8414 - 0.6659 = 0.1755$: traffic accounts for an additional $17.6\%$ of the total variability of revenue beyond what temp already explained. (It is not the case that traffic alone explains $17.6\%$ of the variability.)

Exercise 3.1.15

The owner wonders whether the day of the month affects revenue (for example, because of paydays), and adds day to the model:

> fit4 <- lm(revenue ~ temp + traffic + day, data = truck)
> summary(fit4)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
...
day          -0.0829     0.1558     [E]    0.599

Residual standard error: 6.999 on 26 degrees of freedom
Multiple R-squared:  0.8431,    Adjusted R-squared:  0.8250

The residual sum of squares of this model is $RSS = 1273.602$. The owner concludes: “$R^2$ increased from $0.8414$ to $0.8431$, so the model improved and day should stay in the model.”

Is this reasoning correct? Explain, in particular, why $R^2$ increased (even if day were a completely useless variable).

Hint

Recall from Module 08 what happens to the minimum of the $RSS$ when a covariate is added to the model, and what $R^2 = 1 - RSS/TSS$ implies.

Solution

No. Adding a covariate can never increase the minimum of the $RSS$ (the model with day includes the one without it as a special case, $\beta_3 = 0$), so $R^2 = 1 - RSS/TSS$ can never decrease when we add a variable, even if the variable is pure noise. Such a small increase ($0.0017$) can occur even for a completely useless variable, so $R^2$ cannot be used by itself to decide whether to keep a covariate.

Exercise 3.1.16

In the output of Exercise 3.1.15, compute [E]. Which other quantities in the output indicate that day is not worth keeping?

Hint

Compare $R^2_{\text{adj}}$ and $\widehat{\sigma}$ of the model with day with those of the model without it ($R^2_{\text{adj}} = 0.8296$ and $\widehat{\sigma} = 6.905$).

Solution

[E] $= \frac{-0.0829}{0.1558} = -0.532$, with a $p$-value of $0.599$: there is insufficient evidence of a linear association between day and revenue after accounting for temp and traffic. Moreover, the adjusted $R^2$ decreased ($0.8296 \to 0.8250$) and the residual standard error increased ($6.905 \to 6.999$): the reduction in $RSS$ is not enough to compensate for the loss of one degree of freedom.

Exercise 3.1.17

Skip for now

This exercise uses the F-test, which we have not covered yet. Come back to it after we cover the F-test (after the midterm).

Use the residual sums of squares to carry out an F-test of $H_0:\beta_3 = 0$ (the coefficient of day) at the 5% level, and compare it with the $t$-test in the output of Exercise 3.1.15. Use $F^*_{1,\,26;\,0.95} = 4.2252$.

Hint

$F = \dfrac{(RSS_{\text{reduced}} - RSS_{\text{full}})/q}{RSS_{\text{full}}/(n-(p+1))}$. The $RSS$ of the model without day is $1287.453$.

Solution

$F = \dfrac{(1287.453 - 1273.602)/1}{1273.602/26} = \dfrac{13.851}{48.985} = 0.283 < 4.2252$, so we do not reject $H_0$. Indeed, $F = t^2 = (-0.532)^2 = 0.283$, and the $p$-value ($0.599$) is the same as for the two-sided $t$-test.

Exercise 3.1.18

Can we conclude that the day of the month has no effect on revenue?

Solution

No. Failing to reject $H_0:\beta_3=0$ means that, with $n=30$ days, we do not detect an association between day and revenue after accounting for temperature and traffic. Absence of evidence is not evidence of absence: the data are simply not informative enough to tell a small effect from no effect. Also, the model is linear in day; a different pattern (for example, only the first days of the month) would not be captured by a single slope.

Exercise 3.1.19

The owner also recorded the rainfall. The table summarizes four models fitted to the same $30$ days. For all of them, $TSS = 8117.174$. Complete the table.

Model	Covariates	$RSS$	Residual df
M1	`temp`	2711.859	28
M2	`temp`, `traffic`	1287.453	27
M3	`temp`, `traffic`, `rain`	1043.451	26
M4	`temp`, `traffic`, `day`	1273.602	26

Hint

$R^2 = 1 - \dfrac{RSS}{TSS}$, $R^2_{\text{adj}} = 1 - \dfrac{RSS/(n-(p+1))}{TSS/(n-1)}$, and $\widehat{\sigma}^2 = \dfrac{RSS}{n-(p+1)}$.

Solution

Model	Covariates	$RSS$	Residual df	$R^2$	$R^2_{\text{adj}}$	$\widehat{\sigma}$
M1	`temp`	2711.859	28	0.6659	0.6540	9.841
M2	`temp`, `traffic`	1287.453	27	0.8414	0.8296	6.905
M3	`temp`, `traffic`, `rain`	1043.451	26	0.8715	0.8566	6.335
M4	`temp`, `traffic`, `day`	1273.602	26	0.8431	0.8250	6.999

For example, for M3: $R^2 = 1 - \frac{1043.451}{8117.174} = 0.8715$, $R^2_{\text{adj}} = 1 - \frac{1043.451/26}{8117.174/29} = 1 - \frac{40.13}{279.90} = 0.8566$, and $\widehat{\sigma} = \sqrt{40.13} = 6.335$.

Exercise 3.1.20

Skip for now

This exercise uses the F-test, which we have not covered yet. Come back to it after we cover the F-test (after the midterm).

Using the models M1–M4 of Exercise 3.1.19, which pairs of models can be compared with an F-test? Which cannot?

Solution

The F-test requires nested models fitted to the same data: the reduced model must be a special case of the full model. M1 is nested in M2, M3 and M4; M2 is nested in M3 and in M4. M3 and M4 cannot be compared with an F-test: neither contains the other (M3 has rain but not day, and M4 has day but not rain).

Exercise 3.1.21

Skip for now

This exercise uses the F-test, which we have not covered yet. Come back to it after we cover the F-test (after the midterm).

In M3, let $\beta_3$ be the coefficient of rain. Test, at the 5% level, whether rain is needed in the model that already has temp and traffic (M2 vs M3), i.e., $H_0:\beta_3=0$, using $F^*_{1,\,26;\,0.95} = 4.2252$. Check that the result agrees with the $t$-test for rain in M3 ($\widehat{\beta}_3 = -0.7186$, $\widehat{\text{SE}} = 0.2914$), and interpret the estimated coefficient of rain.

Hint

$F = \dfrac{(RSS_{\text{reduced}} - RSS_{\text{full}})/q}{RSS_{\text{full}}/(n-(p+1))}$, where $n-(p+1)$ is the residual degrees of freedom of the full model.

Solution

$F = \dfrac{(1287.453 - 1043.451)/1}{1043.451/26} = \dfrac{244.002}{40.133} = 6.08 > 4.2252$, so we reject $H_0:\beta_3=0$ at the 5% level ($p$-value $0.021$). This agrees with the $t$-test: $T = \frac{-0.7186}{0.2914} = -2.466$ and $T^2 = 6.08 = F$. There is evidence that rain is associated with revenue after accounting for temperature and traffic. The coefficient $-0.7186$ means that an increase of $1$ mm of rain is associated with an expected daily revenue that is $0.72$ units lower (i.e., about $\$7.19$ lower), keeping the temperature and the foot traffic constant.

Exercise 3.1.22

Skip for now

This exercise uses the F-test, which we have not covered yet. Come back to it after we cover the F-test (after the midterm).

In M3, let $\beta_2$ and $\beta_3$ be the coefficients of traffic and rain, respectively. Test, at the 5% level, whether at least one of traffic and rain is needed in the model that has temp (M1 vs M3), i.e., $H_0:\beta_2=\beta_3=0$. Use $F^*_{2,\,26;\,0.95} = 3.3690$.

Hint

Now $q=2$: the reduced model M1 is obtained from M3 by setting two coefficients to zero.

Solution

$F = \dfrac{(2711.859 - 1043.451)/2}{1043.451/26} = \dfrac{834.204}{40.133} = 20.79 > 3.3690$ (the $p$-value is about $4\times10^{-6}$), so we reject $H_0:\beta_2=\beta_3=0$: at least one of traffic and rain is associated with revenue given temp.

1. Formulae Reminder

Exercise 1: Some Useful Identities

2. Simple Linear Regression

Exercise 2.1 - Least Squares Estimation

Exercise 2.1.1

Exercise 2.1.2

Exercise 2.1.3

Exercise 2.1.4

Exercise 2.1.5

Exercise 2.1.6

Exercise 2.2 - Distribution of \(Y_i\)

Exercise 2.2.1

Exercise 2.2.2

Exercise 2.3 - LSE: Infinite Solutions

Exercise 2.3.1

Exercise 2.3.2

Exercise 2.4 - LSE: Dummy Variables

Exercise 2.5 - LSE: Properties

Exercise 2.5.1

Exercise 2.5.2

Exercise 2.5.3

Exercise 2.5.4

Exercise 2.5.5

Exercise 2.5.6

Exercise 2.5.7

Exercise 2.5.8

Exercise 2.5.9

Exercise 2.5.10

Exercise 2.5.11

Exercise 2.5.12

Exercise 2.5.13

Exercise 2.5.14

Exercise 2.5.15

Exercise 2.5.16

Exercise 2.5.17

Exercise 2.5.18

Exercise 2.5.19

Exercise 2.5.20 (Challenging!)

Exercise 2.6 - Expected Response & Prediction

Exercise 2.6.1

Exercise 2.6.2

Exercise 2.6.3

Exercise 2.7 - Properties of the residuals

Exercise 2.7.1

Exercise 2.7.2

Exercise 2.7.3

Exercise 2.8 - Application: cargo e-bike energy consumption

Exercise 2.8.1

Exercise 2.8.2

Exercise 2.8.3

Exercise 2.8.4

Exercise 2.8.5

3. Multiple Linear Regression (MLR)

Exercise 3.1 - Application: food-truck daily revenue

Exercise 3.1.1

Exercise 3.1.2

Exercise 3.1.3

Exercise 3.1.4

Exercise 3.1.5

Exercise 3.1.6

Exercise 3.1.7

Part A

Part B

Part C

Exercise 3.1.8

Exercise 3.1.9

Exercise 3.1.10

Exercise 3.1.11

Exercise 3.1.12

Exercise 3.1.13

Exercise 3.1.14

Exercise 3.1.15

Exercise 3.1.16

Exercise 3.1.17

Exercise 3.1.18

Exercise 3.1.19

Exercise 3.1.20

Exercise 3.1.21

Exercise 3.1.22