Practice Exercises

1. Formulae Reminder

Recall the following formulae from introductory statistics:

  1. Sample Mean: \(\bar{X} = \frac{1}{n} \displaystyle\sum_{i=1}^{n} X_i\)

  2. Sample Variance: \(\displaystyle S^2_X = \frac{1}{n-1} \sum_{i=1}^{n} (X_i - \bar{X})^2\)

  3. Sample Standard Deviation: \(\displaystyle S_X = \sqrt{S^2_X}\)

  4. Sample Covariance: \(\displaystyle \text{cov}(X, Y) = S_{XY} = \frac{1}{n-1} \sum_{i=1}^{n} (X_i - \bar{X})(Y_i - \bar{Y})\)

  5. Sample Correlation: \(\displaystyle \text{cor}(X, Y) = r_{XY} = \frac{S_{XY}}{S_X S_Y}\)


Exercise 1: Some Useful Identities

Prove the following equations:

(a) Prove that \((n-1)S^2_X = \displaystyle\sum_{i=1}^{n} X_i^2 - n\bar{X}^2\).

Expand the squared term \((X_i - \bar{X})^2 = X_i^2 - 2X_i\bar{X} + \bar{X}^2\). Then distribute the summation, noting that \(\bar{X}\) is a constant with respect to the summation index \(i\), so \(\sum_{i=1}^n \bar{X}^2 = n\bar{X}^2\) and \(\sum_{i=1}^n X_i = n\bar{X}\).

We start with the definition of sample variance:

\[ \begin{aligned} (n-1)S_X^2 &= \sum_{i=1}^{n} (X_i - \bar{X})^2 \\ &= \sum_{i=1}^{n} (X_i^2 - 2X_i\bar{X} + \bar{X}^2) \\ &= \sum_{i=1}^{n} X_i^2 - \sum_{i=1}^{n} 2X_i\bar{X} + \sum_{i=1}^{n} \bar{X}^2 \\ &= \sum_{i=1}^{n} X_i^2 - 2\bar{X}\sum_{i=1}^{n} X_i + n\bar{X}^2 \end{aligned} \]

We know that \(\bar{X} = \frac{\sum_{i=1}^{n} X_i}{n} \implies \sum_{i=1}^{n} X_i = n\bar{X}\). Substituting this into the equation:

\[ \begin{aligned} (n-1)S_X^2 &= \sum_{i=1}^{n} X_i^2 - 2\bar{X}\sum_{i=1}^{n} X_i + n\bar{X}^2 \\ &= \sum_{i=1}^{n} X_i^2 - 2\bar{X}(n\bar{X}) + n\bar{X}^2 \\ &= \sum_{i=1}^{n} X_i^2 - 2n\bar{X}^2 + n\bar{X}^2 \\ &= \sum_{i=1}^{n} X_i^2 - n\bar{X}^2 \quad \blacksquare \end{aligned} \]


(b) Prove that \(S_{XY} = \frac{1}{n-1} \left( \displaystyle\sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y} \right)\), or equivalently, \((n-1)S_{XY} = \displaystyle\sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y}\).

Expand the product \((X_i - \bar{X})(Y_i - \bar{Y}) = X_i Y_i - X_i\bar{Y} - \bar{X}Y_i + \bar{X}\bar{Y}\). Distribute the summation over each term, factor out constants \(\bar{X}\) and \(\bar{Y}\), and substitute \(\sum_{i=1}^n X_i = n\bar{X}\) and \(\sum_{i=1}^n Y_i = n\bar{Y}\).

We start with the definition of sample covariance:

\[ \begin{aligned} S_{XY} &= \frac{1}{n-1}\sum_{i=1}^{n} (X_i - \bar{X})(Y_i - \bar{Y}) \\ &= \frac{1}{n-1}\sum_{i=1}^{n} (X_i Y_i - X_i\bar{Y} - \bar{X}Y_i + \bar{X}\bar{Y}) \\ &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - \sum_{i=1}^{n} X_i\bar{Y} - \sum_{i=1}^{n} \bar{X}Y_i + \sum_{i=1}^{n} \bar{X}\bar{Y}\right) \\ &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - \bar{Y}\sum_{i=1}^{n} X_i - \bar{X}\sum_{i=1}^{n} Y_i + n\bar{X}\bar{Y}\right) \end{aligned} \]

Substituting \(\sum_{i=1}^{n} X_i = n\bar{X}\) and \(\sum_{i=1}^{n} Y_i = n\bar{Y}\):

\[ \begin{aligned} S_{XY} &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - \bar{Y}(n\bar{X}) - \bar{X}(n\bar{Y}) + n\bar{X}\bar{Y}\right) \\ &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y} - n\bar{X}\bar{Y} + n\bar{X}\bar{Y}\right) \\ &= \frac{1}{n-1}\left(\sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y}\right) \end{aligned} \]

Multiplying both sides by \(n - 1\) gives the unscaled form:

\[(n-1)S_{XY} = \sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y} \quad \blacksquare\]


2. Simple Linear Regression

Exercise 2 - Least Squares Estimation

In this exercise you will find the line that best fits a data set. Suppose we have \(n\) points in \(\mathbb{R}^2\) given by \((x_1, Y_1), (x_2, Y_2), \ldots, (x_n, Y_n)\).

Any straight line can be defined by the equation \(Y = b_0 + b_1x\), where \(b_0\) is the \(y\)-intercept and \(b_1\) is the slope. The challenge is to find the optimal line, i.e., the optimal values of \(b_0\) and \(b_1\).

To do this, we first need a way to measure how “wrong” a particular line is. For any single data point \((x_i, Y_i)\), our line predicts the value \(\hat{Y}_i = b_0 + b_1 x_i\). The error for that point, called residual, is the difference between the actual observed value \(Y_i\) and the predicted value \(\hat{Y}_i\), i.e., the residual of a point \((x_i, Y_i)\) is given by \(e_i = Y_i - \hat{Y}_i = Y_i - \left(b_0 + b_1x_i\right)\). The residual is the vertical difference between the point and the line.

We want to make the total error as small as possible. If we just added up all the residuals, the positive errors (points above the line) and negative errors (points below the line) would cancel each other out, which isn’t helpful.

To solve this, we square each residual before adding them up. This has two great benefits:

  • All the error terms become positive.
  • It penalizes larger errors more heavily (well, this can also be a drawback – more on this later).

This gives us the Residual Sum of Squares (RSS): \[RSS(b_0, b_1) = \sum_{i=1}^{n} (Y_i - (b_0 + b_1 x_i))^2\]

The “best” line is the one that makes this total squared error as small as possible. This is the method of least squares.

In this case, what values of \(b_0\) and \(b_1\) minimize RSS? Use the sliders below to choose these values.

Naturally, we don’t want to do this manually. Instead, we can use calculus to find the optimal values of \(b_0\) and \(b_1\). In this exercise we will find the values of \(b_0\) and \(b_1\) that minimize \(RSS\).


Exercise 2.1

Find the partial derivatives of \(RSS\) with respect to \(b_0\) and \(b_1\).

Use the chain rule.

The \(RSS\) is given by:

\[RSS(b_0, b_1) = \sum_{i=1}^{n} (Y_i - b_0 - b_1 x_i)^2\]

The partial derivative with respect to \(b_0\) is:

\[ \begin{aligned} \frac{\partial RSS}{\partial b_0} &= \sum_{i=1}^{n} 2(Y_i - b_0 - b_1 x_i)(-1) \\ &= -2\sum_{i=1}^{n} (Y_i - b_0 - b_1 x_i) \end{aligned} \]

The partial derivative with respect to \(b_1\) is:

\[ \begin{aligned} \frac{\partial RSS}{\partial b_1} &= \sum_{i=1}^{n} 2(Y_i - b_0 - b_1 x_i)(-x_i) \\ &= -2\sum_{i=1}^{n} x_i(Y_i - b_0 - b_1 x_i) \quad \blacksquare \end{aligned} \]


Exercise 2.2

Set the partial derivatives equal to zero and solve \(\frac{\partial RSS}{\partial b_0} = 0\) for \(\hat{\beta}_0\) in terms of \(\hat{\beta}_1\).

When you set the partial derivatives equal to zero, the solutions are the estimators \(\hat{\beta}_0\) and \(\hat{\beta}_1\), so replace \(b_0\) with \(\hat{\beta}_0\) and \(b_1\) with \(\hat{\beta}_1\). Then divide both sides by \(-2\) and split the sum into three separate summations: \(\sum Y_i\), \(\sum \hat{\beta}_0\), and \(\sum \hat{\beta}_1 x_i\).

Set \(\frac{\partial RSS}{\partial b_0} = 0\):

\[ \begin{aligned} -2\sum_{i=1}^{n} (Y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) &= 0 \\ \sum_{i=1}^{n} Y_i - \sum_{i=1}^{n} \hat{\beta}_0 - \sum_{i=1}^{n} \hat{\beta}_1 x_i &= 0 \\ n\bar{Y} - n\hat{\beta}_0 - \hat{\beta}_1(n\bar{x}) &= 0 \\ \bar{Y} - \hat{\beta}_0 - \hat{\beta}_1\bar{x} &= 0 \end{aligned} \]

Solving for \(\hat{\beta}_0\):

\[\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1\bar{x} \quad \blacksquare\]

Rearranging this result gives:

\[\bar{Y} = \hat{\beta}_0 + \hat{\beta}_1\bar{x}\]

This establishes a fundamental geometric property of ordinary least squares regression: the fitted regression line always passes through the point of averages (the centroid), \((\bar{x}, \bar{Y})\).


Exercise 2.3

For the equation \(\frac{\partial RSS}{\partial b_1} = 0\), replace the value of \(\hat{\beta}_0\) from Exercise 2.2 into the equation and solve for \(\hat{\beta}_1\).

Divide by \(-2\) and expand the summand: \(x_i(Y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) = x_i Y_i - x_i\hat{\beta}_0 - \hat{\beta}_1 x_i^2\). Replace \(\hat{\beta}_0\) with \((\bar{Y} - \hat{\beta}_1\bar{x})\), collect all terms involving \(\hat{\beta}_1\) on one side of the equation, factor out \(\hat{\beta}_1\), and divide.

Set \(\frac{\partial RSS}{\partial b_1} = 0\):

\[ \begin{aligned} -2\sum_{i=1}^{n} x_i(Y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i) &= 0 \\ \sum_{i=1}^{n} (x_i Y_i - x_i\hat{\beta}_0 - \hat{\beta}_1 x_i^2) &= 0 \\ \sum_{i=1}^{n} x_i Y_i - \hat{\beta}_0\sum_{i=1}^{n} x_i - \hat{\beta}_1\sum_{i=1}^{n} x_i^2 &= 0 \end{aligned} \]

Substitute \(\hat{\beta}_0 = \bar{Y} - \hat{\beta}_1\bar{x}\):

\[ \begin{aligned} \sum_{i=1}^{n} x_i Y_i - (\bar{Y} - \hat{\beta}_1\bar{x})\sum_{i=1}^{n} x_i - \hat{\beta}_1\sum_{i=1}^{n} x_i^2 &= 0 \\ \sum_{i=1}^{n} x_i Y_i - \bar{Y}(n\bar{x}) + \hat{\beta}_1\bar{x}(n\bar{x}) - \hat{\beta}_1\sum_{i=1}^{n} x_i^2 &= 0 \\ \sum_{i=1}^{n} x_i Y_i - n\bar{x}\bar{Y} &= \hat{\beta}_1\left(\sum_{i=1}^{n} x_i^2 - n\bar{x}^2\right) \end{aligned} \]

Solving for \(\hat{\beta}_1\):

\[\hat{\beta}_1 = \frac{\sum_{i=1}^{n} x_i Y_i - n\bar{x}\bar{Y}}{\sum_{i=1}^{n} x_i^2 - n\bar{x}^2} \quad \blacksquare\]

(Verification of Global Minimum): Setting the first derivatives to zero finds the stationary point \((\hat{\beta}_0, \hat{\beta}_1)\). To verify that this point minimizes \(RSS\), consider the Hessian matrix of second-order partial derivatives:

\[ H = \begin{pmatrix} \frac{\partial^2 RSS}{\partial b_0^2} & \frac{\partial^2 RSS}{\partial b_0 \partial b_1} \\ \frac{\partial^2 RSS}{\partial b_1 \partial b_0} & \frac{\partial^2 RSS}{\partial b_1^2} \end{pmatrix} = \begin{pmatrix} 2n & 2\sum_{i=1}^{n} x_i \\ 2\sum_{i=1}^{n} x_i & 2\sum_{i=1}^{n} x_i^2 \end{pmatrix} \]

The first leading principal minor is \(2n > 0\). The determinant of \(H\) is:

\[ \det(H) = (2n)\left(2\sum_{i=1}^{n} x_i^2\right) - \left(2\sum_{i=1}^{n} x_i\right)^2 = 4n\sum_{i=1}^{n} x_i^2 - 4(n\bar{x})^2 = 4n\left(\sum_{i=1}^{n} x_i^2 - n\bar{x}^2\right) = 4n(n-1)S_x^2 \]

Whenever not all \(x_i\) are identical (\(S_x^2 > 0\)), \(\det(H) > 0\). Because both the leading principal minor and the determinant are strictly positive, \(H\) is strictly positive definite everywhere. Thus, \(RSS(b_0, b_1)\) is strictly convex, and \((\hat{\beta}_0, \hat{\beta}_1)\) is the unique global minimum.


Exercise 2.4

Using the equalities from Exercise 1, show that you can write \(\hat{\beta}_1\) as \(\hat{\beta}_1 = r_{XY} \frac{S_Y}{S_X}\).

Notice that from Exercise 1, the numerator of \(\hat{\beta}_1\) is \((n-1)S_{XY}\) and the denominator is \((n-1)S_X^2\). Recall the definition of the correlation coefficient \(r_{XY} = \frac{S_{XY}}{S_X S_Y}\), so \(S_{XY} = r_{XY} S_X S_Y\).

From Exercise 1, we have:

  • \((n-1)S_X^2 = \sum_{i=1}^{n} X_i^2 - n\bar{X}^2\)
  • \((n-1)S_{XY} = \sum_{i=1}^{n} X_i Y_i - n\bar{X}\bar{Y}\)

Substituting these into the expression for \(\hat{\beta}_1\):

\[ \hat{\beta}_1 = \frac{(n-1)S_{XY}}{(n-1)S_X^2} = \frac{S_{XY}}{S_X^2} \]

The correlation coefficient is defined as \(r_{XY} = \frac{S_{XY}}{S_X S_Y}\). Therefore, \(S_{XY} = r_{XY} S_X S_Y\). Substituting this into the expression for \(\hat{\beta}_1\):

\[ \hat{\beta}_1 = \frac{r_{XY} S_X S_Y}{S_X^2} = r_{XY}\frac{S_Y}{S_X} \quad \blacksquare \]

Here are some different (but equivalent) ways to write \(\hat{\beta}_1\):

\[ \hat{\beta}_1 = \frac{\sum_{i=1}^{n} x_i Y_i - n\bar{x}\bar{Y}}{\sum_{i=1}^{n} x_i^2 - n\bar{x}^2} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{i=1}^{n} (x_i - \bar{x})^2} = \frac{\sum_{i=1}^{n} (x_i - \bar{x})Y_i}{\sum_{i=1}^{n} (x_i - \bar{x})^2} = r_{XY}\frac{S_Y}{S_X} \]

Why can we drop \(\bar{Y}\) in the numerator? Notice that:

\[\sum_{i=1}^{n} (x_i - \bar{x})(Y_i - \bar{Y}) = \sum_{i=1}^{n} (x_i - \bar{x})Y_i - \bar{Y}\sum_{i=1}^{n} (x_i - \bar{x})\]

Since \(\sum_{i=1}^{n} (x_i - \bar{x}) = \sum x_i - n\bar{x} = 0\), the second term drops out entirely.

Writing \(\hat{\beta}_1\) as \(\frac{\sum_{i=1}^{n}(x_i - \bar{x})Y_i}{\sum_{i=1}^{n}(x_i - \bar{x})^2}\) shows that \(\hat{\beta}_1\) is a linear combination of the response observations, \(\hat{\beta}_1 = \sum_{i=1}^{n} w_i Y_i\), with deterministic weights \(w_i = \frac{x_i - \bar{x}}{\sum (x_i - \bar{x})^2}\). This linear form is the cornerstone of variance calculations and the Gauss-Markov theorem later in the course.


Exercise 2.5

The data in the plot can be loaded into R using the data frame provided in the code block below.

Use this data to compute the numerical values of \(\hat{\beta}_0\) and \(\hat{\beta}_1\). Once you calculate both values, plug them into the sliders in the plot above to find the corresponding RSS!

Tip💡 Hint

Recall from Exercises 2.2 and 2.4 how the slope and intercept relate to summary statistics:

\[\hat{\beta}_1 = r_{XY}\frac{S_Y}{S_X}, \qquad \hat{\beta}_0 = \bar{Y} - \hat{\beta}_1 \bar{x}\]

Think about which base R functions calculate sample correlation, standard deviation, and sample mean from the vectors data$x and data$y.

Note📋 Solution

Method 1: Using the derived summary-statistic formulas

beta_1_hat <- cor(data$x, data$y) * (sd(data$y) / sd(data$x))
beta_0_hat <- mean(data$y) - beta_1_hat * mean(data$x)

print(beta_1_hat)
# [1] 1.630748

print(beta_0_hat)
# [1] 4.791133

Method 2: Verifying with R’s built-in lm() function

model <- lm(y ~ x, data = data)
coef(model)
# (Intercept)           x 
#    4.791133    1.630748


Exercise 2.6

This time, compute the Residual Sum of Squares (RSS) yourself in R. Does your computed value match the one in the interactive plot?

Tip💡 Hint

Recall the definition of the Residual Sum of Squares:

\[RSS = \sum_{i=1}^{n} (Y_i - \hat{Y}_i)^2\]

  1. First, calculate the fitted values \(\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i\) using the coefficients you computed in Exercise 2.5 and vector data$x.
  2. Then, take the differences \((Y_i - \hat{Y}_i)\), square them, and sum them across all observations using sum().
Note📋 Solution 
y_hat <- beta_0_hat + beta_1_hat * data$x
rss <- sum((data$y - y_hat)^2)

cat("Exact RSS:   ", rss, "\n")
cat("Rounded RSS: ", round(rss, 3), "\n")
# Exact RSS:    67.69049 
# Rounded RSS:  67.691

Notice that the exact minimum RSS is \(\approx 67.6905\). In the interactive plot, the sliders step by \(0.01\), rounding the parameters to \(\beta_0 = 4.79\) and \(\beta_1 = 1.63\). Evaluating the RSS at those rounded values gives:

\[RSS(4.79, 1.63) \approx 67.6907 \approx 67.691\]

which matches the title in the interactive plot.

Exercise 3 - Distribution of \(Y_i\)

Consider the following linear regression model:

\[ Y_i = \beta_0 + \beta_1 x_i + \epsilon_i \tag{1}\]

where \(\epsilon_i \sim N(0, \sigma^2)\) are independent and identically distributed normal random variables.


Exercise 3.1

Calculate the conditional expectation and variance of \(\left.Y_i\right|X_i=x_i\).

Remember, given \(X_i = x_i\), the values \(x_i\), \(\beta_0\), and \(\beta_1\) are deterministic (fixed constants).

The conditional expected value of \(Y_i\) given \(X_i = x_i\) is:

\[ E[Y_i \mid X_i = x_i] = E[\beta_0 + \beta_1 x_i + \epsilon_i] = E[\beta_0] + E[\beta_1 x_i] + E[\epsilon_i] = \beta_0 + \beta_1 x_i + 0 = \beta_0 + \beta_1 x_i \]

The conditional variance of \(Y_i\) given \(X_i = x_i\) is:

\[ \text{Var}(Y_i \mid X_i = x_i) = \text{Var}(\beta_0 + \beta_1 x_i + \epsilon_i) = \text{Var}(\epsilon_i) = \sigma^2 \quad \blacksquare \]


Exercise 3.2

What is the distribution of \(Y_i\)?

Recall that any linear transformation of a normally distributed random variable is also normally distributed.

Since \(\epsilon_i \sim N(0, \sigma^2)\) and \(Y_i\) is a linear transformation of \(\epsilon_i\), \(Y_i\) conditional on \(X_i = x_i\) is also normally distributed:

\[ Y_i \mid X_i = x_i \sim N(\beta_0 + \beta_1 x_i, \sigma^2) \quad \blacksquare \]


Exercise 4

Let \((x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)\) be a sample of size \(n\) of \(X\) and \(Y\). Suppose \(x_i = c\) for all \(i = 1, \ldots, n\).

Exercise 4.1

Derive the least squares estimators of \(\beta_0\) and \(\beta_1\) in this case considering the model Equation 1.

Substitute \(x_i = c\) into the \(RSS(b_0, b_1)\) objective function. Differentiate with respect to \(b_0\) and \(b_1\), set the partial derivatives to zero, and examine the resulting system of equations. Do you obtain two independent equations, or two copies of the same relationship?

The Residual Sum of Squares (RSS) is given by:

\[ RSS(b_0, b_1) = \sum_{i=1}^{n} (y_i - b_0 - b_1 x_i)^2 \]

Since \(x_i = c\) for all \(i\), we have:

\[ RSS(b_0, b_1) = \sum_{i=1}^{n} (y_i - b_0 - b_1 c)^2 \]

To find the least squares estimators, we take the partial derivatives with respect to \(b_0\) and \(b_1\) and set them to zero:

\[ \begin{aligned} \frac{\partial RSS}{\partial b_0} = 0 \implies -2\sum_{i=1}^{n} (y_i - \hat{\beta}_0 - \hat{\beta}_1 c) &= 0 \\ \sum_{i=1}^{n} y_i - n\hat{\beta}_0 - nc\hat{\beta}_1 &= 0 \\ n\bar{y} - n\hat{\beta}_0 - nc\hat{\beta}_1 &= 0 \\ \bar{y} - \hat{\beta}_0 - c\hat{\beta}_1 &= 0 \end{aligned} \]

which leads to \(\hat{\beta}_0 + c\hat{\beta}_1 = \bar{y}\).

Next, taking the partial derivative with respect to \(b_1\):

\[ \begin{aligned} \frac{\partial RSS}{\partial b_1} = 0 \implies -2c\sum_{i=1}^{n} (y_i - \hat{\beta}_0 - \hat{\beta}_1 c) &= 0 \\ \bar{y} - \hat{\beta}_0 - c\hat{\beta}_1 &= 0 \end{aligned} \]

which gives the exact same equation: \(\hat{\beta}_0 + c\hat{\beta}_1 = \bar{y}\).

Since both partial derivatives lead to the same equation, we have a single equation with two unknowns. This means there is no unique solution for \(\hat{\beta}_0\) and \(\hat{\beta}_1\). Any pair of estimators \((\hat{\beta}_0, \hat{\beta}_1)\) that satisfies the condition:

\[\hat{\beta}_0 + c\hat{\beta}_1 = \bar{y}\]

will minimize the RSS.

This result is intuition-friendly: when all \(x_i\) are the same, the data contains zero variability in \(X\) (\(S_X^2 = 0\)), providing no information to estimate a slope. Any straight line passing through the point of averages \((c, \bar{Y})\) is a valid least squares regression line and achieves the exact same minimal RSS. \(\blacksquare\)


Exercise 4.2

Suppose \(c=1\) and \(\bar{Y} = 10\). Draw the graph of the regression line using \(\hat{\beta}_1 = 10\) and another graph using \(\hat{\beta}_1 = 0\). What do you notice?

Plug \(c=1\) and \(\bar{Y}=10\) into the constraint \(\hat{\beta}_0 + c\hat{\beta}_1 = \bar{Y}\). For each candidate slope \(\hat{\beta}_1\), solve for the corresponding intercept \(\hat{\beta}_0\), write the line equation \(y = \hat{\beta}_0 + \hat{\beta}_1 x\), and check where both lines intersect.

If \(c = 1\) and \(\bar{Y} = 10\), the condition is:

\[\hat{\beta}_0 + \hat{\beta}_1 = 10\]

  • If we choose \(\hat{\beta}_1 = 10\), then \(\hat{\beta}_0 = 10 - 10 = 0\), and the regression line is \(y = 10x\).
  • If we choose \(\hat{\beta}_1 = 0\), then \(\hat{\beta}_0 = 10 - 0 = 10\), and the regression line is \(y = 10\).
  • If we choose \(\hat{\beta}_1 = -10\), then \(\hat{\beta}_0 = 10 - (-10) = 20\), and the regression line is \(y = 20 - 10x\).

Both lines (and indeed infinitely many other lines satisfying the constraint) pass through the point \((1, 10)\), which is \((c, \bar{Y})\). When all \(x_i\) are identical, the data provides no information about how \(Y\) changes as \(X\) varies. Any line pivoting through the centroid \((c, \bar{Y})\) fits the data equally well and achieves the same minimum RSS.

Scatter plot of Y vs X when X is constant (c = 1, Ȳ = 10) with three alternative least squares regression lines.


The plot below shows the surface of the Residual Sum of Squares (RSS) as a function of \(\beta_0\) and \(\beta_1\). This is what the objective function you minimized in Exercise 2.1 looks like when \(c=1\) and \(\bar{Y} = 10\). The red line indicates the theoretical minimum of the RSS, which occurs when \(\beta_0 = \bar{Y} - c\beta_1\). As you can see, the RSS does not have a unique minimum because \(X\) is constant. Instead, there is a valley of minima along which the RSS is minimized.

Figure 1: Surface plot of the Sum of Squared Residuals (SSR) as a function of b_0 and b_1.