Sampling Distributions of Least Squares Estimators

STAT 306 - Module 03

iClicker 1: Review

Suppose we fitted a model \(\widehat{Y} = \widehat{\beta}_0 + \widehat{\beta}_1 x\), and now \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\) are known. For \(x=3\), what would be our prediction for \(Y\)?

  1. \(\widehat{Y} = 3\widehat{\beta}_1\)
  2. \(\widehat{Y} = \widehat{\beta}_0 + 3\widehat{\beta}_1x\)
  3. \(\widehat{Y} = \beta_0 + 3\beta_1\)
  4. \(\widehat{Y} = \widehat{\beta}_0 + 3\widehat{\beta}_1\)
  5. \(\widehat{Y} = b_0 + 3b_1\)

iClicker 1: Review Solution

Suppose we fitted a model \(\widehat{Y} = \widehat{\beta}_0 + \widehat{\beta}_1 x\), and now \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\) are known. For \(x=3\), what would be our prediction for \(Y\)?

  1. \(\widehat{Y} = 3\widehat{\beta}_1\)
  2. \(\widehat{Y} = \widehat{\beta}_0 + 3\widehat{\beta}_1x\)
  3. \(\widehat{Y} = \beta_0 + 3\beta_1\)
  4. \(\widehat{Y} = \widehat{\beta}_0 + 3\widehat{\beta}_1\)
  5. \(\widehat{Y} = b_0 + 3b_1\)

Learning Objectives

  • By the end of this lecture, you should be able to:
    • Compute \(\mathbb{E}\left[\widehat{\beta}_0\right]\) and \(\mathbb{E}\left[\widehat{\beta}_1\right]\) and demonstrate that both estimators are unbiased.
    • Compute \(\text{Var}(\widehat{\beta}_0)\) and \(\text{Var}(\widehat{\beta}_1)\) and identify factors affecting their precision.
    • State the exact sampling distributions of \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\) under normal error assumptions.

Review: Properties of Covariance

  • For any two random variables \(X\) and \(Y\), we have that \[ \begin{align*} \text{Cov}(X,Y) &= \mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])]\\ &= \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y] \;. \end{align*}\]

  • For any random variables \(X, Y, Z\) and constants \(a, b, c, d\), we have that

    • \(\text{Cov}(X,Y) = \text{Cov}(Y,X)\)
    • \(\text{Cov}(X,X) = \text{Var}(X)\)
    • \(\text{Cov}(aX+b, cY+d) = ac\text{Cov}(X,Y)\)
    • \(\text{Cov}(X+Y,Z) = \text{Cov}(X,Z) + \text{Cov}(Y,Z)\)
    • If \(X\) and \(Y\) are independent, then \(\text{Cov}(X,Y) = 0\).
    • \(\text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y) + 2\text{Cov}(X,Y)\)
    • \(\text{Var}(X-Y) = \text{Var}(X) + \text{Var}(Y) - 2\text{Cov}(X,Y)\)
    • For any random variables \(X_1,\dotsc,X_n\) and \(Y_1,\dotsc,Y_m\), we have: \[ \text{Cov}\left(\sum_{i=1}^nX_i,\sum_{j=1}^mY_j\right) = \sum_{i=1}^n\sum_{j=1}^m\text{Cov}(X_i,Y_j) \;. \]

Simple Linear Regression Model

  • For a sample of \(n\) observations, the simple linear regression model is: \[ Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \qquad i = 1, \dots, n \]
    • \(x_1, \dots, x_n\) are fixed (non-random) constants with \(\sum_{i=1}^n (x_i - \bar{x})^2 > 0\);
    • \(\beta_0\) and \(\beta_1\) are unknown parameters;
    • \(\varepsilon_i \overset{\text{iid}}{\sim} N(0, \sigma^2)\) are independent and identically distributed errors.

Properties of the Least Squares Estimators (LSE)

Estimators are random variables

  • Estimators are statistics, therefore random variables (because they depend on a random sample): \[ \widehat{\beta}_1 = \frac{\sum_{i=1}^n(x_i-\bar{x})(Y_i-\bar{Y})}{\sum_{i=1}^n(x_i-\bar{x})^2}\qquad\quad \widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1\bar{x} \; \]

  • We want to interpret them, but how can we interpret and make decisions based on quantities that can change from sample to sample?

  • We need to understand how they behave, i.e., what is their sampling distribution.

Our objective now

  • We want to derive the distributions of \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\).

  • That’s all we want to do for now!



  • Let’s get started!

Question

Consider the least squares estimators

\[ \begin{align*} \widehat{\beta}_1 &= \frac{\sum_{i=1}^nx_iY_i - n\bar{x}\bar{Y}}{\sum_{i=1}^nx_i^2-n\bar{x}^2} \;, & \widehat{\beta}_0 &= \bar{Y} - \widehat{\beta}_1\bar{x} \;. \end{align*} \]

Explain why \(\widehat{\beta}_1\) and \(\widehat{\beta}_0\) are normal random variables.

Solution

Because both \(\widehat{\beta}_1\) and \(\widehat{\beta}_0\) are linear combinations of independent normal random variables \(Y_1, \dots, Y_n\), they are also normally distributed.

What is left?

  • We now know that both \(\widehat{\beta}_1\) and \(\widehat{\beta}_0\) are normal random variables.

  • All that is left is to find their means and variances, i.e., \(\mathbb{E}[\widehat{\beta}_1]\), \(\text{Var}(\widehat{\beta}_1)\), \(\mathbb{E}[\widehat{\beta}_0]\) and \(\text{Var}(\widehat{\beta}_0)\).

  • With these quantities, we will be able to write down the distributions of \(\widehat{\beta}_1\) and \(\widehat{\beta}_0\).

  • Let’s start with the slope estimator \(\widehat{\beta}_1\).

Exercise: Properties of \(\widehat{\beta}_1\), \(\mathbb{E}[\widehat{\beta}_1]\)

Compute \(\mathbb{E}[\widehat{\beta}_1]\).

\[ \begin{align} \widehat{\beta}_1 &= r_{XY}\frac{S_Y}{S_X} = \frac{\sum_{i=1}^n(x_i-\bar{x})(Y_i-\bar{Y})}{\sum_{i=1}^n(x_i-\bar{x})^2} = \frac{\sum_{i=1}^nx_iY_i - n\bar{x}\bar{Y}}{\sum_{i=1}^nx_i^2-n\bar{x}^2} \; \end{align} \]

Solution

\[ \begin{align} \mathbb{E}[\widehat{\beta}_1] &= \mathbb{E}\left[\frac{\sum_{i=1}^n(x_i-\bar{x})(Y_i-\bar{Y})}{\sum_{i=1}^n(x_i-\bar{x})^2}\right] \\ &= \frac{\sum_{i=1}^n(x_i-\bar{x})\left(\mathbb{E}\left[Y_i\right]-\mathbb{E}\left[\bar{Y}\right]\right)}{\sum_{i=1}^n(x_i-\bar{x})^2} \\ &= \frac{\sum_{i=1}^n(x_i-\bar{x})\left[\left(\beta_0+\beta_1x_i\right)-\left(\beta_0+\beta_1\bar{x}\right)\right]}{\sum_{i=1}^n(x_i-\bar{x})^2} \\ &= \frac{\beta_1\sum_{i=1}^n(x_i-\bar{x})\left(x_i-\bar{x}\right)}{\sum_{i=1}^n(x_i-\bar{x})^2} \\ &= \beta_1 \\ \end{align} \]

  • Since \(\mathbb{E}[\widehat{\beta}_1] = \beta_1\), we say that \(\widehat{\beta}_1\) is an unbiased estimator of \(\beta_1\).

Exercise: Properties of \(\widehat{\beta}_1\), \(\text{Var}(\widehat{\beta}_1)\)

Compute \(\text{Var}(\widehat{\beta}_1)\).

\[ \begin{align} \widehat{\beta}_1 &= r_{XY}\frac{S_Y}{S_X} = \frac{\sum_{i=1}^n(x_i-\bar{x})(Y_i-\bar{Y})}{\sum_{i=1}^n(x_i-\bar{x})^2} = \frac{\sum_{i=1}^nx_iY_i - n\bar{x}\bar{Y}}{\sum_{i=1}^nx_i^2-n\bar{x}^2} \; \end{align} \]

Solution \[ \begin{align} \text{Var}(\widehat{\beta}_1) &= \text{Var}\left[\frac{\sum_{i=1}^n(x_i-\bar{x})(Y_i-\bar{Y})}{\sum_{i=1}^n(x_i-\bar{x})^2}\right] \\ &= \text{Var}\left[\frac{\sum_{i=1}^n(x_i-\bar{x})Y_i}{\sum_{i=1}^n(x_i-\bar{x})^2}-\frac{\bar{Y}\sum_{i=1}^n(x_i-\bar{x})}{\sum_{i=1}^n(x_i-\bar{x})^2}\right] \\ &= \text{Var}\left[\frac{\sum_{i=1}^n(x_i-\bar{x})Y_i}{\sum_{i=1}^n(x_i-\bar{x})^2}\right] \quad \left(\text{since } \sum_{i=1}^n(x_i-\bar{x}) = 0\right)\\ &= \frac{\sum_{i=1}^n(x_i-\bar{x})^2\text{Var}\left[Y_i\right]}{\left(\sum_{i=1}^n(x_i-\bar{x})^2\right)^2} \quad \left(\text{since } Y_1, \dots, Y_n \text{ are independent}\right)\\ &= \frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2} \\ \end{align} \]

Exercise: Properties of \(\widehat{\beta}_0\), \(\mathbb{E}[\widehat{\beta}_0]\)

  • Compute \(\mathbb{E}[\widehat{\beta}_0]\). \[ \begin{align} \widehat{\beta}_0 &= \bar{Y} - \widehat{\beta}_1\bar{x} \; \end{align} \]

Solution \[ \begin{align} \mathbb{E}[\widehat{\beta}_0] &= \mathbb{E}[\bar{Y}] - \mathbb{E}[\widehat{\beta}_1]\bar{x} \\ &= (\beta_0 + \beta_1\bar{x}) - \beta_1\bar{x} \\ &= \beta_0 \;. \end{align} \]

  • Therefore, \(\widehat{\beta}_0\) is also an unbiased estimator of \(\beta_0\).

Exercise: Properties of \(\widehat{\beta}_0\), \(\text{Var}(\widehat{\beta}_0)\)

  • Compute \(\text{Var}(\widehat{\beta}_0)\). \[ \begin{align} \widehat{\beta}_0 &= \bar{Y} - \widehat{\beta}_1\bar{x} \; \end{align} \]

Solution

\[ \begin{align} \text{Var}(\widehat{\beta}_0) &= \text{Var}\left[\bar{Y} - \widehat{\beta}_1\bar{x}\right] \\ &= \text{Var}[\bar{Y}] + \text{Var}[\widehat{\beta}_1]\bar{x}^2 - 2\bar{x}\text{Cov}[\bar{Y},\widehat{\beta}_1] \\ &= \frac{\sigma^2}{n} + \frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\bar{x}^2 - 2\bar{x}\text{Cov}\left[\bar{Y},\frac{\sum_{i=1}^n(x_i-\bar{x})Y_i}{\sum_{i=1}^n(x_i-\bar{x})^2}\right] \\ &= \frac{\sigma^2}{n} + \frac{\sigma^2\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2} \\ &= \sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\right) \; \end{align} \]

  • This is because \(\text{Cov}(Y_i,\bar{Y}) = \text{Cov}\left(Y_i, \frac{1}{n}\sum_{k=1}^n Y_k\right) = \frac{1}{n}\text{Var}(Y_i) = \frac{\sigma^2}{n}\):

\[ \begin{align*} \text{Cov}[\bar{Y}, \widehat{\beta}_1] &= \text{Cov}\left[ \bar{Y}, \frac{\sum_{j=1}^{n} (x_j - \bar{x})Y_j}{\sum_{k=1}^{n} (x_k - \bar{x})^2} \right] \\\\ &= \frac{\sum_{j=1}^{n} (x_j - \bar{x})\text{Cov}\left[ \bar{Y}, Y_j \right]}{\sum_{k=1}^{n} (x_k - \bar{x})^2} \\\\ &= \frac{\sigma^2}{n}\frac{\sum_{j=1}^{n} (x_j - \bar{x})}{\sum_{k=1}^{n} (x_k - \bar{x})^2} \\\\ &= \frac{\sigma^2}{n}\frac{0}{\sum_{k=1}^{n} (x_k - \bar{x})^2} \\\\ &= 0 \end{align*} \]

iClicker 1

Are \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\) unbiased estimators of \(\beta_0\) and \(\beta_1\), respectively?

  1. Yes, both are unbiased estimators.
  2. No, only \(\widehat{\beta}_0\) is an unbiased estimator.
  3. No, only \(\widehat{\beta}_1\) is an unbiased estimator.
  4. No, both are biased.

iClicker 1: Solution

Are \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\) unbiased estimators of \(\beta_0\) and \(\beta_1\), respectively?

  1. Yes, both are unbiased estimators.
  2. No, only \(\widehat{\beta}_0\) is an unbiased estimator.
  3. No, only \(\widehat{\beta}_1\) is an unbiased estimator.
  4. No, both are biased.

iClicker 2

What is \(\text{Var}[\widehat{\beta}_1]\)?

  1. \(\frac{1}{\sum_{i=1}^n(x_i-\bar{x})^2}\)
  2. \(\frac{\sigma}{\sum_{i=1}^n(x_i-\bar{x})}\)
  3. \(\frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\)
  4. \(\sigma^2\)
  5. \(\beta_1\)

iClicker 2: Solution

What is \(\text{Var}[\widehat{\beta}_1]\)?

  1. \(\frac{1}{\sum_{i=1}^n(x_i-\bar{x})^2}\)
  2. \(\frac{\sigma}{\sum_{i=1}^n(x_i-\bar{x})}\)
  3. \(\frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\)
  4. \(\sigma^2\)
  5. \(\beta_1\)

iClicker 3

What is \(\text{Cov}(\bar{Y}, \widehat{\beta}_1)\)?

  1. \(\sigma\)
  2. \(\beta_1 \sigma^2\)
  3. \(\frac{\sigma^2}{n}\)
  4. \(\sigma^2\)
  5. 0

iClicker 3: Solution

What is \(\text{Cov}(\bar{Y}, \widehat{\beta}_1)\)?

  1. \(\sigma\)
  2. \(\beta_1 \sigma^2\)
  3. \(\frac{\sigma^2}{n}\)
  4. \(\sigma^2\)
  5. 0

iClicker 4

What is \(\text{Var}[\widehat{\beta}_0]\)?

  1. \(\sigma^2\)
  2. \(\sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\right)\)
  3. \(\frac{\sigma^2}{n}\)
  4. \(\frac{\sigma^2\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\)
  5. \(\frac{\sigma^2}{n\sum_{i=1}^n(x_i-\bar{x})^2}\)

iClicker 4: Solution

What is \(\text{Var}[\widehat{\beta}_0]\)?

  1. \(\sigma^2\)
  2. \(\sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\right)\)
  3. \(\frac{\sigma^2}{n}\)
  4. \(\frac{\sigma^2\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\)
  5. \(\frac{\sigma^2}{n\sum_{i=1}^n(x_i-\bar{x})^2}\)

iClicker 5: Factors Affecting Precision

Recall that \(\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\). Which of the following will decrease the variance (increase precision) of \(\widehat{\beta}_1\)?

  1. Increasing the error variance \(\sigma^2\).
  2. Decreasing the sample size \(n\).
  3. Choosing \(x\) values that are more widely spread out from \(\bar{x}\).
  4. Choosing \(x\) values that are clustered closely around \(\bar{x}\).

iClicker 5: Factors Affecting Precision Solution

Recall that \(\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\). Which of the following will decrease the variance (increase precision) of \(\widehat{\beta}_1\)?

  1. Increasing the error variance \(\sigma^2\).
  2. Decreasing the sample size \(n\).
  3. Choosing \(x\) values that are more widely spread out from \(\bar{x}\).
  4. Choosing \(x\) values that are clustered closely around \(\bar{x}\).

Properties of \(\widehat{\beta}_1\): Distribution

  • \(\widehat{\beta}_1=\frac{\sum_{i=1}^n(x_i-\bar{x})Y_i}{\sum_{i=1}^n(x_i-\bar{x})^2}\) is a linear combination of normal random variables; hence, also normally distributed.

  • We have that \(\mathbb{E}[\widehat{\beta}_1] = \beta_1\) and \(\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\).

  • Therefore, \(\widehat{\beta}_1 \sim N\left(\beta_1,\:\frac{\sigma^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\right)\).

Properties of \(\widehat{\beta}_0\): Distribution

  • We have that \(\mathbb{E}[\widehat{\beta}_0] = \beta_0\) and \(\text{Var}(\widehat{\beta}_0) = \frac{\sigma^2}{n} + \frac{\sigma^2\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\).

  • Additionally, \(\widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1\bar{x}\) is a linear combination of normal random variables.

  • Therefore, \(\widehat{\beta}_0 \sim N\left(\beta_0,\:\sigma^2\left(\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n(x_i-\bar{x})^2}\right)\right)\).

Simulating the Sampling Distribution

  • Let us verify our theoretical results with a simulation.

  • We fix the ground truth: \(\beta_0 = 10\), \(\beta_1 = 2\), \(\sigma = 3\).

  • The steps are:

    • Draw \(1,000\) independent samples with \(50\) observations each and fixed \(x\) values.
    • Fit lm() to each;
    • Plot the histogram of the sampling distribution of \(\widehat{\beta}_1\) and compare it to the theoretical normal curve.

What We Learned

Property Slope Estimator (\(\widehat{\beta}_1\)) Intercept Estimator (\(\widehat{\beta}_0\))
Mean \(\mathbb{E}[\widehat{\beta}_1] = \beta_1\) (unbiased) \(\mathbb{E}[\widehat{\beta}_0] = \beta_0\) (unbiased)
Variance \(\text{Var}(\widehat{\beta}_1) = \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\) \(\text{Var}(\widehat{\beta}_0) = \sigma^2 \left(\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\right)\)
Distribution \(N\left(\beta_1, \frac{\sigma^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\right)\) \(N\left(\beta_0, \sigma^2 \left(\frac{1}{n} + \frac{\bar{x}^2}{\sum_{i=1}^n (x_i - \bar{x})^2}\right)\right)\)

Looking Ahead

  • We have derived the exact sampling distributions of \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\).

  • Notice that both variances depend on \(\sigma^2\), which is unknown in practice.

  • Next module:

    • How do we estimate \(\sigma^2\) from the residuals?
    • What happens when we plug in the estimate \(s^2\)? (Enter the \(t\)-distribution!)
    • How to construct confidence intervals and conduct hypothesis tests for \(\beta_0\) and \(\beta_1\).