Least Squares Estimators

STAT 306 - Module 02

Learning Objectives

  • By the end of this lecture, you should be able to:
    • Distinguish model response random variables \(Y_i\) from observed sample values \(y_i\) (with fixed \(x_i\)).
    • Define residuals and formulate the Residual Sum of Squares (RSS) criterion.
    • Derive the least squares estimators \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\) by minimizing RSS.
    • Fit simple linear regression models in R using lm() and extract parameter estimates.

Simple Linear Regression

  • A simple linear model is written as \[ Y = \beta_0 + \beta_1 x + \varepsilon \]
    • \(Y\) is the response variable;
    • \(x\) is the explanatory variable;
    • \(\varepsilon\) is the error term.
  • We assume that the mean of \(Y\) is linear in \(x\) (where \(x\) is considered a constant), i.e., \[ \mathbb{E}[Y] = \beta_0 + \beta_1 x \]

Notation

  • Unfortunately, the parameters \(\beta_0\) and \(\beta_1\) are unknown;

    • We need to estimate them from the data.

  • We collect a sample of \(n\) data points: \((x_1, y_1), \dots, (x_n, y_n)\).

  • For example, for the penguins:

    • \(i\) refers to each penguin and can vary from 1 to 344;
    • \(x_i\) is the flipper length of the \(i\)-th penguin;
    • \(y_i\) is the body mass of the \(i\)-th penguin.
i flipper_length_mm (x) body_mass_g (y)
1 181 3750
2 186 3800
3 195 3250
4 NA NA
5 193 3450
6 190 3650
7 181 3625
8 195 4675
9 193 3475
10 190 4250
11 186 3300
12 180 3700
13 182 3200
14 191 3800
15 198 4400
16 185 3700
17 195 3450
18 197 4500
19 184 3325
20 194 4200
21 174 3400
22 180 3600
23 189 3800
24 185 3950
25 180 3800
26 187 3800
27 183 3550
28 187 3200
29 172 3150
30 180 3950
31 178 3250
32 178 3900
33 188 3300
34 184 3900
35 195 3325
36 196 4150
37 190 3950
38 180 3550
39 181 3300
40 184 4650
41 182 3150
42 195 3900
43 186 3100
44 196 4400
45 185 3000
46 190 4600
47 182 3425
48 179 2975
49 190 3450
50 191 4150
51 186 3500
52 188 4300
53 190 3450
54 200 4050
55 187 2900
56 191 3700
57 186 3550
58 193 3800
59 181 2850
60 194 3750
61 185 3150
62 195 4400
63 185 3600
64 192 4050
65 184 2850
66 192 3950
67 195 3350
68 188 4100
69 190 3050
70 198 4450
71 190 3600
72 190 3900
73 196 3550
74 197 4150
75 190 3700
76 195 4250
77 191 3700
78 184 3900
79 187 3550
80 195 4000
81 189 3200
82 196 4700
83 187 3800
84 193 4200
85 191 3350
86 194 3550
87 190 3800
88 189 3500
89 189 3950
90 190 3600
91 202 3550
92 205 4300
93 185 3400
94 186 4450
95 187 3300
96 208 4300
97 190 3700
98 196 4350
99 178 2900
100 192 4100
101 192 3725
102 203 4725
103 183 3075
104 190 4250
105 193 2925
106 184 3550
107 199 3750
108 190 3900
109 181 3175
110 197 4775
111 198 3825
112 191 4600
113 193 3200
114 197 4275
115 191 3900
116 196 4075
117 188 2900
118 199 3775
119 189 3350
120 189 3325
121 187 3150
122 198 3500
123 176 3450
124 202 3875
125 186 3050
126 199 4000
127 191 3275
128 195 4300
129 191 3050
130 210 4000
131 190 3325
132 197 3500
133 193 3500
134 199 4475
135 187 3425
136 190 3900
137 191 3175
138 200 3975
139 185 3400
140 193 4250
141 193 3400
142 187 3475
143 188 3050
144 190 3725
145 192 3000
146 185 3650
147 190 4250
148 184 3475
149 195 3450
150 193 3750
151 187 3700
152 201 4000
153 211 4500
154 230 5700
155 210 4450
156 218 5700
157 215 5400
158 210 4550
159 211 4800
160 219 5200
161 209 4400
162 215 5150
163 214 4650
164 216 5550
165 214 4650
166 213 5850
167 210 4200
168 217 5850
169 210 4150
170 221 6300
171 209 4800
172 222 5350
173 218 5700
174 215 5000
175 213 4400
176 215 5050
177 215 5000
178 215 5100
179 216 4100
180 215 5650
181 210 4600
182 220 5550
183 222 5250
184 209 4700
185 207 5050
186 230 6050
187 220 5150
188 220 5400
189 213 4950
190 219 5250
191 208 4350
192 208 5350
193 208 3950
194 225 5700
195 210 4300
196 216 4750
197 222 5550
198 217 4900
199 210 4200
200 225 5400
201 213 5100
202 215 5300
203 210 4850
204 220 5300
205 210 4400
206 225 5000
207 217 4900
208 220 5050
209 208 4300
210 220 5000
211 208 4450
212 224 5550
213 208 4200
214 221 5300
215 214 4400
216 231 5650
217 219 4700
218 230 5700
219 214 4650
220 229 5800
221 220 4700
222 223 5550
223 216 4750
224 221 5000
225 221 5100
226 217 5200
227 216 4700
228 230 5800
229 209 4600
230 220 6000
231 215 4750
232 223 5950
233 212 4625
234 221 5450
235 212 4725
236 224 5350
237 212 4750
238 228 5600
239 218 4600
240 218 5300
241 212 4875
242 230 5550
243 218 4950
244 228 5400
245 212 4750
246 224 5650
247 214 4850
248 226 5200
249 216 4925
250 222 4875
251 203 4625
252 225 5250
253 219 4850
254 228 5600
255 215 4975
256 228 5500
257 216 4725
258 215 5500
259 210 4700
260 219 5500
261 208 4575
262 209 5500
263 216 5000
264 229 5950
265 213 4650
266 230 5500
267 217 4375
268 230 5850
269 217 4875
270 222 6000
271 214 4925
272 NA NA
273 215 4850
274 222 5750
275 212 5200
276 213 5400
277 192 3500
278 196 3900
279 193 3650
280 188 3525
281 197 3725
282 198 3950
283 178 3250
284 197 3750
285 195 4150
286 198 3700
287 193 3800
288 194 3775
289 185 3700
290 201 4050
291 190 3575
292 201 4050
293 197 3300
294 181 3700
295 190 3450
296 195 4400
297 181 3600
298 191 3400
299 187 2900
300 193 3800
301 195 3300
302 197 4150
303 200 3400
304 200 3800
305 191 3700
306 205 4550
307 187 3200
308 201 4300
309 187 3350
310 203 4100
311 195 3600
312 199 3900
313 195 3850
314 210 4800
315 192 2700
316 205 4500
317 210 3950
318 187 3650
319 196 3550
320 196 3500
321 196 3675
322 201 4450
323 190 3400
324 212 4300
325 187 3250
326 198 3675
327 199 3325
328 201 3950
329 193 3600
330 203 4050
331 187 3350
332 197 3450
333 191 3250
334 203 4050
335 202 3800
336 194 3525
337 206 3950
338 189 3650
339 195 3650
340 207 4000
341 202 3400
342 193 3775
343 210 4100
344 198 3775

Note

Note that \(Y_i\) is random, while \(y_i\) is its observed value and is treated as fixed.

  • Before sampling, we do not know the value of \(Y_i\).
  • After sampling, we observe \(y_i\).

Fitting the model

You might need to refresh this page to show the plot

  • Let \((x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)\) be a sample of size \(n\) of the variables \(X\) and \(Y\);

  • These are just points in a scatter plot; e.g.,

  • Naturally, we could use many different lines to fit these points;
    • Which line should we use?

Fitting the model: the residuals

  • We want a line that is “close” to the points;

  • The difference between the \(i\)-th observed point and the line is called a residual and is denoted by \(e_i\), \[ e_i = y_i - (b_0 + b_1 x_i) \]

    • \(b_0\) is the intercept of a potential line;
    • \(b_1\) is the slope of a potential line.

iClicker: Residuals

For an observed point \((x_i, y_i)\) and candidate line with intercept \(b_0\) and slope \(b_1\), the residual \(e_i\) is:

  1. \(e_i = Y_i - \mathbb{E}[Y_i]\)
  2. \(e_i = y_i - (b_0 + b_1 x_i)\)
  3. \(e_i = (b_0 + b_1 x_i) - \bar{y}\)
  4. \(e_i = x_i - \bar{x}\)

iClicker: Residuals Solution

For an observed point \((x_i, y_i)\) and candidate line with intercept \(b_0\) and slope \(b_1\), the residual \(e_i\) is:

  1. \(e_i = Y_i - \mathbb{E}[Y_i]\)
  2. \(e_i = y_i - (b_0 + b_1 x_i)\)
  3. \(e_i = (b_0 + b_1 x_i) - \bar{y}\)
  4. \(e_i = x_i - \bar{x}\)

Fitting the model: Residual Sum of Squares

  • A line close to the points means small residuals;

  • But how do we measure it?

  • One common way is to use the Residual Sum of Squares (RSS): \[ \begin{align*} RSS(b_0, b_1) &= \sum_{i=1}^n e_i^2\\ &= \sum_{i=1}^n \left[ y_i - (b_0 + b_1 x_i) \right]^2 \end{align*} \]

  • So we want to find the values of \(b_0\) and \(b_1\) that minimize the RSS.

Fitting the model: minimizing RSS

You might need to refresh this page to show the plot

  • Which values should we use for \(b_0\) and \(b_1\)?

Fitting the model: minimizing RSS

You might need to refresh this page to show the plot

  • We want \(b_0\) and \(b_1\) that minimize the Residual Sum of Squares (RSS);

Fitting the model: normal equations

  • We can use calculus to find the values of \(b_0\) and \(b_1\) that minimize the RSS for our sample: \[ RSS(b_0, b_1) = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n \left( y_i - (b_0 + b_1 x_i) \right)^2 \]
    1. Take the partial derivatives of \(RSS(b_0, b_1)\) with respect to \(b_0\) and \(b_1\): \[ \frac{\partial RSS(b_0, b_1)}{\partial b_0} = -2\sum_{i=1}^n \left( y_i - (b_0 + b_1 x_i) \right) \] \[ \frac{\partial RSS(b_0, b_1)}{\partial b_1} = -2\sum_{i=1}^n \left( y_i - (b_0 + b_1 x_i) \right)x_i \]
    2. Set the partial derivatives to zero: \[ \frac{\partial RSS(b_0, b_1)}{\partial b_0} = -2\sum_{i=1}^n \left( y_i - (\widehat{\beta}_0 + \widehat{\beta}_1 x_i) \right) = 0 \] \[ \frac{\partial RSS(b_0, b_1)}{\partial b_1} = -2\sum_{i=1}^n \left( y_i - (\widehat{\beta}_0 + \widehat{\beta}_1 x_i) \right)x_i = 0 \]
      • The solutions to these equations are the least squares estimates \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\).
    3. We have a system of two linear equations. Let’s organize it: \[ \widehat{\beta}_0 + \bar{x}\widehat{\beta}_1 = \bar{y} \] \[ \widehat{\beta}_0\sum_{i=1}^n x_i + \widehat{\beta}_1 \sum_{i=1}^n x_i^2 = \sum_{i=1}^n y_ix_i \]
      • These are the so-called normal equations.
    4. Solve the system of equations to get the estimates \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\): \[ \widehat{\beta}_0 = \bar{y} - \widehat{\beta}_1\bar{x} \] \[ \widehat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = r_{xy}\frac{s_y}{s_x} \]

iClicker: Property of LSE Residuals

Setting \(\frac{\partial RSS(b_0, b_1)}{\partial b_0} = -2\sum_{i=1}^n \left(y_i - (\widehat{\beta}_0 + \widehat{\beta}_1 x_i)\right) = 0\) reveals an important property. What is the sum of the least squares residuals \(\sum_{i=1}^n e_i\)?

  1. \(\sum_{i=1}^n e_i > 0\)
  2. \(\sum_{i=1}^n e_i = 1\)
  3. \(\sum_{i=1}^n e_i = 0\)
  4. It depends on the dataset.

iClicker: Property of LSE Residuals Solution

Setting \(\frac{\partial RSS(b_0, b_1)}{\partial b_0} = -2\sum_{i=1}^n \left(y_i - (\widehat{\beta}_0 + \widehat{\beta}_1 x_i)\right) = 0\) reveals an important property. What is the sum of the least squares residuals \(\sum_{i=1}^n e_i\)?

  1. \(\sum_{i=1}^n e_i > 0\)
  2. \(\sum_{i=1}^n e_i = 1\)
  3. \(\sum_{i=1}^n e_i = 0\)
  4. It depends on the dataset.

Example: Fitting a model

  • Let’s fit a linear model relating Flipper Length and Body Mass of penguins.
  • To fit a linear model in R, we use the lm function.
  • Yay! Our formulae match the output of lm() exactly.

Activity: Your Turn!

  • Fit a linear model predicting body_mass_g using bill_length_mm on the penguins_clean dataset.
  • Replace the blank ______ below with the lm() call and run your code to check your answer.

Tip

Remember that lm() takes a formula of the form response ~ explanatory. Here, body_mass_g is the response and bill_length_mm is the explanatory variable. Make sure to specify the dataset with data = penguins_clean.

Solution

# Fit a model predicting body_mass_g from bill_length_mm
model_bill <- lm(body_mass_g ~ bill_length_mm, data = penguins_clean)
summary(model_bill)$coefficients
  • Discussion:
    • What is the estimated slope \(\widehat{\beta}_1\)?
    • How would you interpret it in terms of body mass per millimeter of bill length?

Estimators are random variables

  • For a fixed sample, we computed numerical estimates: \[ \widehat{\beta}_1 = \frac{\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^n(x_i-\bar{x})^2} \qquad\quad \widehat{\beta}_0 = \bar{y} - \widehat{\beta}_1\bar{x} \]

  • Across repeated samples, the responses are random variables \(Y_1, \dots, Y_n\).

  • Replacing the observed \(y_i\) with random variables \(Y_i\) gives the estimators (statistics): \[ \widehat{\beta}_1 = \frac{\sum_{i=1}^n(x_i-\bar{x})(Y_i-\bar{Y})}{\sum_{i=1}^n(x_i-\bar{x})^2}\qquad\quad \widehat{\beta}_0 = \bar{Y} - \widehat{\beta}_1\bar{x} \; \]

  • Because estimators are random variables, their values change from sample to sample.

  • We need to understand how they behave: what is their sampling distribution?

What We Learned

  • Residuals: \(e_i = y_i - (b_0 + b_1 x_i)\) measure the vertical discrepancies between observed points and a candidate regression line.
  • Least Squares Criterion: We choose \(b_0\) and \(b_1\) that minimize the Residual Sum of Squares: \[RSS(b_0, b_1) = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n \left[y_i - (b_0 + b_1 x_i)\right]^2\]
  • Normal Equations & Estimators: Setting partial derivatives to zero yields closed-form solutions: \[\widehat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = r_{xy}\frac{s_y}{s_x}, \qquad \widehat{\beta}_0 = \bar{y} - \widehat{\beta}_1\bar{x}\]
  • Fitting in R: We use lm(response ~ explanatory, data = ...) to compute least squares estimates directly.

Looking Ahead

  • Because the responses \(Y_i\) are random variables, the estimators \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\) are also random variables.
  • A new sample yields different numerical estimates.
  • Next module we will derive the sampling distributions of \(\widehat{\beta}_0\) and \(\widehat{\beta}_1\).