Standardization

STAT 200 - Chapter 5 Part I

Comparing different things

Problem 1: Books vs Movies

Have you ever participated in the discussion:

“The book was much better than the movie”?

Walt Hickey from FiveThirtyEight collected users’ reviews ratings of 382 movies and the books that originated the film.

Problem 1: Books vs Movies

Here’s a sample of the data:

FILM	METACRITIC SCORE (out of 100)	GOODREADS RATING (out of 5)
Up in the Air (2009)	83	2.9
Apocalypse Now (1979)	90	3.4
Metropolis (1927)	98	3.6
Scent of a Woman (1993)	59	3.0
Mr. Holmes (2015)	67	3.2
Sideways (2005)	94	3.7
The Graduate (1967)	77	3.4
Taking Lives (2004)	38	2.8
Dr. Strangelove (1964)	96	3.7
There Will Be Blood (2008)	92	3.7
Addicted (2014)`	32	4.3
Vampire Academy (2014)	30	4.2
The Jackal (1997)	36	4.2
Safe Haven (2013)	34	4.2
The Last Song (2010)	33	4.1
The Mortal Instruments: City of Bones (2013)	33	4.1
Percy Jackson: Sea of Monsters (2013)	39	4.2
Left Behind (2014)	12	3.8
A Walk to Remember (2002)	35	4.1
Harry Potter and the Deathly Hallows: Part 1 (2010)	65	4.6
Divergent (2014)	48	4.3
Unbroken (2014)	59	4.5
The Book Thief (2013)	53	4.4
Mortdecai (2015)	27	4.0
Where the Heart Is (2000)	30	4.0
Pay It Forward (2000)	40	4.1
The Three Musketeers (2011)	35	4.0
Seventh Son (2015)	30	4.0
Ender’s Game (2013)	51	4.3
The Help (2011)	62	4.4

Data standardization

Data standardization will measure how far a data point is from the mean in terms of standard deviations.
There are two main operations in data standardization:
- Centering: subtracting the mean from each data point.
- Scaling: dividing each data point by the standard deviation.
The result is a new data set with a mean of 0 and a standard deviation of 1.

Shifting data

To shift the data, we add a constant to each data point.
Everything looks the same, but the location of the data changes.

viewof shifter ={
  let input = Inputs.range([-10, 10], {value: 0, step: 1, label: "Shifting:"})

  return input;
}

Shifting data

Question:

When shifting the data, will the mean change?

When shifting the data, will the variance change?

Shifting data

When shifting the data, we only add a constant to the data points.
Location measures (e.g., mean, median, quantiles) will also be shifted by the same amount.
Any measure of spread (e.g., variance, std. dev., IQR) will not change.

Scaling data

To scale the data we multiply each data point by a positive constant, \(c\).

{
// set the dimensions and margins of the graph
  var margin = {top: 10, right: 30, bottom: 10, left: 100},
      width = 1050 - margin.left - margin.right,
      height = 50 - margin.top - margin.bottom;

  d3.select("#scaling-plot").html("");
  var svg = d3.select("#scaling-plot")
    .append("svg")
      .attr("width", width + margin.left + margin.right)
      .attr("height", height + margin.top + margin.bottom)
    .append("g")
      .attr("transform",
          "translate(" + margin.left + "," + margin.top + ")");


  var x = d3.scaleLinear()
    .domain([0, 300])
    .range([0, width]);
  var y = d3.scaleLinear()
    .domain([0, 100])
    .range([0, height/4]);

  let x_axis = d3.axisBottom().ticks(10).scale(x);
  
  svg.append("g")
    .attr("transform", "translate(" + 0 + "," + (height - margin.bottom) + ")")
    .call(x_axis);
  svg
    .selectAll("circle")
    .data(dataset)
    .enter()
    .append("circle")
    .attr("cx", d => x(d*scaler))
    .attr("cy", y(-5))
    .attr("r", "3px")
    .attr("stroke", "black")
    .attr("fill", "black");
}

viewof scaler ={
  let input = Inputs.range([0, 3], {value: 1, step: .1, label: "Scaling Constant:"})

  return input;
}

Scaling data

This will affect both the location and the spread of the data.
- The location measures (e.g., mean, quantiles) and measures of spread (e.g., SD, IQR) will be multiplied by \(c\).
- Note that variance is the square of the standard deviation so that it will be multiplied by \(c^2\).

Standardizing

To standardize the data, we:
1. shift the data so the mean is zero (subtract the mean)
2. scale the data so the std. dev. is one (divide by the std. dev.).
The result is

\[ \text{z-score} = \frac{\text{deviation from the mean}}{\text{Std. Dev.}} = \frac{y - \bar{y}}{s} \]

Standardizing

A z-score gives the distance between an observation and the mean in terms of standard deviation.
Z-scores are useful because they are independent of the scale and location of the data.

Standardizing

For example,
- a z-score of 1 means that the observation is 1 standard deviation above the mean.
- A z-score of -2 means that the observation is 2 standard deviations below the mean.
- A z-score of 0 means that the observation is equal to the mean.

Example: Movies vs. Books

Let’s standardize the Metacritic Scores for the movies and books data.

FILM	METACRITIC SCORE
Up in the Air (2009)	83
Apocalypse Now (1979)	90
Metropolis (1927)	98
Scent of a Woman (1993)	59
Mr. Holmes (2015)	67
Sideways (2005)	94
The Graduate (1967)	77
Taking Lives (2004)	38
Dr. Strangelove (1964)	96
There Will Be Blood (2008)	92
Addicted (2014)	32
Vampire Academy (2014)	30
The Jackal (1997)	36
Safe Haven (2013)	34
The Last Song (2010)	33
The Mortal Instruments: City of Bones (2013)	33
Percy Jackson: Sea of Monsters (2013)	39
Left Behind (2014)	12
A Walk to Remember (2002)	35
Harry Potter and the Deathly Hallows: Part 1 (2010)	65
Divergent (2014)	48
Unbroken (2014)	59
The Book Thief (2013)	53
Mortdecai (2015)	27
Where the Heart Is (2000)	30
Pay It Forward (2000)	40
The Three Musketeers (2011)	35
Seventh Son (2015)	30
Ender`s Game (2013)	51
The Help (2011)	62

First, we calculate the mean;

FILM	METACRITIC SCORE	AVERAGE
Up in the Air (2009)	83	52.6
Apocalypse Now (1979)	90	52.6
Metropolis (1927)	98	52.6
Scent of a Woman (1993)	59	52.6
Mr. Holmes (2015)	67	52.6
Sideways (2005)	94	52.6
The Graduate (1967)	77	52.6
Taking Lives (2004)	38	52.6
Dr. Strangelove (1964)	96	52.6
There Will Be Blood (2008)	92	52.6
Addicted (2014)	32	52.6
Vampire Academy (2014)	30	52.6
The Jackal (1997)	36	52.6
Safe Haven (2013)	34	52.6
The Last Song (2010)	33	52.6
The Mortal Instruments: City of Bones (2013)	33	52.6
Percy Jackson: Sea of Monsters (2013)	39	52.6
Left Behind (2014)	12	52.6
A Walk to Remember (2002)	35	52.6
Harry Potter and the Deathly Hallows: Part 1 (2010)	65	52.6
Divergent (2014)	48	52.6
Unbroken (2014)	59	52.6
The Book Thief (2013)	53	52.6
Mortdecai (2015)	27	52.6
Where the Heart Is (2000)	30	52.6
Pay It Forward (2000)	40	52.6
The Three Musketeers (2011)	35	52.6
Seventh Son (2015)	30	52.6
Ender`s Game (2013)	51	52.6
The Help (2011)	62	52.6

Now, we calculate the deviation from the mean.

FILM	METACRITIC SCORE	AVERAGE	CENTERED SCORE
Up in the Air (2009)	83	52.6	30.4
Apocalypse Now (1979)	90	52.6	37.4
Metropolis (1927)	98	52.6	45.4
Scent of a Woman (1993)	59	52.6	6.4
Mr. Holmes (2015)	67	52.6	14.4
Sideways (2005)	94	52.6	41.4
The Graduate (1967)	77	52.6	24.4
Taking Lives (2004)	38	52.6	-14.6
Dr. Strangelove (1964)	96	52.6	43.4
There Will Be Blood (2008)	92	52.6	39.4
Addicted (2014)	32	52.6	-20.6
Vampire Academy (2014)	30	52.6	-22.6
The Jackal (1997)	36	52.6	-16.6
Safe Haven (2013)	34	52.6	-18.6
The Last Song (2010)	33	52.6	-19.6
The Mortal Instruments: City of Bones (2013)	33	52.6	-19.6
Percy Jackson: Sea of Monsters (2013)	39	52.6	-13.6
Left Behind (2014)	12	52.6	-40.6
A Walk to Remember (2002)	35	52.6	-17.6
Harry Potter and the Deathly Hallows: Part 1 (2010)	65	52.6	12.4
Divergent (2014)	48	52.6	-4.6
Unbroken (2014)	59	52.6	6.4
The Book Thief (2013)	53	52.6	0.4
Mortdecai (2015)	27	52.6	-25.6
Where the Heart Is (2000)	30	52.6	-22.6
Pay It Forward (2000)	40	52.6	-12.6
The Three Musketeers (2011)	35	52.6	-17.6
Seventh Son (2015)	30	52.6	-22.6
Ender`s Game (2013)	51	52.6	-1.6
The Help (2011)	62	52.6	9.4

Finally, we scale the deviations from the mean in terms of standard deviation.

FILM	METACRITIC SCORE	AVERAGE	CENTERED SCORE	STD. DEV.	Z-SCORE METACRITIC
Up in the Air (2009)	83	52.6	30.4	24.57	1.24
Apocalypse Now (1979)	90	52.6	37.4	24.57	1.52
Metropolis (1927)	98	52.6	45.4	24.57	1.85
Scent of a Woman (1993)	59	52.6	6.4	24.57	0.26
Mr. Holmes (2015)	67	52.6	14.4	24.57	0.59
Sideways (2005)	94	52.6	41.4	24.57	1.68
The Graduate (1967)	77	52.6	24.4	24.57	0.99
Taking Lives (2004)	38	52.6	-14.6	24.57	-0.59
Dr. Strangelove (1964)	96	52.6	43.4	24.57	1.77
There Will Be Blood (2008)	92	52.6	39.4	24.57	1.60
Addicted (2014)	32	52.6	-20.6	24.57	-0.84
Vampire Academy (2014)	30	52.6	-22.6	24.57	-0.92
The Jackal (1997)	36	52.6	-16.6	24.57	-0.68
Safe Haven (2013)	34	52.6	-18.6	24.57	-0.76
The Last Song (2010)	33	52.6	-19.6	24.57	-0.80
The Mortal Instruments: City of Bones (2013)	33	52.6	-19.6	24.57	-0.80
Percy Jackson: Sea of Monsters (2013)	39	52.6	-13.6	24.57	-0.55
Left Behind (2014)	12	52.6	-40.6	24.57	-1.65
A Walk to Remember (2002)	35	52.6	-17.6	24.57	-0.72
Harry Potter and the Deathly Hallows: Part 1 (2010)	65	52.6	12.4	24.57	0.50
Divergent (2014)	48	52.6	-4.6	24.57	-0.19
Unbroken (2014)	59	52.6	6.4	24.57	0.26
The Book Thief (2013)	53	52.6	0.4	24.57	0.02
Mortdecai (2015)	27	52.6	-25.6	24.57	-1.04
Where the Heart Is (2000)	30	52.6	-22.6	24.57	-0.92
Pay It Forward (2000)	40	52.6	-12.6	24.57	-0.51
The Three Musketeers (2011)	35	52.6	-17.6	24.57	-0.72
Seventh Son (2015)	30	52.6	-22.6	24.57	-0.92
Ender`s Game (2013)	51	52.6	-1.6	24.57	-0.07
The Help (2011)	62	52.6	9.4	24.57	0.38

We can do the same thing for the book ratings;

FILM	GOODREADS RATING	AVERAGE	CENTERED SCORE	STD. DEV.	Z-SCORE GOODREADS
Up in the Air (2009)	2.9	3.91	-1.01	0.48	-2.11
Apocalypse Now (1979)	3.4	3.91	-0.51	0.48	-1.06
Metropolis (1927)	3.6	3.91	-0.31	0.48	-0.64
Scent of a Woman (1993)	3.0	3.91	-0.91	0.48	-1.90
Mr. Holmes (2015)	3.2	3.91	-0.71	0.48	-1.48
Sideways (2005)	3.7	3.91	-0.21	0.48	-0.43
The Graduate (1967)	3.4	3.91	-0.51	0.48	-1.06
Taking Lives (2004)	2.8	3.91	-1.11	0.48	-2.32
Dr. Strangelove (1964)	3.7	3.91	-0.21	0.48	-0.43
There Will Be Blood (2008)	3.7	3.91	-0.21	0.48	-0.43
Addicted (2014)	4.3	3.91	0.39	0.48	0.82
Vampire Academy (2014)	4.2	3.91	0.29	0.48	0.61
The Jackal (1997)	4.2	3.91	0.29	0.48	0.61
Safe Haven (2013)	4.2	3.91	0.29	0.48	0.61
The Last Song (2010)	4.1	3.91	0.19	0.48	0.40
The Mortal Instruments: City of Bones (2013)	4.1	3.91	0.19	0.48	0.40
Percy Jackson: Sea of Monsters (2013)	4.2	3.91	0.29	0.48	0.61
Left Behind (2014)	3.8	3.91	-0.11	0.48	-0.22
A Walk to Remember (2002)	4.1	3.91	0.19	0.48	0.40
Harry Potter and the Deathly Hallows: Part 1 (2010)	4.6	3.91	0.69	0.48	1.45
Divergent (2014)	4.3	3.91	0.39	0.48	0.82
Unbroken (2014)	4.5	3.91	0.59	0.48	1.24
The Book Thief (2013)	4.4	3.91	0.49	0.48	1.03
Mortdecai (2015)	4.0	3.91	0.09	0.48	0.20
Where the Heart Is (2000)	4.0	3.91	0.09	0.48	0.20
Pay It Forward (2000)	4.1	3.91	0.19	0.48	0.40
The Three Musketeers (2011)	4.0	3.91	0.09	0.48	0.20
Seventh Son (2015)	4.0	3.91	0.09	0.48	0.20
Ender`s Game (2013)	4.3	3.91	0.39	0.48	0.82
The Help (2011)	4.4	3.91	0.49	0.48	1.03

Now, we can compare the Z-Scores:

FILM	Z-SCORE METACRITIC	Z-SCORE GOODREADS
Up in the Air (2009)	1.24	-2.11
Apocalypse Now (1979)	1.52	-1.06
Metropolis (1927)	1.85	-0.64
Scent of a Woman (1993)	0.26	-1.90
Mr. Holmes (2015)	0.59	-1.48
Sideways (2005)	1.68	-0.43
The Graduate (1967)	0.99	-1.06
Taking Lives (2004)	-0.59	-2.32
Dr. Strangelove (1964)	1.77	-0.43
There Will Be Blood (2008)	1.60	-0.43
Addicted (2014)	-0.84	0.82
Vampire Academy (2014)	-0.92	0.61
The Jackal (1997)	-0.68	0.61
Safe Haven (2013)	-0.76	0.61
The Last Song (2010)	-0.80	0.40
The Mortal Instruments: City of Bones (2013)	-0.80	0.40
Percy Jackson: Sea of Monsters (2013)	-0.55	0.61
Left Behind (2014)	-1.65	-0.22
A Walk to Remember (2002)	-0.72	0.40
Harry Potter and the Deathly Hallows: Part 1 (2010)	0.50	1.45
Divergent (2014)	-0.19	0.82
Unbroken (2014)	0.26	1.24
The Book Thief (2013)	0.02	1.03
Mortdecai (2015)	-1.04	0.20
Where the Heart Is (2000)	-0.92	0.20
Pay It Forward (2000)	-0.51	0.40
The Three Musketeers (2011)	-0.72	0.20
Seventh Son (2015)	-0.92	0.20
Ender`s Game (2013)	-0.07	0.82
The Help (2011)	0.38	1.03

Note that these values don’t match the ones in the article, as we only used a subset of the data here.

References

Walt Hickey, Five Thirdy Eight: The 20 Most Extreme Cases of ‘The Book Was Better Than The Movie’, 2015