Fall 2025 HDFS 5349 Quantitative Methods I

Welcome to QM I, the department's introductory statistics class. I'm a bit unusual in using a blog to organize class materials, but I think it's worked well over the years. Also, for those of you who haven't been in school here, welcome to Texas Tech! You'll be visiting this welcoming page a lot, as it contains the links for our lecture notes.

I'll do my best to provide a lot of practical, real-world exercises in analyzing data, and I'll try to keep things fun. This passage from a book I read several years ago, Coincidences, Chaos, and All That Math Jazz, by Edward B. Burger and Michael Starbird, provides a concise overview of what statistics can offer:

Statistics can help us understand the world. It is a powerful and effective tool for placing economic, social welfare, sports, and health issues into perspective. It molds data into digestible morsels and shows us a measured way to look at situations that have either random or unknown features. But we must use common sense when applying statistics or other tools that draw on our experience of the world to shape data into meaningful conclusions (p. 60).

In addition, the following article sets forth some goals for what you should learn in this class (and other classes). We can access this article via the Texas Tech Library website or Google Scholar.

Utts, J. (2003). What educated citizens should know about statistics and probability. American Statistician, 57(2), 74-79.

LECTURE NOTES (asterisked [*] pages are from my undergraduate research-methods class).

Units of analysis*

Sampling*

Types of Measures*

Visual depictions of a data distribution:

• Histograms (overview). King and colleagues (2018) offer some advice on interval widths and the appearance of histograms, noting that "it is customary to make the height of the distribution about  three-quarters of the width" (pp. 32-33). To adjust the width of the columns, click on histogram in your output, then go to "Options," "Un-bin Element," and "Bin Element," then in small square in upper-right of screen, under "X Axis," select "Custom," and then enter desired interval width.
• Frequency tables contain similar information to histograms. The cumulative percentages also are roughly similar to percentiles (for a given score, you can see what percent of the sample falls below it).
• Shapes of distributions
• As a class exercise, we will attempt to reproduce via SPSS this histogram of U.S. Presidents' ages upon assuming office (note that presidents who serve two non-consecutive terms are counted as being "two presidents").

Descriptive statistics:* Central tendency (mean, median, and mode) and spread (standard deviation); moments of a distribution; and z-scores (here, here, here, and here)

Probability (here and here)

Correlation and significance-testing

t-tests

Chi-square

Non-parametric statistics

Statistical power

Confidence intervals

Big data

Writing Up Statistical Results in APA Style

Big Data

(Cross-posted and modified from several years ago at Reifman Multivariate Blog) 

The amount of capturable data people generate is truly mind-boggling, from commerce, health care, law enforcement, sports, and other domains. For example, according to one article, "Walmart controls more than 1 million customer transactions every hour..." The study of Big Data (also known as "Data Mining") applies statistical techniques such as correlation to discern patterns in the data and make predictions. Let's start with a brief video, listing numerous examples. Overview articles are available in the Harvard Business Review and on the Wikipedia (plus many other places I'm sure you can find via a Google search). A critique of the approach is available here. 

Perhaps the most common use of Big Data is in the business world. A good entry point in this area is the book Supercrunchers by Ian Ayres. Probably the best-known story about Big Data is "How Target Figured Out A Teen Girl Was Pregnant Before Her Father Did." What Target's statisticians did, essentially, was look for associations of whether customers' names were on the baby registry and whether they made greater purchases of certain products than the average person. Another potential business usage could come from this data-visualization of taxi and Uber trips in New York City (compare to layout of the city).

In the sports domain, I recommend three books pertaining to baseball: Moneyball (by Michael Lewis) on the Oakland Athletics; The Extra 2% (by Jonah Keri) on the Tampa Bay Rays; and Big Data Baseball (by Travis Sawchik) on the Pittsburgh Pirates. These three teams play in relatively small cities by Major League Baseball standards and thus bring in less local-television revenue than teams in bigger cities such as New York and Los Angeles. Therefore, teams such as Oakland, Tampa Bay, and Pittsburgh must use statistical techniques to (a) discover "deceptively good" players, whose skills are not well-known to other teams and who can thus be paid non-exorbitant salaries, and (b) identify effective strategies that other teams aren't using (yet). The Pirates' signature strategy, now copied by other teams, is the defensive shift, using statistical "spray charts" unique to each batter. (Major League Baseball later passed rules to limit shifting.) 

One final book I would recommend is The Victory Lab by Sasha Issenberg on political campaigns, namely how candidates try to convince people to vote for them and get their supporters to actually vote. 

To end the course, here's one final song...

Big Data 

Lyrics by Alan Reifman 

May be sung to the tune of “Green Earrings” (Fagen/Becker for Steely Dan) 

Sales, patterns, 

Companies try, 

Running equations, 

To predict, what we’ll buy, 

Big data, 

Lots of numbers, 

Floating in, the cloud, 

For computers, 

To analyze, now, 

We know, how, 

Sports, owners, 

Wanting to win, 

Seeking, advantage, 

In the numbers, it’s no sin, 

Big data, 

Lots of numbers, 

Floating in, the cloud, 

For computers, 

To analyze, now, 

We know, how 

Instrumentals/solos 

Big data, 

Lots of numbers, 

Floating in, the cloud, 

For computers, 

To analyze, now, 

We know, how 

Instrumentals/solos

Formula for a Paired t-Test

A paired t-test incorporates the possibility that the two variables whose means are being compared may also be correlated. In the following example, participating back-pain patients receive actual medication for one stretch of, say, six weeks and placebo (sugar pills) for a different stretch of six weeks. Ideally, the patients and medical staff who directly provide the pills to the patients would not know what kind of pill was being provided (i.e., a double-blind design) and the order of delivery -- medication then placebo or placebo then medication -- would be varied at random.

The focus of the paired t-test is, of course, whether participants' average reported pain while taking actual medication differs from their average reported pain under placebo. However, because patients with the most severe initial pain might report relatively high pain under medication and even higher pain under placebo, whereas patients with the mildest initial pain might report relatively low pain under placebo and even lower pain while receiving medication, patients' pain reports under medication and placebo could be positively correlated (see following graphic).

As shown in the following screenshot from this University of Georgia webpage, the correlation r between the two variables X and Y (highlighted) enters the paired t-test formula for comparing means.

Another document goes into additional depth regarding the paired/correlatedt-test, including implications of the correlation "r" being included in the formulation. As it notes, a larger correlation between the two variables will increase the size of the (absolute) t.

I've also created a graphic to interpret the SPSS output of a paired t-test, emphasizing the t-test comparison of means but also showing where the correlation between the two variables appears.

Writing Up Statistical Results in APA Style

Our colleague Dr. Shera Jackson has compiled the following list of web resources for how to write up statistical results in APA style:

Reporting Statistics in APA Style (Matthew Hesson-McInnis, Illinois State University)

Reporting Results of Common Statistical Tests in APA Format (Psychology Writing Center, University of Washington)

Statistics in APA Style (Craig Wendorf, University of Wisconsin-Stevens Point)

Statistical Power (Overview)

This week we'll be covering statistical power (also known as power analysis). Power is not a statistical technique like correlation, t-test, and chi-square. Rather, power involves designing your study (particularly getting a large enough sample size) so that you can use correlations, t-tests, etc., more effectively. The core concept of power, like so much else, goes back to the distinction between the population and a sample. When there truly is a basis in the population for rejecting the null hypothesis (e.g., a non-zero correlation, a non-zero difference between means), we want to increase the likelihood that we reject the null from the analysis of our sample. In other words, we want to be able to pronounce a result significant, when warranted. Here are links to my previous entries on statistical power.

Introductory lecture

Why a powerful design is needed: The population may truly have a non-zero correlation, for example, but due to random sampling error, your sample may not; plus, some songs on statistical power!

Remember that there's also the opposite kind of error: The population truly has absolutely no correlation, but again due to random sampling error, you draw a sample that gives the impression of a non-zero correlation.

How to plan a study using power considerations

Chi-Square

My introductory stat notes for methods class have some introductory information on chi-square.

Here are direct links to some old chi-square blog postings. This one discusses the reversibility error and how properly to read an SPSS printout of a chi-square analysis. The other one illustrates the null hypothesis for chi-square analyses in terms of equal pie-charts.

The following photo of the board, containing chi-square tips, was added on November 15, 2011 (thanks to Selen).

Plus a song (added November 1, 2011):

One Degree is Free

Lyrics by Alan Reifman

(May be sung to the tune of “Rock and Roll is Free,” Ben Harper)

Look at your, chi-square table,

If it is, 2-by-2,

One cell can be filled freely,

While the others take their cue,

The formula that you can use,

Come on, from the columns, lose one,

And one, as well, from the rows,

Multiply the two, isn’t this fun?

One degree is free, in your table,

With con-tin-gen-cy, in your table,

One degree is free, in your table,

…free in your table,

…free in your table,

Say, your table is larger,

Maybe it’s 2-by-4,

Multiply one by three,

3 df are in store,

The df’s are essential,

To check significance,

Go to your chi-square table,

And find the right instance,

Three degrees are free, in your table,

With con-tin-gen-cy, in your table,

Three degrees are free, in your table,

…free in your table,

…free in your table,

(Guitar Solo)

Correlation

UPDATED 9/29/2025 

Our next topic is correlational analysis. There are four major areas to address:

1. A general introduction to correlation. Correlation refers to how two variables go together ("co-relate"). There are three main types of correlations:

Positive correlation: As one variable goes up, so does the other. They both follow the same pattern. Knowing where a person stands on one variable, you know roughly where he/she stands on the other (maximum = +1.0). 

• Example: The more hours one studies before a test, the higher the score he or she will likely get. 

Zero correlation: Knowing where a person stands on one variable tells us nothing about where he/she stands on the other. Someone who has a high score on one variable is equally likely to have a high or a low value on the other. 

• Example: A person’s number of sneezes per week is (probably) uncorrelated with the percent of a person’s shirts that are blue. 

Negative correlation: As one goes up, the other goes down. They follow an inverse pattern. Knowing where a person stands on one variable, you again know roughly where he/she stands on the other (minimum = -1.0). 

• Example: The higher the winter temperatures where one lives (e.g., Miami), the fewer the heavy jackets people buy. 

Graphical depictions of positive, zero, and negative correlations. Note that the correlation (symbolized r) is based upon the slope of the best-fitting line (line which comes closest to all the points) and degree to which points are close to the line vs. being scattered. 

A song to nail down our understanding of correlation, best-fit lines, upward and downward slopes, etc. 

Fitting the Line 

Lyrics by Alan Reifman 

(May be sung to the tune of “Draggin’ the Line,” James/King) 

Dr. Reifman meeting singer Tommy James, after the latter's concert at the 2013 South Plains Fair. 

Plotting the data, on X and Y, 

Finding the slope, with most points nearby, 

We want to find the angle, of the trend’s incline, 

Fitting the line (fitting the line), 

Upward slopes make r positive, 

Slopes trending down, make it negative, 

From minus-one to plus-one, r can feel fine, 

Fitting the line (fitting the line), 

Fitting the line (fitting the line), 

Points align, how will the data shine? 

If you have upward slopes, it’ll give you a plus sign, 

Fitting the line (fitting the line), 

Fitting the line (fitting the line), 

How strongly will your variables relate? 

Is there a trend, or just a zero flat state? 

You want to know what your analysis will find, 

Fitting the line (fitting the line), 

Fitting the line (fitting the line), 

Points align, how will the data shine? 

Your r will be minus, if the slope declines, 

Fitting the line (fitting the line), 

Fitting the line (fitting the line), 

(Guitar solo) 

Points align, how will the data shine? 

If you have upward slopes, it’ll give you a plus sign, 

Fitting the line (fitting the line), 

Fitting the line (fitting the line)… 

2. Running correlations in SPSS. This graphic of SPSS output tries to make clear that a sample correlation and its significance/probability level are two different things (although related to each other).

Second, in graphing the data points and best-fitting line, you start in "Graphs" and select "Scatter/Dot." Then, select "Simple Scatter" and click on "Define." You will then insert the variables you want to display on the X and Y axes, and say "OK." When the scatter plot first appears, you can click on it to do more editing. To add the best-fit line, under "Elements," choose "Fit Line at Total."

Initially the dots will all look the same throughout the scatter plot. To make each dot represent the number of cases at that point (either by thickness of the dot or through color-coding), click on "Options" and then "Bin Element."

3. Statistical significance and testing the null hypothesis, as applied to correlation. Subthemes within this topic include how sample size affects the ease of getting a statistically significant result (i.e., rejecting the null hypothesis of zero correlation in the full population), and one- vs. two-tailed significance. 

4. Partial correlation (i.e., the correlation between two variables, holding constant one or more "lurking" variables).

Here are some additional tips:

5. In evaluating the meaning of a correlation that appears as positive or negative in the SPSS output, you must know how each of the variables is keyed (i.e., does a high score reflect more of the behavior or less of the behavior?).

6. Statistical significance is not necessarily indicative of social importance. With really large sample sizes (such as we have available in the GSS), even a correlation that seems only modestly different from zero may be statistically significant. To remedy this situation, the late statistician Jacob Cohen devised criteria for "small," "medium," and "large" correlations.

7. Correlations should also be interpreted in the context of range restriction. Here's a song to reinforce the ideas:

Restriction in the Range 

Lyrics by Alan Reifman

(May be sung to the tune of “Laughter in the Rain,” Sedaka/Cody)

Why do you get such a small correlation,
With variables you think should be related?
Seems you’re not studying the full human spectrum,
Just looking at part of bivariate space,
All kinds of thoughts start to race, through your mind…

Ooh, there’s restriction in the range,
Dampening the slope of the best-fit line,
Ooh, I can correct r for this,
Put a better rho estimate in its place...

Further t-Test Info (SPSS Output, Independent Samples)

(Updated October 26, 2014)

I have just created a new graphic on how to interpret SPSS print-outs for the Independent-Samples t-test (where any given participant is in only one of two mutually exclusive groups).


This new chart supplements the t-test lecture notes I showed recently, reflecting a change in my thinking about what to take from the SPSS print-outs.

One of the traditional assumptions of an Independent-Samples t-test is that, before we can test whether the difference between the two groups' means on the dependent variable is significant (which is of primary interest), we must verify that the groups have similar variances (standard-deviations squared) on the DV. This assumption, which is known as homoscedasticity, basically says that we want some comparability to the two groups' distributions in terms of their being equally spread out, before we can compare their means (you might think of this as "outlier protection insurance," although I don't know if this is technically the correct characterization of the problem).

If the homoscedasticity (equal-spread) assumption is violated, all is not lost. SPSS provides a test for possible violation of this assumption (the Levene's test) and, if violated, an alternative solution to use for the t-test. The alternative t-test (known as the Welch t-test or t') corrects for violations of the equal-spread assumption by "penalizing" the researcher with a reduction of degrees of freedom. Fewer degrees of freedom, of course, make it harder to achieve a statistically significant result, because the threshold t-value to attain significance is higher.

Years ago, I created a graphic for how to interpret the Levene's test and implement the proper t-test solution (i.e., the one for equal variances or for unequal variances, as appropriate). Even with the graphic, however, students still found the output confusing. Stemming from these student difficulties and some literature of which I have become aware, I have changed my opinion.

I now subscribe to the opinion of Glass and Hopkins (1996) that, “We prefer the t’ in all situations” (p. 305, footnote 30). Always using the t-test solution for when the two groups are assumed to have unequal spread (as depicted in the top graphic) is advantageous for a few reasons.

It is simpler to always use one solution than go through what many students find to be a cumbersome process for selecting which solution to use. Also, despite the two solutions (assuming equal spread and not assuming equal spread) being different and having different formulas in part, the bottom-line conclusion one draws (e.g., that men drink significantly more frequently than do women) often is the same under both solutions. If anything, the preferred (not assuming equal spread) solution is a little more conservative; in other words, it makes it a little harder to obtain a significant difference between means than does the equal-spread solution. As a result, our findings will have to be a little stronger for us to claim significance, which is not a bad thing.

Reference

Glass, G. V., & Hopkins, K. D. (1996). Statistical methods in psychology and education (3rd ed.). Needham Heights, MA: Allyn & Bacon.

Significance Testing for Correlations

UPDATED 10/15/24

This website nicely illustrates one- vs. two-tailed significance tests.

Here's a photo of the board from a long ago class, covering correlation and significance testing (thanks to Kristina for the photo). I've annotated the information with some additional clarification.

z-Scores and Percentiles

(Updated September 17, 2025)

If a body of data is normally distributed (i.e., follows the bell-shaped curve), we can convert an individual's z-score on a given variable into a percentile. A percentile refers to the percentage of sample members an individual stands above on the variable.

If we go to this website and look at the bottom diagram, we can see how z-scores translate into percentiles (again, as long as the distribution is normal). For instance, we can see that a z-score of +1 (shown along the horizontal axis as 1 sigma) places someone at roughly the 84th percentile. Fifty percent of the sample lies below mu, and another 34.1% lie between mu and mu +1 sigma, thus adding up to 84.1.

On this webpage, you can see how z-scores and percentiles correspond to areas under the normal curve. There is also a website where you can simply type in a z-score and get the corresponding percentile, and one on which you can drag the z-score horizontally and see the percentage under the normal curve.

This photo of the board from a recent class meeting (thanks to Kristina) summarizes some of the major properties of z-scores.

Confidence Interval (CI) Calculation

Updated November 23, 2015

The general form for calculating CI's is:

95% CI = Sample estimate +/- (1.96) (Standard Error)
............     (e.g., r or Mean)

The specific forms of this calculation for CI's around a mean, a correlation, and a proportion, respectively, are shown here, here, and here. This document (specifically Figures 7 and 8) explains why a step know as the Fisher z transformation must be implemented in finding the CI for a correlation. Because calculating the CI of a correlation is somewhat complicated, you may wish to use this online calculator for doing so.

Note how increasing one's sample size (N) will shrink the SE and hence, the CI.

Also, here's a potentially useful article:

Kalinowski, P., & Fidler, F. (2010). Interpreting ‘significance’: The difference between statistical and practical importance. Newborn and Infant Nursing Review, 10, 50-54.

Finally, I also have written a new song:

True Value
Lyrics by Alan Reifman
(May be sung to the tune of “Moon Shadow,” Cat Stevens; the song has also been recorded by real musicians, as commissioned by the Consortium for the Advancement of Undergraduate Statistics Education or "CAUSE")

Within your CI, you get the true value, true value, true value,
With 95%, you get the true value, true value, true value,

You get a sample statistic, a sample r, or sample M,
You then take plus-or-minus two (it’s really 1.96…), standard errors beyond your stat,
And within this new interval, we can be, so confident,
That the true value, mu or rho, will be somewhere… inside…, our confidence interval,

Within your CI, you get the true value, true value, true value,
With 95%, you get the true value, true value, true value...

t-Test Overview

(Updated October 11, 2025)

We will now be covering t-tests (for comparing the means of two groups) for the next week or so. As we'll discuss, there are two ways to design studies for a t-test:

INDEPENDENT SAMPLES, where a participant in one group (e.g., Trump voters in the 2024 election) cannot be in the other group (Harris voters). The technical term is that the groups are "mutually exclusive." The Trump and Harris voters could be compared, for example, on their average income.

PAIRED/CORRELATED GROUPS, where the same (or matched) person(s) can serve in both groups. For example, the same participant could be asked to complete math problems both during a period where loud hard-rock music is played and during a period where quiet, soothing music is played. Or, if you were comparing men and women on some attitude measure and your participants were heterosexual married couples, that would be considered a correlated design.

The Naked Statistics book briefly discusses the formula for an independent-samples t-test on pp. 164-165. Here's a simplified graphic I found from the web (original source):

Notice from the "Xbar1 - Xbar2" portion that the t statistic is gauging the amount of difference between the two means, in the context of the respective groups' standard deviations (s) and sample sizes (n). Your obtained t value will be compared to the t distribution (which is similar to the normal z distribution) to see if the t value is extreme enough to be unlikely to stem from chance. You will also need to take account of "degrees of freedom," which for an independent-samples t-test are closely based on total sample size.

There's an online graphic that visually illustrates the difference between z (normal) and t distributions (link). As noted on this page from Columbia University, "tails of the t-distribution are thicker and extend out further than those of the Z distribution. This indicates that for a given confidence level, t-scores [needed for significance] are larger than Z scores." (Remember our earlier term for distributions' tendency to have thick tails and generate outliers... kurtosis.)

More technically, as Westfall and Henning (2013) point out, "Compared to the standard normal distribution, the t-distribution has the same median (0.0) but with variance df/(df-2), which is larger than the standard normal's variance of 1.0" (p. 423). Remember that the variance is just the standard deviation squared.

In this table are shown values your obtained t statistic needs to exceed (known as "critical values") for statistical significance, depending on your df and target significance level (typically p < .05, two-tailed). 

This website provides a nice overview of one- and two-tailed tests. One-tailed tests are appropriate when there is a directional hypothesis (i.e., among students with no prior calculus instruction, those who receive calculus instruction during a summer workshop will score higher, on average, on a calculus post-test than will students who did not receive a summer calculus workshop, with the opposite prediction making no sense). Despite one-tailed tests seeming to be the best choice in some situations, however, two-tailed tests are nearly always used, presumably because they are more conservative (i.e., harder to obtain significance with). This 2024 article argues for greater use of one-tailed tests.

I have created a little tutorial on how to interpret SPSS output for independent-samples t-tests.

Later, we will take up the paired/correlated/dependent samples t-test at this link.

To end this part of the lesson, let's have a song! I have not written any lyrics for t-tests but, fortunately, Dr. Jeff Witmer of Oberlin College did in 2005 ("Use a t," which may be sung to the tune of "Let it Be," Lennon-McCartney). I attended the first-ever U.S. Conference on Teaching Statistics (USCOTS) in 2005 at The Ohio State University. When I walked into the opening reception of this meeting, Jeff was on stage singing "Use a t." It was my first exposure to statistical lyrics, which inspired me to write a few of my own over the years. 

The Consortium for the Advancement of Undergraduate Statistics Education (CAUSE), which sponsors the USCOTS meetings, several years ago commissioned musicians to record many statistical songs written by CAUSE/USCOTS participants, including "Use a t." Let's have a class sing-along! (click here and then search on Witmer). The references in the song to William Gosset and "Student" are explained in this article. When I saw Dr. Witmer at this past summer's (2025) 20th anniversary USCOTS meeting at Iowa State University, I absolutely had to ask for a selfie with him! Here it is... 

Hypothesis Testing with Correlations

NOTE: I have edited and reorganized some of my writings on correlation to present the information more coherently (10/11/2012).

The correlation statistic presents the first instance in which we'll be examining statistical significance (here and here). The question is whether we can reject the null hypothesis (Ho) that the correlation between a given pair of variables in the full population is zero (RHO = 0).

We, of course, obtain correlations (r) for our sample, and then see if our sample correlation is sufficiently different from zero (in either a positive or negative direction) so that it would have been sufficiently unlikely to have arisen from pure chance when the population RHO was truly zero. That's what we mean by statistical significance. When we achieve statistical significance, we can reject the Ho of zero RHO.

In order to have a statistically significant correlation, the correlation (r) itself should be appreciably different from zero, either above zero (a positive correlation) or below zero (a negative correlation).

Also, in order for the correlation to be significant, the significance (or probability) level displayed for a given correlation in your SPSS output must be very small (p < .05, or if the probability is even smaller, you can use one of the other conventional cut-off points, p < .01 or p < .001). Any time the probability p is larger than .05, the correlation is nonsignificant (in my opinion, if you get a correlation with a p level of .06 or .07, it's OK to note in your report that the correlation narrowly missed being significant under conventional standards).

Suppose you find that the correlation between two variables is r = .30, p < .01. This is telling us that, if the null hypothesis (Ho) is true -- that is, there truly is no correlation in the population from which the sample was drawn (rho = 0, where rho looks like a curvy capital P) -- then it would be extremely unlikely (p < .01) for a correlation of .30 to crop up purely by chance when the correlation throughout the society is truly zero.

[Here's a figure I've added in October 2007, to convey the idea of there truly being no correlation in a large population, but a correlation occurring in one's sample purely by random sampling error:


This web document is also helpful. The opposite problem, where the full population truly has a correlation, but you draw a sample that fails to show it, will be discussed later in the course.]

A significant correlation in our sample thus allows us to reject the null hypothesis and assert, based on an inference from our sample, that there is a correlation in the population. Again, note the inference from sample to population.

The essence of scientific hypothesis testing can thus be distilled to three steps:

1. State the null hypothesis (Ho) that there is no correlation between your two variables in the population (rho = 0). The investigator probably doesn't believe Ho (generally, we seek to uncover significant relationships), but Ho is part of the scientific protocol.

2. Obtain the sample correlation (r) between your two variables and the associated significance (p) level.

3. If the correlation is statistically significant (r is well above or well below zero, and p < .05), reject Ho. If the correlation is nonsignificant (r is close to zero and p is larger than .05), then the null hypothesis that there is no correlation between your two variables in the population must be maintained. We never accept the truth of the null hypothesis for certain; we just say it cannot be rejected.

I've stated above that, in order to be significant, a correlation (r) needed to be well above or well below zero. That's not always true, however. As we saw in some of our SPSS illustrations, with a very large sample size (n = 1,000 or more), a correlation does not necessarily need to be that far from zero to be significant. If a correlation appears to be small, yet is listed in the output as being significant (probably only due to the large sample size), you can say the correlation was "significant, though weak." A Wikipedia document on correlation (which I've just added to the links section on the right) displays guidelines developed by the late statistician Jacob Cohen for labeling correlations as "small, medium, and large."

Two concepts we will be taking up later in the course, statistical power and confidence intervals, will elaborate upon the issue of small correlations sometimes being significant and, conversely, relatively large correlations not being significant.

Probability Paradoxes

(Updated September 29, 2014)

To close out our coverage of probability, let's look at three brain-teasers.

1. One is the famous "Birthday Paradox." Upon first learning that a group size of only 23 people is necessary for the probability to be .50 that two of the people will have the same birthday, most observers find this very counterintuitive. The Wikipedia's page on the topic may help clarify the key points. One of the approaches taken on the Wikipedia page uses the "n choose k" principle. Another approach elaborates on the "and/multiplication" principle. The probability of at least one pair of people having the same birthday is 1 minus the probability of no one having the same birthday. The latter can be thought of as the product of the following probabilities:

The first person definitely has his/her birthday on some day (1) times...

The second person having it on one of the other 364 days of the year (364/365) times...

The third person having his or hers on one of the remaining 363 days (363/365) times...

2. The second puzzle is the famous Monty Hall Problem (named after the host of the old game show "Let's Make a Deal"), which is described in detail here. To my mind, the clearest explanation of the surprising solution is that given by Leonard Mlodinow's book The Drunkard's Walk. Based on Mlodinow's writing, here's a diagram we created in class a few years ago (thanks to Kristina for taking the picture):


The basic idea is that scenarios in which switching helps occur twice as often as ones in which switching hurts. The Naked Statistics book has a mini-chapter devoted to the Monty Hall Problem. There are also various YouTube videos on the problem, such as this one.

3. Finally, the third brain-teaser involves winning the lottery twice. Statisticians emphasize the distinction between a particular, named individual winning twice (which had an estimated probability of 1-in-17 trillion in a New Jersey example) and the probability that someone, somewhere, sometime would win twice. The latter probability, because it takes into account the huge number of people who play the lottery and the frequency and volume of tickets sold, is estimated at something more like 1-in-30 (the exact calculations are not shown).

Based on the linked article, let's use the n-choose-k and multiplication/and rules to derive the 1-in-17 trillion probability for a particular individual.

Comparing the Olympic Swimming Times of Michael Phelps (2008) vs. Mark Spitz (1972) via z-Scores

I have now corroborated the results of our Michael Phelps/Mark Spitz Olympic swimming z-score class exercise, which I'll show below. This activity was inspired by an earlier study published in the Baseball Research Journal that used z-scores to compare home-run sluggers of different eras.

I shared the activity with two listserve discussion groups, those of the APA Division of Evaluation, Measurement, and Statistics and the Society for Personality and Social Psychology, offering to provide the raw data and documentation on how to conduct the exercise. I'm pleased to report that over 100 people have requested these materials to use in their own statistics classes. The materials can still be requested, via my faculty webpage (see link in the right-hand column). I framed the exercise as follows, in the documentation:

Michael Phelps, with eight gold medals in the 2008 Beijing Olympics (on top of six golds from the 2004 Athens games), and Mark Spitz, with seven gold medals in the 1972 Munich Olympics, are swimming’s two greatest champions.

The two swam many of the same events. Though the respective times by Phelps are several seconds faster than Spitz’s, the 36 years between 1972 and 2008 are a long time for improvements in training, technique, nutrition, and facilities. A statistic known as the z-score allows us to see which swimmer was more dominant relative to his contemporary peers.

Phelps and Spitz had three individual (non-relay) events in common, the 200-meter freestyle, 200-meter butterfly, and the 100-meter butterfly. Because I have a relatively small class and wanted to have groups of three or four students each work on a different segment of the data, we looked at only the first two of the aforementioned events. Two considerations to note are that (a) times were converted into total seconds to facilitate computations; and (b) where an athlete swam multiple races of the same event (i.e., heats, semifinals, and finals), his fastest time was used. Here are the results:

200 FREESTYLE

2008

Mean = 109.42 seconds
SD = 3.20
Phelps time = 102.96 seconds (1:42.96)
Phelps z = -2.02

[For non-statisticians who may be reading this, z = an individual's value minus the mean, with the difference then divided by the standard deviation. The latter represents how spread out the data are.]

1972

Mean = 120.33 seconds
SD = 4.57 seconds
Spitz time = 112.78 seconds (1:52.78)
Spitz z = -1.65

200 BUTTERFLY

2008

Mean = 117.85 seconds
SD = 3.01
Phelps time = 112.03 seconds (1:52.03)
Phelps z = -1.93

1972

Mean = 129.51 seconds
SD = 5.32
Spitz time = 120.70 seconds (2:00.70)
Spitz z = -1.66

Note that negatively signed z scores are a "good" thing, indicating by how much Phelps or Spitz was faster (i.e., consuming less time) than his respective competitors. As can be seen, Phelps was more dominating against the 2008 fields of his events, than Spitz was against the 1972 fields. It would also be interesting to look at the 100-meter freestyle, which of course, Phelps won by the narrowest of margins.

I thank Nancy Genero of Wellesley College, a fellow University of Michigan Ph.D., for sharing the results from her class; by comparing our respective data files for possible typographical errors, we were able to reconcile some minor differences. Also, as a technical note, an "outlier" swimmer who had a time of 2:33.75 in the 1972 200 freestyle (when the next slowest time was around 2:13) was excluded. An extreme value would have affected both the mean and SD, of course.

Unlike the above analyses, which used all competitors in an event (regardless of whether they reached the finals or even the semifinals), one could also look exclusively at the finals. To the extent that qualifying rules for the Olympics may have changed between 1972 and 2008, or that other factors were operative, the proportion of weak swimmers (in a world-class context) in the fields might have been different in the two Games, again possibly affecting the z-score results. University of Nevada Reno graduate student Irem Uz indeed analyzed only the finals, and these were his results:

Phelps 200 free z = -1.92
Spitz 200 free z = -1.34
Phelps 200 fly z = -1.62
Spitz 200 fly z = -2.01

Under this method, there's a little redemption for Spitz. Examining the results of Spitz's 200 fly win in 1972, his dominance is clear:

1. Mark Spitz 2:00.70 WR
2. Gary Hall 2:02.86
3. Robin Backhaus 2:03.23
4. Jorge Delgado, Jr. 2:04.60
5. Hans Faßnacht 2:04.69
6. András Hargitay 2:04.69
7. Hartmut Flöckner 2:05.34
8. Folkert Meeuw 2:05.57

The mean was roughly 2:04, putting Spitz 3.30 seconds faster than it. Meanwhile, the extremely tight clustering of the fourth- through eighth-place swimmers served to keep the overall SD small (1.63). The upshot is a very big z for Spitz.

ADDENDA

The Wall Street Journal's "Numbers Guy," Carl Bialik, provided some other types of Phelps-Spitz comparisons as this year's Olympics were going on.

The New York Times created an amazing slide show of graphics, showing how the swimming times of Phelps and Spitz stacked up against each other, and also how each fared against his respective competition.

Another blogger, Jeremy Yoder, independently came up with the idea to analyze z-scores for Phelps and Spitz. Yoder's results are different from the comparable analyses reported above, for some reason.

Here's a 2014 application of z-scores to golf.

Practical Issues in Power Analysis

Below, I've added a new chart, based on things we discussed in class. William Trochim's Research Methods Knowledge Base, in discussing statistical power, sample size, effect size, and significance level, notes that, "Given values for any three of these components, it is possible to compute the value of the fourth." The table I've created attempts to convey this fact in graphical form.


You'll notice the (*) notation by "S, M, L" in the chart. Those, of course, stand for small, medium, and large effect sizes. As we discussed in class, Jacob Cohen developed criteria for what magnitude of result constitutes small, medium, and large for correlational studies and those studies comparing means of two groups (t-test type studies, but t itself is not an indicator of effect size).

When planning a new study, naturally you cannot know what your effect size will be ahead of time. However, based on your reading of the research literature in your area of study, you should be able to get an idea of whether findings have tended to be small, medium, or large, which you can convert to the relevant values for r or Cohen's d. These, in turn, can be submitted to power-analysis computer programs and online calculators.

I try to err on the side of expecting a small effect size. This will have the effect of requiring me to obtain a large sample size, to be able to detect a small effect, which seems like good practice, anyway.

UPDATE 1: Westfall and Henning (2013) argue that post hoc power analysis, which is what the pink column depicts in the above table, is "useless and counterproductive" (p. 508).

UPDATE 2: Lakens (2022, full text) provides extensive practical advice on sample-size determination and power analysis, including the option of "sensitivity power analysis" when one's sample size is already fixed.

Illustration of a "Miss" in Hypothesis Testing (and Relevance for Power Analysis)

My previous blog notes on this topic are pretty extensive, so I'll just add a few more pieces of information (including a couple of songs).

As we've discussed, the conceptual framework underlying statistical power involves two different kinds of errors: rejecting the null (thus claiming a significant result) when the null hypothesis is really true in the population (known as a "false alarm"); and failing to reject the null when the true population correlation (rho) is actually different from zero (known as a "miss"). The latter is illustrated below:



And here are my two new power-related songs...

Everything’s Coming Up Asterisks
Lyrics by Alan Reifman (updated 11/18/2014)
(May be sung to the tune of “Everything’s Coming Up Roses,” from Gypsy, Styne/Sondheim)

We've got a scheme, to find p-values, baby.
Something you can use, baby.
But, is it a ruse? Maybe...

State the null! (SLOWLY),
Run the test!
See if H-oh should be, put to rest,

If p’s less, than oh-five,
Then H-oh cannot be, kept alive (SLOWLY),

With small n,
There’s a catch,
There could be findings, you will not snatch,

There’s a chance, you could miss,
Rejecting the null hypothesis… (SLOWLY),

(Bridge)
H-oh testing, how it’s always been done,
Some resisting, will anyone be desisting?

With large n,
You will find,
A problem, of the opposite kind,

Nearly all, you present,
Will be sig-nif-i-cant,

You must start to look more at effect size (SLOWLY),
’Cause, everything’s coming up asterisks, oh-one and oh-five! (SLOWLY)

Believe it or not, the above song has been cited in the social-scientific literature:

Find the Power
Lyrics by Alan Reifman
(May be sung or rapped to the tune of “Fight the Power,” Chuck D/Sadler/Shocklee/Shocklee, for Public Enemy)

People do their studies, without consideration,
If H-oh, can receive obliteration,
Is your sample large enough?
To find interesting stuff,

P-level and tails, for when H-oh fails,
Point-eight-oh’s the way to go,
That’s the kind of power,
That you’ve got to show,

Look into your mind,
For the effect, you think you’ll find,

Got to put this all together,
Got to build your study right,
Got to give yourself enough, statistical might,

You’ve got to get the sample you need,
Got to learn the way,
Find the power!

Find the power!
Get the sample you need!

Find the power!
Get the sample you need!

Non-Parametric/Assumption-Free Statistics

(Updated October 31, 2025)

This week, we'll be covering non-parametric (or assumption-free) statistical tests (brief overview). Parametric techniques, which include the correlation r and the t-test, refer to the use of sample statistics to estimate population parameters (e.g., rho, mu).* Thus far, we've come across a number of assumptions that technically are required to be met for doing parametric analyses, although in practice there's some leeway in meeting the assumptions.

Assumptions for parametric analyses are as follows (for further information, see here):

o Data for a given variable are normally distributed in the population.

o Equal-interval measurement.

o Random sampling is used.

o Homogeneity of variance between groups (for t-test).

One would generally opt for a non-parametric test when there's violation of one or more of the above assumptions and sample size is small. According to King, Rosopa, and Minium (2011), "...the problem of violation of assumptions is of great concern when sample size is small (< 25)" (p. 382). In other words, if assumptions are violated but sample size is large, you still may be able to use parametric techniques (example). The reason is something called the Central Limit Theory. I've annotated the following screenshot from Sabina's Stats Corner to show what the CLT does.

It is important first to distinguish between a frequency plot of raw data (which appears in the top row of Sabina's diagram) and something else known as a sampling distribution. A sampling distribution is what you get when you draw repeated random samples from a full population of individual persons (sometimes known as a "parent" population) and plot the means of all the samples you have drawn. Under the CLT, a frequency plot of these means (the sampling distribution) tends toward normality regardless of the parent shape. Further, the sampling distribution increasingly resembles a bell-curve shape as the size of the multiple samples increases. In essence, what the CLT does for us is get us back to normal distributions when the original parent population distribution is non-normal.

We'll be doing a neat demonstration with dice that conveys the role of large samples in salvaging data from the normal-distribution assumption, under the CLT.

Now that we've established that non-parametric statistics typically are used when one or more assumptions of parametrics statistics are violated and sample size is small, we can ask: What are some actual non-parametric statistical techniques?

To a large extent, the different non-parametric techniques represent analogues to parametric techniques. For example, the non-parametric Mann-Whitney U test (song below) is analogous to the parametric t-test, when comparing data from two independent groups, and the non-parametric Wilcoxon signed-ranks test is analogous to a repeated-measures t-test. This PowerPoint slideshow demonstrates these two non-parametric techniques corresponding to t-tests. As you'll see, non-parametric statistics operate on ranks (e.g., who has the highest score, the second highest, etc.) rather than original scores, which may have outliers or other problems. A fully worked-out example of the Mann-Whitney U test is available here.

The parametric Pearson correlation has the non-parametric analogue of a Spearman rank-order correlation. Let's work out an example involving the 2024-25 Miami Heat, a National Basketball Association team suggested by one of the students. The roster of one team gives us a small sample size of players, on whom we will correlate their annual salary with their career performance on a metric called Win Shares (i.e., how many wins are attributed to each player based on his points scored, rebounds, assists, etc.). Here are our data (as of November 1, 2024):

These data clearly have some outliers. Jimmy Butler is the highest-paid Heat player at $48.8 million per year and he is credited statistically with personally contributing 115.5 wins to his teams in his 14-year career. Other, younger players have smaller salaries (although still huge in layperson terms) and have had many fewer wins attributed to them. The ordinary Pearson correlation and Spearman rank-order correlations are shown at the bottom of this posting,** if you'd like to calculate them in suspense. Which do you think would be larger and why?

Finally, we'll close with our song...

Mann-Whitney U
Lyrics by Alan Reifman
(May be sung to the tune of “Suzie Q.,” Hawkins/Lewis, covered by John Fogerty)

Mann-Whitney U,
When your groups are two,
If your scaling’s suspect, and your cases are few,
Mann-Whitney U,

The cases are laid out,
Converted to rank scores,
You then add these up, done within each group,
Mann-Whitney U,

(Instrumental)

There is a formula,
That uses the summed ranks,
A distribution’s what you, compare the answer to,
Mann-Whitney U

---
*According to King, Rosopa, and Minium (2011), "Many people call chi-square a nonparametric test, but it does in fact assume the central limit theorem..." (p. 382).

**The Pearson correlation is r = .57 (p = .03), whereas the Spearman correlation is r_s = .40 (nonsignificant due to the small sample size).

Partial-Correlation Oddity

While grading the correlation assignments, I came across an interesting finding in one of the students' papers (we use the "GSS93 subset" practice data set in SPSS, and each student can select his or her own variables for analysis).

Among the variables selected by this one student were number of children and frequency of sex during the last year. At the bivariate, zero-order level, these two variables were correlated at r = -.102, p < .001 (n = 1,327).

The student then conducted a partial correlation, focusing on the same two variables, but this time controlling for age (the student used the four-category age variable, although a continuous age variable is also available). This partial correlation turned out to be r = .101, p < .001.

Having graded several dozen papers from this assignment over the years, my impression was that, at least among the variables chosen by my students from this data set, partialling out variables generally had little impact on the magnitude of correlation between the two focal variables. Granted, neither of the correlations in the present example are all that huge, but the changing of the correlation's sign from negative (zero-order) to positive (first-order partial), with each of the respective correlations significantly different from zero, was noteworthy in my mind.

Also of interest was that age had both a fairly substantial positive correlation with number of children (r = .437) and a comparably powerful negative correlation with frequency of sex (r = -.410).

To probe the difference between the zero-order and partial correlations between number of children and frequency of sex, I went back to the (PowerPoint) drawing board, and created a scatter plot, color coding for age (also, in scatter plots created in SPSS, a dot only appears to indicate the presence of at least one case at a given spot on the graph, not the number of cases, so I attempted to remedy that, too).

My plot is shown below (you can click to enlarge it). I added trend lines after studying the SPSS scatter plots to see where the lines would go. As can be seen, the full-sample trend is indeed of a negative correlation, whereas all the age-specific trends are positive. We'll discuss this further in class.

How Significance Cut-Offs for Correlations Vary by Sample Size

Here's a more elaborate diagram (click to enlarge) of what I started sketching on the board at yesterday's class. It shows that, with smaller sample sizes, larger (absolute) values of r (i.e., further away from zero) are needed to attain statistical significance, than is the case with larger samples. In other words, with smaller samples, it takes a stronger correlation (in a positive or negative direction) to reject the null hypothesis of no true correlation in the full population (rho = 0) and rule out (to the degree of certainty indicated by the p level) that the correlation in your sample (r) has arisen purely from chance.


As you can see, statisticians sometimes talk about sample sizes in terms of degrees of freedom (df). We'll discuss df more thoroughly later in the course in connection with other statistical techniques. For now, though, suffice it to say that for ordinary correlations, df and sample size (N) are very similar, with df = N - 2 (i.e., the sample size, minus the number of variables in the correlation).

For a partial correlation that controls (holds constant) one variable beyond the two main variables being correlated (a first-order partial), df = N - 3; for one that controls for two variables beyond the two main ones (a second-order partial), df = N - 4, etc.

This web document also has some useful information.