Research Methods in Psychology: Thinking Like a Psychological Scientist

Textbook Information

Research Methods in Psychology: Thinking Like a Psychological Scientist © 2026 by Nicolette P. Rickert, Virginia Wickline, Cassandra Baldwin, Daniel Gambacorta, Amy Hackney, Audrey Molter is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-sa/4.0/

For questions or feedback about this textbook, please contact the first author:

Nicolette P. Rickert, Associate Professor, Department of Psychology, Georgia Southern University, Statesboro, GA nrickert@georgiasouthern.edu

Acknowledgements

Thank you to Dr. Nikki Cannon-Rech for her assistance throughout this process. This work was made possible through the Round 28 Affordable Learning Georgia Affordable Materials Grant (Transformation Grant #796).

Module 1: Foundations of Psychological Research

Chapter 1: Thinking like a Psychological Scientist

In this chapter, you will examine how people acquire knowledge and why some sources of knowledge are more reliable than others. You will also explore the limits of human judgment, including the biases that can shape how we interpret information. Finally, you will learn how scientific thinking and scientific theories help psychologists develop more dependable explanations of behavior and why these skills matter even if you never plan to become a researcher.

How do we know what we know?

Take a minute to ponder some of what you know and how you acquired that knowledge. Perhaps you know that you should make your bed in the morning because your mother or father told you this is what you should do, perhaps you know that swans are white because all of the swans you have seen are white, perhaps you know that your friend is lying to you because she is acting strange and won’t look you in the eye, or perhaps you know the skincare products that will get you glass skin because your favorite influencer showed you on Instagram. But should we trust knowledge from these sources? The methods we use to acquire knowledge can be broken down into four categories each with its own strengths and weaknesses.

Intuition

The first method of knowing is intuition. When we use our intuition, we are relying on our guts, our emotions, and/or our instincts to guide us. Rather than examining facts or using rational thought, intuition involves believing what feels true. Imagine you are walking alone to your car at night and notice someone approaching in the distance. Before consciously analyzing the situation, you may experience an immediate feeling that the person is either safe or potentially threatening. This rapid judgment is an example of intuition. The problem with relying on intuition is that our intuitions can be wrong because they are driven by cognitive and motivational biases rather than logical reasoning or scientific evidence. For example, while any strange behavior you see from your friend may lead you to think she is lying to you, it may just be that she is trying to hold in a bit of gas or is preoccupied with some other issue that is irrelevant to you.

Authority

Perhaps one of the most common methods of acquiring knowledge is through authority. This method involves accepting new ideas because some authority figure states that they are true. These authorities include parents, the media, doctors, priests and other religious authorities, the government, professors, and even social media influencers. While in an ideal world we should be able to trust authority figures, history has taught us otherwise and many instances of atrocities against humanity are a consequence of people unquestioningly following unethical authority (e.g., Salem Witch Trials, Nazi War Crimes). On a more benign level, while your parents may have told you that you should make your bed in the morning, making your bed provides the warm damp environment in which mites thrive. Keeping the sheets open provides a less hospitable environment for mites. These examples illustrate that the problem with using authority to obtain knowledge is that they may be wrong, they may just be using their intuition to arrive at their conclusions, and they may have their own reasons to mislead you. Nevertheless, much of the information we acquire is through authority because we don’t have time to question and independently research every piece of knowledge we learn through authority. But we can learn to evaluate the credentials of authority figures, to evaluate the methods they used to arrive at their conclusions, and evaluate whether they have any reasons to mislead us.

Rationalism

Rationalism involves using logic and reasoning to acquire new knowledge. Using this method, premises are stated and logical rules are followed to arrive at sound conclusions. For instance, if I am given the premise that all swans are white and the premise that this is a swan then I can come to the rational conclusion that this swan is white without actually seeing the swan. The problem with this method is that if the premises are wrong or there is an error in logic then the conclusion will not be valid. For instance, the premise that all swans are white is incorrect; there are black swans in Australia. Also, unless formally trained in the rules of logic it is easy to make an error. Nevertheless, if the premises are correct and logical rules are followed appropriately then this is a sound means of acquiring knowledge.

Empiricism

Empiricism involves acquiring knowledge through observation and experience. Once again many of you may have believed that all swans are white because you have only ever seen white swans. For centuries people believed the world is flat because it appears to be flat. These examples and the many visual illusions that trick our senses illustrate the problems with relying on empiricism alone to derive knowledge. We are limited in what we can experience and observe and our senses can deceive us. Moreover, our prior experiences can alter the way we perceive events. Nevertheless, empiricism is at the heart of the scientific method. Science relies on observations. But not just any observations, science relies on structured observations which is known as systematic empiricism, a topic we’ll turn to later on.

The Limits of Human Judgment

Although intuition, authority, rationalism, and empiricism can provide valuable information, none of these ways of knowing is perfect. Humans are not objective observers of the world. Instead, our judgments are often influenced by cognitive biases, systematic patterns of thinking that can lead us to wrongful conclusions. Cognitive biases affect what we notice, how we interpret information, and how confident we are in our beliefs. Because these biases operate automatically, even intelligent and well-intentioned people are susceptible to them.

Four particularly important biases are the confirmation bias, the availability heuristic, the representativeness heuristic, and the bias blind spot. Understanding these biases helps explain why people sometimes reach incorrect conclusions and highlights the importance of scientific methods for evaluating information and claims.

The confirmation bias is the tendency to seek out, notice, and remember information that supports our existing beliefs while overlooking or discounting information that contradicts them. For example, if you believe that your professor is unfriendly, you may pay close attention to a few negative interactions while ignoring numerous friendly and supportive ones. As a result, your belief may become stronger even if it is inaccurate.

The availability heuristic occurs when people judge the likelihood of an event based on how easily examples come to mind. Because vivid, recent, or memorable events are easier to recall, they can exert an outsized influence on our judgments. For example, after hearing several news stories about shark sightings, a person may overestimate the likelihood of encountering a shark during their next trip to the beach despite the rarity of such events.

The representativeness heuristic occurs when people make judgments based on how closely a person, object, or event matches their existing expectations or stereotypes. For example, you may assume that your professors spend their free time reading books and engaging in intellectual conversations because those activities fit your mental image of a professor. Likewise, a healthcare provider may be slower to recognize symptoms of a heart attack in a woman than in a man if the provider's mental prototype of a heart attack patient is male.

Finally, people are susceptible to the bias blind spot, the tendency to recognize cognitive biases in other people while failing to see those same biases in themselves. Most individuals readily acknowledge that others can be influenced by stereotypes, selective attention, or faulty reasoning, yet believe that their own judgments are objective and unbiased. Ironically, the bias blind spot can make it more difficult for people to recognize and correct their own errors in thinking.

Pause and Reflect: Can you think of a time when you searched for information that confirmed something you already believed? How might confirmation bias have influenced your thinking?

Taken together, these biases illustrate an important limitation of human judgment: people are not always aware of the factors influencing their beliefs and decisions. Because intuition and everyday reasoning are vulnerable to systematic errors, psychologists rely on scientific methods designed to minimize the influence of bias and evaluate claims more objectively.

Why Science Matters

The scientific method is a process of systematically collecting and evaluating evidence to test ideas and answer questions. While scientists may use intuition, authority, rationalism, and empiricism to generate new ideas, they do not stop there. Scientists go a step further by using systematic empiricism to make careful observations under controlled conditions and rationalism to draw logical conclusions from those observations. While the scientific method is the most likely of all methods of knowing to produce valid knowledge, it is not without limitations. Scientific research often requires substantial time, effort, and resources, and not all questions can be answered scientifically. The scientific method is best suited for answering empirical questions—questions that can be addressed through observation and evidence. This textbook examines how psychologists use the scientific method to advance our understanding of human behavior and mental processes.

Scientific Versus Everyday Reasoning

Each day, people offer statements as if they are facts, such as, “It looks like rain today,” or, “Dogs are very loyal.” These conclusions represent hypotheses about the world: best guesses as to how the world works. Scientists also draw conclusions, claiming things like, “There is an 80% chance of rain today,” or, “Dogs tend to protect their human companions.” You’ll notice that the two examples of scientific claims use less certain language and are more likely to be associated with probabilities. Understanding the similarities and differences between scientific and everyday (non-scientific) statements is essential to our ability to accurately evaluate the trustworthiness of various claims.

Scientific and everyday reasoning both employ induction, drawing general conclusions from specific observations. For example, a person’s opinion that cramming for a test increases performance may be based on her memory of passing an exam after pulling an all-night study session. Similarly, a researcher’s conclusion against cramming might be based on studies comparing the test performances of people who studied the material in different ways (e.g., cramming versus study sessions spaced out over time). In these scenarios, both scientific and everyday conclusions are drawn from a limited sample of potential observations.

Science also uses deduction, the process of applying general principles or theories to generate specific predictions. For example, if a theory suggests that learning improves when study sessions are spaced out over time, a researcher might predict that students who spread out their studying will perform better on an exam than students who cram the night before. Deductive reasoning allows scientists to move from broad theories to specific hypotheses that can be tested empirically.

Although both everyday thinking and science use induction, science does not rely on induction alone. By combining inductive observations with deductive testing of predictions, science provides a more reliable way of evaluating competing explanations. Consider a student trying to decide how to prepare for an exam. One source of information encourages her to cram, while another suggests that spacing out her studying time is the best strategy. How should she decide which advice to follow? To make the best decision with the information at hand, we need to appreciate the differences between personal opinions and scientific statements, which requires an understanding of science and the nature of scientific reasoning.

Characteristics of Scientific Thinking

Psychological scientists differ from everyday thinkers not because they are smarter or less susceptible to bias, but because they use methods designed to minimize the influence of bias. Scientific thinking is characterized by skepticism, systematic observation, testable explanations, and a willingness to revise conclusions when new evidence becomes available. Rather than accepting claims at face value, scientists ask whether the available evidence supports the claim and whether alternative explanations have been ruled out.

One hallmark of scientific thinking is skepticism. Scientific skepticism does not mean believing that all news is fake news, rejecting every claim made by others, or assuming that new ideas are false. Instead, it means withholding judgment until sufficient evidence is available. Skeptical scientists remain open to new ideas while also requiring that those ideas be supported by reliable evidence. In this sense, skepticism is a healthy habit of questioning rather than doubting everything.

Scientific thinking also relies on systematic empiricism. Unlike everyday observations, scientific observations are made in a carefully organized and standardized manner. Researchers develop procedures for collecting data, measuring variables, and documenting their methods so that others can evaluate and replicate their work. Systematic observation helps reduce many of the errors that arise from relying solely on personal experience or anecdotal evidence.

Another defining characteristic of scientific thinking is that scientific claims must be testable. Researchers should be able to collect evidence that either supports or challenges their ideas. If no possible evidence could demonstrate that a claim is incorrect, then the claim cannot be evaluated scientifically. This principle, known as falsifiability, has become one of the defining characteristics of modern science and is discussed in more detail below.

Finally, scientific thinking is self-correcting. Scientific conclusions are not considered final truths. Instead, researchers continually evaluate new evidence, refine existing theories, and revise conclusions when warranted. As additional evidence accumulates over time, scientific understanding becomes increasingly accurate, even though individual studies may sometimes produce conflicting findings.

Together, these characteristics help explain why science is one of the most reliable methods humans have developed for acquiring knowledge. Although scientists are not immune to cognitive biases, the scientific process provides tools for recognizing and reducing their influence.

Falsifiability

Although each of these characteristics contributes to the strength of scientific thinking, one has had an especially profound influence on modern science: falsifiability. Proposed by philosopher Karl Popper, falsifiability provides a useful way of distinguishing scientific claims from claims that cannot be evaluated scientifically. Understanding falsifiability helps explain why scientists emphasize testable hypotheses and why the scientific method is designed to rule out incorrect explanations rather than simply accumulate evidence in favor of preferred beliefs.

In the early 20th century, Karl Popper (1902-1994) suggested that science can be distinguished from pseudoscience (or just everyday reasoning) because scientific claims are capable of being falsified. That is, a claim can be conceivably demonstrated to be untrue. For example, a person might claim that “all people are right handed.” This claim can be tested and—ultimately—thrown out because it can be shown to be false: There are people who are left-handed. An easy rule of thumb is to not get confused by the term “falsifiable” but to understand that—more or less—it means testable.

On the other hand, some claims cannot be tested and falsified. Imagine, for instance, that a magician claims that he can teach people to move objects with their minds. The trick, he explains, is to truly believe in one’s ability for it to work. When his students fail to budge chairs with their minds, the magician scolds, “Obviously, you don’t truly believe.” The magician’s claim does not qualify as falsifiable because there is no way to disprove it. It is unscientific.

Popper was particularly concerned about theories that could explain every possible outcome. He argued that if a theory can account for any result, it becomes difficult or impossible to test scientifically. As an example, Popper criticized some of Freud’s explanations of personality and mental illness. Imagine a person who grows up to be an obsessive perfectionist. If she were raised by messy, relaxed parents, Freud might argue that her adult perfectionism is a reaction to her early family experiences—an effort to maintain order and routine instead of chaos. Alternatively, imagine the same person being raised by harsh, orderly parents. In this case, Freud might argue that her adult tidiness is simply her internalizing her parents’ way of being.

As you can see, according to Freud’s rationale, both opposing scenarios are possible; no matter what the disorder, Freud’s theory could explain its childhood origin—thus failing to meet the principle of falsifiability. Popper argued that when a theory can explain multiple contradictory scenarios equally well, it becomes difficult to identify evidence that would disprove it, making the theory less scientifically useful.

Figure 1.1 Karl Popper^[1]

Popper argued that scientific progress comes not from proving ideas correct, but from eliminating explanations that do not fit the evidence. The explanations that survive repeated testing become the ones scientists have the greatest confidence in. In other words, scientists should be able to identify, before collecting data, what evidence would cause them to question or revise their hypothesis.

Test Yourself: Can It Be Falsified?

For each statement below, determine whether it is falsifiable. If it is, describe what evidence could demonstrate that the statement is false.

A. Chocolate tastes better than pasta.

B. We live in the most violent time in history.

C. Time can run backward as well as forward.

D. All swans are white.

[See answer at end of this module]

Although the idea of falsification remains central to scientific data and theory development, these days it’s not used strictly the way Popper originally envisioned it. To begin with, scientists aren’t solely interested in demonstrating what isn’t. Scientists are also interested in providing descriptions and explanations for the way things are. Whether researchers are investigating when children begin speaking in complete sentences, whether exercise reduces depression, or why people are happier on weekends, they must draw conclusions from limited samples of data. As a result, scientific conclusions are based on probability rather than absolute certainty. Evidence may support or challenge a hypothesis, but scientific knowledge always remains open to revision as new evidence accumulates.

Scientific Theories

Because scientific conclusions are based on evidence rather than absolute proof, scientific knowledge is organized into theories that explain observations and guide future research. In everyday conversation, people often use the word theory to mean a guess or speculation. For example, someone might say, "I have a theory about why my roommate is always late," or "My theory is that our team will win the championship." In these cases, the word theory refers to a personal opinion or educated guess.

In science, however, a theory is much more than a guess. A scientific theory is a well-supported explanation for a set of observations or findings. Scientific theories are based on extensive empirical evidence and are used to organize existing knowledge, explain why phenomena occur, and generate predictions about future observations.

Scientific theories differ from facts, but they are not opposed to facts. Facts are observations about the world, whereas theories explain those observations. For example, it is a fact that objects released near the Earth's surface fall toward the ground. The theory of gravity helps explain why this occurs. Similarly, psychologists develop theories to explain patterns of human thought, emotion, and behavior.

One of the strengths of science is that theories can change when new evidence emerges. For example, people once believed that the Sun revolved around the Earth. As astronomers collected more systematic observations, the evidence supported a different explanation: the Earth and other planets revolve around the Sun. Scientists view the willingness to revise theories in light of new evidence as a strength rather than a weakness.

Figure 1.2 Early theories placed the Earth at the center of the solar system.^[2]

Scientists often develop multiple theories to explain the same phenomenon. Not all scientific theories are equally useful. Scientists evaluate theories using several criteria, including their ability to accurately explain observations, remain internally consistent, account for a broad range of phenomena, provide simple explanations when appropriate, and stimulate new research. Table 1 summarizes these characteristics of strong scientific theories.

Table 1.1 Five features of good scientific theories (Kuhn, 2011^)[3]

Together, these characteristics help scientists evaluate competing theories. Strong theories not only explain existing evidence but also make accurate predictions, organize knowledge, and inspire new questions for researchers to investigate. Over time, theories that consistently perform well across these criteria become increasingly influential in guiding scientific research.

Pause and Reflect: Have you ever heard someone say, “It’s just a theory”? How has your understanding of the word theory changed after reading this chapter?

Psychological scientists rarely begin a study with no direction. Instead, they use existing theories, previous research, and observations of the world to identify unanswered questions and develop testable hypotheses. In the next chapter, we will examine how psychologists transform broad theories into specific research questions and hypotheses that can be investigated using the scientific method.

Why Understanding Psychological Science Matters

Although relatively few psychology students will become researchers, every student will become a consumer of research. Throughout your life, you will encounter claims about what causes depression, whether certain parenting styles are more effective than others, how to improve memory, which study strategies work best, or how social media affects mental health. News articles, podcasts, social media posts, advertisements, politicians, and influencers often present scientific findings in ways that are incomplete, misleading, or even inaccurate.

Understanding psychological science provides the tools to evaluate these claims critically. Rather than accepting information because it “sounds right” or because it comes from an authority figure, you will learn to ask important questions. What evidence supports the claim? Could there be an alternative explanation? Was the conclusion based on a single study or many studies? Are the findings consistent with existing scientific theories? By applying the principles of scientific reasoning, you can make more informed decisions in your personal life, your career, and as an engaged member of society.

The goal of this textbook is not simply to teach you how psychologists conduct research. It is to help you develop scientific thinking skills that you can use long after this course ends.

Answer Key: Test Yourself 1

A. Chocolate tastes better than pasta.
Not falsifiable (as written). "Tastes better" is a subjective preference that can differ from person to person. Because there is no objective criterion for determining which food "tastes better," the statement cannot be conclusively shown to be false.

B. We live in the most violent time in history.
Potentially falsifiable. Researchers could collect historical data on rates of violence across different time periods and compare them. Although obtaining complete historical data is challenging, evidence could support or contradict the claim.

C. Time can run backward as well as forward.
Potentially falsifiable. Scientists could search for evidence that time runs backward or make predictions that would be expected if it did. If repeated observations consistently fail to support the claim, confidence in the hypothesis decreases.

D. All swans are white.
Falsifiable. Finding a single swan that is not white would contradict the claim. In fact, black swans exist in Australia, demonstrating that the statement is false.

Media Attributions

^[1] Karl Popper. Photograph by Lucinda Douglas-Menzies. Source: Wikimedia Commons, https://commons.wikimedia.org/wiki/File:Karl_Popper2.jpg. Rights status: No known copyright restrictions (Flickr Commons; permission verified by Wikimedia VRT).

^[2] Ptolemaic (geocentric) model. Pearson Scott Foresman. Source: Wikimedia Commons, File:Ptolemaic system (PSF).png - Wikimedia Commons. License: Public domain.

^[3] Features of good scientific theories. Smith, E. I. (2026). Thinking like a psychological scientist. In R. Biswas-Diener & E. Diener (Eds), Noba textbook series: Psychology. Champaign, IL: DEF publishers. Retrieved from http://noba.to/nt3ysqcm

Text Attributions

Anguiano, R., & O’Neil, M. (n.d.). Methods of Knowing. In S.D’Costa, M.Ukeye, M. O’Neil, & R. Anguiano, Critical research methods in psychology. Saint Mary’s College of California. Licensed under CC BY-NC-SA 4.0. Modified by current authors.

Smith, E. I. (2026). Thinking like a psychological scientist. In R. Biswas-Diener & E. Diener (Eds), Noba textbook series: Psychology. DEF publishers. Retrieved from http://noba.to/nt3ysqcm. Modified by current authors.

Spielman, R. M., Jenkins, W. J., & Lovett, M. D. (2020). Problem Solving. In Psychology 2e. OpenStax, Rice University. Access for free at https://openstax.org/books/psychology-2e/pages/1-introduction Licensed under CC BY-NC-SA 4.0. Modified by current authors.

References

Kuhn, T. S. (2011). Objectivity, value judgment, and theory choice, in T. S. Kuhn (Ed.), The essential tension: Selected studies in scientific tradition and change (pp. 320-339). Chicago: University of Chicago Press. Retrieved from http://ebookcentral.proquest.com

Chapter 2: Generating Research Questions and Hypotheses

From Theories to Research Questions

In Chapter 1, we learned that scientific theories organize existing knowledge, explain observations, and generate predictions about future events. But theories alone do not produce scientific discoveries. To evaluate a theory, psychologists must design research that tests its predictions. Every research study begins with a question: What do we want to understand about behavior or mental processes? Research questions and hypotheses are closely related, but they are not the same thing. A research question asks what a researcher wants to understand, whereas a hypothesis is a specific prediction about what the researcher expects to find. Throughout this chapter, you will learn how psychologists move from broad theories to focused research questions and finally to hypotheses that can be tested using scientific methods.

Research Questions and Variables

Before we address where research questions in psychology come from—and what makes them more or less interesting—it is important to understand the kinds of questions psychologists ask and the variables they study. This requires a brief introduction to several basic concepts, many of which we will return to in more detail later in the book.

Research questions in psychology are about variables. A variable is a quantity or quality that varies across people or situations. In other words, a variable is any characteristic that differs across people or situations. For example, the height of the students in a psychology class is a variable because it varies from student to student. The sex of the students is also a variable as long as there are both male and female students in the class.

Variables are the building blocks of psychological research. Some research questions are descriptive and focus on a single variable, such as how common depression is in college students. Others are associational, asking whether two or more variables are related, such as whether sleep quality is associated with academic performance. Still others are causal, asking whether changes in one variable produce changes in another, such as whether a new type of jury instruction improves jurors’ understanding of the law. Understanding how psychologists ask these questions is the first step toward designing meaningful research. Different types of research questions require different research designs. Throughout this textbook, you’ll learn how psychologists choose the most appropriate design to answer each type of question.

In the next sections, we’ll examine where research questions come from, what makes some questions stronger than others, and how psychologists transform research questions into specific, testable hypotheses.

Where do Research Questions Come From?

Research questions rarely appear out of thin air. Instead, they often begin with curiosity about human (or other animal) behavior or mental processes. A researcher may notice an interesting pattern, encounter a practical problem, read about a previous study, or recognize that an existing scientific theory makes a prediction that has not yet been tested. These initial ideas are then refined into research questions that can be investigated scientifically.

Generating Good Research Questions

Finding Inspiration

Research questions often begin as more general research ideas—usually focusing on some behavior or psychological characteristic: talkativeness, memory, depression, jury verdicts, and so on. Before looking at how to turn such ideas into empirically testable research questions, it is worth looking at where such ideas come from in the first place. Four common sources of inspiration are informal observations, practical problems, previous research, and scientific theories.

Informal Observations

Informal observations include direct observations of our own and others’ behavior as well as secondhand observations from nonscientific sources such as newspapers, books, and so on. For example, you might notice that you always seem to be in the slowest moving line at the grocery store. Could it be that most people think the same thing? Or you might read in the local newspaper about people donating money and food to a local family whose house has burned down and begin to wonder about who makes such donations and why. Some of the most famous research in psychology has been inspired by informal observations. Stanley Milgram’s famous research on obedience, for example, was inspired in part by journalistic reports of the trials of accused Nazi war criminals—many of whom claimed that they were only obeying orders. This led him to wonder about the extent to which ordinary people will commit immoral acts simply because they are ordered to do so by an authority figure (Milgram, 1963).

Practical Problems

Practical problems can also inspire research ideas, leading directly to applied research in such domains as law, health, education, and sports. For example:

• Can human figure drawings help children remember details about being physically or sexually abused?
• How effective is psychotherapy for depression compared to drug therapy?
• Does using a cell phone impair driving performance?
• What study strategies produce the greatest long-term learning?
• What is the best mental preparation for running a marathon?

Questions like these often lead directly to applied psychological research.

Previous Research

Probably the most common inspiration for new research ideas, however, is previous research. Science is cumulative. Researchers read one another’s work, identify unanswered questions, and design new studies that extend existing knowledge. Of course, experienced researchers are familiar with previous research in their area of expertise and probably have a long list of ideas. This suggests that novice researchers can find inspiration by consulting with a more experienced researcher (e.g., students can consult a faculty member). But they can also find inspiration by picking up a copy of almost any professional journal and reading the titles and abstracts. In one issue of Psychological Science, for example, you can find articles on the perception of shapes, anti-Semitism, police lineups, the meaning of death, second-language learning, people who seek negative emotional experiences, and many other topics. If you can narrow your interests down to a particular topic (e.g., memory) or domain (e.g., health care), you can also look through more specific journals, such as Memory & Cognition or Health Psychology. Reading the discussion section of a published research article is often an excellent way to identify ideas for future research because authors typically describe limitations of their studies and suggest important directions for future investigation.

Scientific Theories

Scientific theories are another important source of research questions. As you learned in Chapter 1, theories organize existing knowledge and generate predictions about future observations. Researchers often begin by asking whether a prediction derived from a theory is supported by empirical evidence. If the prediction has not yet been tested, or if previous findings are inconsistent, it becomes an opportunity for new research.

For example, a theory suggesting that sleep improves memory might lead a researcher to ask whether students who obtain eight hours of sleep before an exam perform better than students who stay awake studying. In this way, theories provide a roadmap for future research by identifying questions that remain unanswered.

Regardless of where a research idea originates, it is only the starting point. Researchers must still transform broad ideas into focused, empirical questions that can be answered through scientific observation. In the next section, we’ll examine how psychologists develop research questions that are both scientifically meaningful and practically feasible.

Generating Empirically Testable Research Questions

Once you have a research idea, you need to use it to generate one or more empirically testable research questions, that is, questions expressed in terms of a single variable or relationship between variables. Once you have an idea, ask yourself:

• “What causes this behavior?”
• “What are the consequences of this behavior?”
• “Who is most likely to exhibit this behavior?”
• “Under what conditions does it occur?”

Table 2.1 Summary of Research Ideas and Research Questions

Pause and Reflect: Think of a topic that interests you, such as personality, social media, romantic relationships, sports, or music. What is one research question a psychologist could investigate about that topic?

Evaluating Research Questions

Researchers usually generate many more research questions than they can realistically study. This means they must have some way of evaluating the research questions they generate so that they can choose which ones to pursue. Two important criteria are whether the question is interesting and whether it is feasible to answer.

Is the Question Interesting?

How often do people tie their shoes? Do people feel pain when you punch them in the jaw? Are women more likely to wear makeup than men? Do people prefer vanilla or chocolate ice cream? Although it would be a fairly simple matter to design a study and collect data to answer these questions, you probably would not want to because they are not interesting. We are not talking here about whether a research question is interesting to us personally but whether it is interesting to people more generally and, especially, to the scientific community. But what makes a research question interesting in this sense? Here we look at three factors that affect the interestingness of a research question: the answer is in doubt, the answer fills a gap in the research literature, and the answer has important practical implications.

First, a research question is interesting to the extent that its answer is in doubt. Obviously, questions that have been answered by scientific research are no longer interesting as the subject of new empirical research. But the fact that a question has not been answered by scientific research does not necessarily make it interesting. There has to be some reasonable chance that the answer to the question will be something that we did not already know. But how can you assess this before actually collecting data? One approach is to try to think of reasons to expect different answers to the question—especially ones that seem to conflict with common sense. If you can think of reasons to expect at least two different answers, then the question might be interesting. If you can think of reasons to expect only one answer, then it probably is not. The question of whether women are more talkative than men is interesting because there are reasons to expect both answers. The existence of the stereotype itself suggests the answer could be yes, but the fact that women’s and men’s verbal abilities are fairly similar suggests the answer could be no. The question of whether people feel pain when you punch them in the jaw is not interesting because there is little reason to think that the answer could be anything other than a resounding yes.

A second important factor to consider when deciding if a research question is interesting is whether answering it will fill a gap in the research literature. Researchers often identify these gaps by reading published studies and noting the limitations or future directions suggested by the authors.

A final factor to consider when deciding whether a research question is interesting is whether its answer has important practical implications. Questions about effective treatments for depression, reducing distracted driving, improving eyewitness memory, or enhancing student learning have the potential to improve people’s lives.

Is the Question Feasible?

A second important criterion for evaluating research questions is the feasibility of successfully answering them. There are many factors that affect feasibility, including time, money, equipment and materials, technical knowledge and skill, ethical considerations, and access to research participants. Clearly, researchers need to take these factors into account so that they do not waste time and effort pursuing research that they cannot complete successfully.

Looking through a sample of professional journals in psychology will reveal many studies that are complicated and difficult to carry out. These include longitudinal designs in which participants are tracked over many years, neuroimaging studies in which participants’ brain activity is measured while they carry out various mental tasks, and complex non-experimental studies involving several variables and complicated statistical analyses. Keep in mind, though, that such research tends to be carried out by teams of highly trained researchers whose work is often supported in part by government and private grants. Keep in mind also that research does not have to be complicated or difficult to produce interesting and important results. Looking through a sample of professional journals will also reveal studies that are relatively simple and easy to carry out—perhaps involving a convenience sample of college students and a paper-and-pencil task.

A final point here is that it is generally good practice to use methods that have already been used successfully by other researchers. For example, if you want to manipulate people’s moods to make some of them happy, it would be a good idea to use one of the many approaches that have been used successfully by other researchers (e.g., paying them a compliment). This is good not only for the sake of feasibility—the approach is “tried and true”—but also because it provides greater continuity with previous research. This makes it easier to compare your results with those of other researchers and to understand the implications of their research for yours, and vice versa.

Thinking Like a Researcher

When evaluating a possible research question, ask yourself:

• Is the answer genuinely unknown?
• Does the question build on previous research or theory?
• Does it have scientific or practical importance?
• Can it be answered ethically?
• Do I have the time, resources, and participants needed to answer it?

If the answer to these questions is “yes,” then the research question is probably worth pursuing.

From Research Questions to Hypotheses

A good research question identifies what a psychologist wants to understand, but it does not specify what the researcher expects to find. To answer a research question scientifically, psychologists develop one or more hypotheses. A hypothesis is a specific, testable prediction about the relationship between variables or about the outcome of a study. Unlike a research question, which asks a question, a hypothesis proposes a possible answer that can be evaluated with empirical evidence.

For example, consider the following research question:

Does sleep affect academic performance?

This question identifies an important topic, but it does not predict what the researcher expects to observe. One possible hypothesis is:

College students who sleep between seven and nine hours the night before an exam will earn higher exam scores than students who sleep fewer than six hours.

Notice that this hypothesis is much more specific than the research question. It identifies the variables being studied, predicts the expected relationship between them, and can be tested through observation. Throughout the remainder of this chapter, you will learn how psychologists develop strong hypotheses that can be investigated using scientific research methods.

Characteristics of Good Hypotheses

Not all hypotheses are equally useful. A strong hypothesis provides a clear prediction that can be tested through scientific research. Well-written hypotheses help researchers design studies, select appropriate variables, and interpret their findings. Although there is no single formula to writing a hypothesis, effective hypotheses share several important characteristics.

A Good Hypothesis is Testable

A hypothesis must be capable of being evaluated using empirical evidence. Researchers should be able to collect observations or measurements that either support or do not support the prediction. If no possible evidence could demonstrate that a hypothesis is incorrect, it cannot be tested scientifically.

A Good Hypothesis is Specific

A useful hypothesis clearly identifies the variables being studied and predicts the expected relationship between them. Vague hypotheses make it difficult to determine what should be measured or how the results should be interpreted.

A Good Hypothesis is Falsifiable

As you learned in Chapter 1, scientific hypotheses must be capable of being shown to be incorrect if contradictory evidence exists. Research should be able to describe what findings would cause them to question or reject the hypothesis.

A Good Hypothesis is Grounded in Existing Knowledge

Researchers do not develop hypotheses at random. Most hypotheses are based on previous research, established scientific theories, or systematic observations. Building on existing knowledge helps ensure that research contributes meaningfully to scientific understanding.

A Good Hypothesis Makes a Prediction

Unlike a research question, which asks what the researcher wants to understand, a hypothesis predicts what the researcher expects to find. Although predictions are not always correct, they provide a clear expectation that can be evaluated through research.

When researchers write strong hypotheses, they create a clear roadmap for the rest of the research process. A well-written hypothesis helps determine what variables should be measured, what research design is appropriate, and how the results should be interpreted.

Although research hypotheses often propose a relationship, difference, or effect of one variable on another, a well-designed study can also test the prediction that there will be no difference or no relationship. For example, in clinical psychology, demonstrating that two different types of therapy provide equivalent mental health outcomes is an important scientific finding.

Table 2.2 Examples of Poor and Better Hypotheses

Pause and Reflect: Think back to your research question. Can you rewrite it as a specific, testable hypothesis?

Identifying Variables

Earlier in this chapter, you learned that research questions are built around variables - characteristics, behaviors, or conditions that can vary across people, situations, or time. Once a researcher has developed a hypothesis, the next step is to identify the variables it contains.

Consider the following hypothesis:

College students who sleep at least eight hours the night before an exam will earn higher exam scores than students who sleep fewer than six hours.

This hypothesis contains two variables: sleep duration and exam performance.

Identifying the variables is only the beginning. Researchers must also determine exactly how each variable will be measured or manipulated. For example, should sleep duration be measured using a sleep diary, a smartwatch, or participants’ self-reports? Should exam performance be measured using a classroom exam, a standardized test, or a laboratory memory task?

These questions illustrate an important feature of psychological research. Many concepts studied by psychologists, such as stress, intelligence, prejudice, motivation, and happiness, cannot be observed directly. Before these concepts can be studied scientifically, researchers must decide exactly how they will be measured or manipulated. These precise descriptions are called operational definitions, which are discussed in Module 2.

Types of Hypotheses

Not all hypotheses make the same kind of prediction. Some hypotheses predict the direction of a relationship, whereas others simply predict that a relationship or difference exists.

A directional hypothesis predicts both that a relationship exists and the direction of that relationship. For example:

College students who sleep at least eight hours the night before an exam will earn higher exam scores than students who sleep fewer than six hours.

This hypothesis predicts that students who sleep more will perform better on the exam.

A nondirectional hypothesis predicts that a relationship or difference exists but does not predict which direction the results will go. For example:

Sleep duration will be related to exam performance.

Researchers often use nondirectional hypotheses when previous research is limited or when there is insufficient evidence to predict the direction of the relationship.

Every research hypothesis also has a corresponding null hypothesis, which predicts that no relationship or difference exists between the variables. Researchers use null hypotheses when conducting statistical analyses, a topic that will be covered in a later module.

Independent and Dependent Variables

Once researchers have developed a hypothesis, they must identify the role of each variable. In experimental research, the variable that the researcher manipulates is called the independent variable (IV). The variable that is measured to determine whether it changes is called the dependent variable (DV).

Recall our hypothesis that:

College students who sleep at least eight hours the night before an exam will earn higher exam scores than students who sleep fewer than six hours.

In this example, the independent variable is sleep duration, and the dependent variable is exam score. A helpful way to remember the difference is to ask yourself, “Which variable is expected to influence the other?” Which variables is the “independent leader?” The variable leading in this example is the independent variables. The variable expected to follow, or change, is the dependent variable.

Remember that an independent variable is manipulated in experimental research. Thus in our example, a researcher could randomly assign some college students to sleep at least eight hours the night before an exam and randomly assign other college students to sleep fewer than six hours before an exam. Then the exam performances between the two groups of students would be compared.

Pause and Reflect: If you wanted to test whether listening to music while studying affects exam performance, what would be the independent variable? What would be the dependent variable?

Predictor and Outcome Variables

If you are thinking that it doesn’t sound very easy or even ethical to ask some students to get eight hours of sleep and others to get less before an exam, you’re right. Many times it doesn’t make sense to manipulate a variable of interest. In many psychological studies, researchers simply measure variables as they naturally occur. For example, researchers cannot ethically assign people to experience different levels of traumatic experiences or to different levels of good parenting. Instead, they measure these variables and examine how they relate to one another.

In these studies, researchers often use the terms predictor variables and outcome variables rather than independent and dependent variables.

A predictor variable is a measured variable, and used to predict or explain another variable. An outcome variable is a measured variable being predicted. Notice that both variables are measured variables - nothing is manipulated by the researcher.

For example, imagine that researchers want to know whether stress predicts college students’ grade point average (GPA). The predictor variable is stress, and the outcome variable is GPA.

As you learn more about research methods, you’ll discover that independent and dependent variables correspond with experiments, and predictor and outcome variables correspond with nonexperimental or correlational research.

Putting it All Together

So far, you’ve learned how psychologists move from a broad idea to a testable study. The process begins with a scientific theory or an observation that inspires a research question. The researcher then develops a hypothesis, identifies the variables involved, and determines the role of each variable in the study. The remaining chapters of this textbook will build on these ideas as you learn how psychologists design studies, collect data, and draw conclusions about behavior and mental processes. See Figure 2.1.

Figure 2.1 The Theory Data Cycle^[1]

Pause and Reflect Answer: If you wanted to test whether listening to music while studying affects exam performance, what would be the independent variable? What would be the dependent variable?

In this example, the independent variable is whether students listen to music while studying. This is the variable the researcher changes. The dependent variable is exam performance, usually measured by the students’ scores on an exam. This is the outcome variable the researcher measures to determine whether the independent variable had an effect.

Media Attributions

^[1] The theory data cycle. Developing a hypothesis. In Research Methods in Psychology (4^th ed.), by R.S. Jhangiani, I.-C.A. Chiang, C. Cuttler, & D.C. Leighton (2019). Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Text Attributions

Dudley, M. (2019, January 24). Research methods in psychology". OER Commons. https://oercommons.org/authoring/51456-research-methods-in-psychology Licensed under CC BY-NC-SA 4.0. Modified by current authors.

References

Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67(4), 371-378. https://doi.org/10.1037/h0040525

Chapter 3: Finding and Evaluating Psychological Research

Why Conduct a Literature Review?

Recall that one of the most common sources of inspiration for a research question is previous research. Therefore, it is important to review the literature early in the research process. The research literature in any field is all the published research in that field. Reviewing the research literature means finding, reading, and summarizing the published research relevant to your topic of interest. In addition to helping you discover new research questions, reviewing the literature early in the research process can help you in several other ways.

• It can tell you if a research question has already been answered.
• It can help you evaluate the interestingness of a research question.
• It can give you ideas for how to conduct your own study.
• It can tell you how your study fits into the research literature.

Reviewing the literature may seem overwhelming at first, especially when you discover hundreds or even thousands of articles on your topic. The good news is that experienced researchers rarely read every article they find. Instead, they use effective search strategies to quickly locate the most relevant and highest-quality research. This chapter will show you how to search efficiently so you can spend less time searching and more time learning.

Although the research literature includes millions of publications, not all sources are equally valuable when conducting a literature review. Psychologists rely primarily on peer-reviewed journal articles and scholarly books because these sources provide the strongest scientific evidence.

Pause and Reflect: Imagine you’re studying the effects of caffeine on anxiety. Why might it be a mistake to conduct a study before reviewing what other researchers have already discovered?

Professional Journals

Professional journals are periodicals that publish original research articles. There are thousands of professional journals that publish research in psychology and related fields. Most articles in professional journals are one of two basic types: empirical research reports and review articles. Empirical research reports describe one or more new empirical studies conducted by the authors. They introduce a research question, explain why it is interesting, review previous research, describe their method and results, and draw their conclusions. Review articles summarize previously published research on a topic and usually present new ways to organize or explain the results. When a review article is devoted primarily to presenting a new theory, it is often referred to as a theoretical article. When a review article provides a statistical summary of all the previous results it is referred to as a meta-analysis. Table 3.1 summarizes the characteristics of empirical research reports, review articles, and meta-analyses.

Table 3.1 Types of Sources

How do researchers get their work published?

Most professional journals in psychology undergo a process of double-blind peer review. Researchers who want to publish their work in the journal submit a manuscript to the editor—who is generally an established researcher too—who in turn sends it to two or three experts on the topic. Each reviewer reads the manuscript, writes a critical but constructive review, and sends the review back to the editor along with recommendations about whether the manuscript should be published or not. The editor then decides whether to accept the article for publication, ask the authors to make changes and resubmit it for further consideration, or reject it outright. In any case, the editor forwards the reviewers’ written comments to the researchers so that they can revise their manuscript accordingly. This entire process is double-blind, as the reviewers do not know the identity of the researcher(s) and vice versa. Double-blind peer review is helpful because it ensures that the work meets basic standards of the field before it can enter the research literature. However, in order to increase transparency and accountability, some journals now utilize an open peer review process wherein the identities of the reviewers (which remain concealed during the peer review process) are published alongside the journal article.

Because of peer review, journal articles are generally more trustworthy than websites, blogs, or social media posts. Although peer review isn’t perfect, it helps ensure that published research meets the standards of the scientific community before becoming part of the scientific literature.

Scholarly Books

Scholarly books are books written by researchers and practitioners mainly for use by other researchers and practitioners. A monograph is written by a single author or a small group of authors and usually gives a coherent presentation of a topic much like an extended review article. Edited volumes have an editor or a small group of editors who recruit many authors to write separate chapters on different aspects of the same topic. In general, scholarly books undergo a peer review process similar to that used by professional journals.

Literature Search Strategies

A Step-by-Step Literature Search

Finding relevant research articles usually takes more time than just typing a topic into a database and finding exactly what you want. Using effective searching strategies can help you locate higher quality sources, save time, and ensure you are finding relevant literature for you.

Step 1.

Start with your research question.

Example: Does social media affect loneliness among college students?

Step 2.

Identify keywords.

Table 3.2 Examples of main ideas and keywords

Step 3.

Use Boolean Operators. Boolean operators help you broaden or narrow your search by combining keywords.

• AND narrows your search by returning articles that contain both search terms
  • Example: social media AND loneliness
• This search will return articles that discuss both social media and loneliness.
• OR broadens your search by returning articles that contain any of the search terms. This is especially useful for searching synonyms or related items.
  • Example: social media OR TikTok OR Instagram OR Facebook
  • This search will return articles that mention any of these terms.

Putting it all together, our search terms might look like this:

(Social media OR TikTok OR Instagram OR Facebook) AND (loneliness OR social isolation)

Step 4.

Search PsycINFO.

PsycINFO is a database produced by the American Psychological Association (APA). It consists of individual records for each article, book chapter, or book in the database. Each record includes basic publication information, an abstract or summary of the work (like the one presented at the start of this chapter), and a list of other works cited by that work. A computer interface allows entering one or more search terms like modeled above, and returns any records that contain those search terms. Each record also contains lists of keywords that describe the content of the work and also a list of index terms. The index terms are especially helpful because they are standardized. Research on differences between females and males, for example, is always indexed under “Human Sex Differences.” Research on note-taking is always indexed under the term “Learning Strategies.” If you do not know the appropriate index terms, PsycINFO includes a thesaurus that can help you find them.

Given that there are nearly four million records in PsycINFO, you may have to try a variety of search terms in different combinations and at different levels of specificity before you find what you are looking for. Imagine, for example, that you are interested in the question of whether males and females differ in terms of their ability to recall experiences from when they were very young. If you were to enter the search term “memory,” it would return far too many records to look through individually. This is where the thesaurus helps. Entering “memory” into the thesaurus provides several more specific index terms—one of which is “early memories.” While searching for “early memories” among the index terms still returns too many to look through individually—combining it with “human sex differences” as a second search term returns fewer articles, many of which are highly relevant to the topic.

Another helpful tip is to filter your results.

• To find empirical journal articles only, select Scholarly/Peer reviewed
• To find full text articles, select Online (Full Text) Only

If your search returns too many articles, try adding another keyword or using more specific search terms. If it returns too few articles, broaden your search by removing a keyword or adding synonyms with OR.

Other Sources of Research

Open Science Framework (OSF)

What is OSF?

It is a free online platform for researchers and students. OSF provides access to research materials, datasets, preprints, and study registrations. It allows researchers to share and organize their work, can be used to find research resources, and learn from existing studies. It promotes openness and collaboration in research.

Google Scholar

Google Scholar is a free search engine to access scholarly literature. Google Scholar’s “Cited by” feature is a powerful way to discover newer research on a topic. When you click “Cited by” beneath an article, Google Scholar displays more recent publications that have references to that article in their own research. This allows you to see how the scientific conversation has evolved over time and quickly identify newer studies related to your topic.

How to access Google Scholar

1. Search Google Scholar or go directly to scholar.google.com
2. Search to find sources
3. Use the Cited by feature in Google Scholar to find newer studies that have referenced an article.

Here is a helpful tutorial: How to Use Google Scholar

AI: Benefits

AI can be a valuable tool during the research process when used appropriately and ethically. AI can help generate keywords and search terms for literature searches. Research-focused AI databases (e.g., Consensus) can help identify relevant scholarly articles, generate keywords for literature searches, and summarize research findings. AI can also assist in brainstorming research questions, refining topics, and discovering related concepts.

Helpful Videos:

Using AI for literature review - ethically!

Consensus vs ChatGPT - using AI for Literature Review and Research

The Best Free AI Tools for Research (2026)

AI: Risks

Although AI can be helpful, it also has important limitations that researchers should understand. General AI chatbots such as ChatGPT can sometimes generate inaccurate information or hallucinate sources that do not actually exist. Due to this be sure to check your sources through reliable databases such as PsycINFO or Google Scholar, and read the original source before citing it or drawing conclusions from it. AI generated summaries may also omit important details, misinterpret findings, or oversimplify complex research. Always verify any AI-generated summaries by consulting the original source. Additionally, relying too much on AI can hinder the development of critical thinking and research skills. To also maintain academic integrity, AI should be used as a tool, not a replacement for your own thinking, analysis, and writing.

Helpful Videos:

How to Stop AI from Killing Your Critical Thinking | Advait Sarkar

Evaluating Sources using the CRAAP test

Finding More Articles

In addition to entering search terms into PsycINFO and other databases, there are several other techniques you can use to search the research literature. First, if you have one good article or book chapter on your topic—a recent review article is best—you can look through the reference list of that article for other relevant articles, books, and book chapters. In fact, you should do this with any relevant article or book chapter you find. You can also start with a classic article or book chapter on your topic, find its record in PsycINFO (by entering the author’s name or article’s title as a search term), and link from there to a list of other works in PsycINFO that cite that classic article. This works because other researchers working on your topic are likely to be aware of the classic article and cite it in their own work. You can also do a general Internet search using search terms related to your topic or the name of a researcher who conducts research on your topic. This might lead you directly to works that are part of the research literature (e.g., articles in open-access journals or posted on researchers’ own websites). The search engine Google Scholar is especially useful for this purpose. A general Internet search might also lead you to websites that are not part of the research literature but might provide references to works that are. Finally, you can talk to people (e.g., your instructor or other faculty members in psychology) who know something about your topic and can suggest relevant articles and book chapters.

Read Articles Efficiently

Research articles can seem intimidating because they often contain technical language and detailed statistical analyses. Fortunately, you do not need to read every article from beginning to end before deciding whether it is useful.

A good strategy is to read the sections in this order:

1. Abstract: What was the study about?
2. Introduction: Why was the study conducted?
3. Discussion: What did the researchers conclude?
4. Results: What did they specifically find?
5. Method: How was the study conducted?

Reading articles in this order will help you quickly determine whether an article is relevant to your research question before investing time reading every detail.

Conducting a literature review is much more than collecting articles. It is the process of discovering what researchers already know, identifying unanswered questions, and building a foundation for your own research. As you gain experience, searching the literature will become faster and more efficient. The time you invest in finding high-quality research will be well worth it.

Pause and Reflect: If you found 500 articles on your topic, would you read every one? Based on this chapter, what strategies would you use to narrow your search?

Text Attributions

Jhangiani, R.S., Chiang, I-C.A., Cuttler, C. & Leighton, D.C. (2019). Research Methods in Psychology (4th ed). KPU Pressbooks. https://kpu.pressbooks.pub/psychmethods4e/. Licensed under CC BY-NC-SA 4.0. Modified by current authors.

Chapter 4: Conducting Ethical Research

Why Research Ethics Matter

Imagine you have developed an interesting research question, found a gap in the scientific literature, and written a strong hypothesis. Your study is well designed and has the potential to contribute to our understanding of human behavior. Before collecting a single piece of data, however, the most important question remains: Should this study be conducted?

Not every scientifically interesting study is ethically acceptable. Researchers have a responsibility to protect the rights, safety, and well-being of the people and animals who participate in their research. In psychology, conducting good research means more than using sound scientific methods - it also means treating participants with honesty, fairness, and respect.

Unfortunately, the importance of research ethics was learned through painful mistakes. Throughout history, some researchers have placed scientific discovery above the wellbeing of participants, leading to studies that caused unnecessary harm and violated basic human rights. These cases changed the way research is conducted and led to the ethical principles and oversight procedures that guide psychological research today.

Historical Cases

Nazi Medical Experiments

One of the darkest chapters in the history of scientific research occurred during World War II, when physicians and scientists in Nazi Germany conducted cruel and often fatal experiments on prisoners held in concentration camps. Many of the victims were Jewish people, but the experiments also targeted Romani people, people with disabilities, political prisoners, Soviet prisoners of war, and other groups persecuted by the Nazi regime. Rather than being treated as human beings with inherent dignity and rights, these individuals were dehumanized and viewed as expendable for the sake of advancing the Nazi regime’s political and ideological goals. Prisoners were subjected to extreme cold, high-altitude conditions, infectious diseases, forced sterilization, and experimental surgeries often performed without anesthesia, and other inhumane procedures without their knowledge or consent. Many suffered permanent injuries or died as a result of these experiments.

Pause and Reflect: What allowed researchers to justify treating certain groups of people so differently from others? Why is recognizing the dignity and worth of every participant essential to ethical research?

Tuskegee Syphilis Study

Although the Nazi medical experiments are among the most infamous examples of unethical research, serious ethical violations have also occurred in the United States. One of the best- known examples is the Tuskegee Syphilis Study, conducted by the U.S. Public Health Service between 1932 and 1972 - a period of 40 years (Reverby, 2009). The study enrolled hundreds of poor African American men in rural Alabama, many of whom had syphilis. However, the participants didn’t know this. Instead, they were told they had “bad blood” and that they would receive treatment. This was a lie. The true purpose of the study was to observe the natural progression of the disease without providing effective treatment. Even after penicillin became the standard and highly effective treatment for syphilis in the 1940s, researchers intentionally withheld treatment and prevented many participants from receiving it elsewhere. As a result, many men experienced unnecessary suffering, developed serious health complications, transmitted the disease to their wives, and died. The Tuskegee Syphilis Study demonstrated the importance of honesty, informed consent, justice, and respect for participants, and it led to major reforms in the oversight of research involving human participants.

Pause and Reflect: If you had been one of the participants and learned years later that an effective treatment had existed but had been intentionally withheld from you, how do you think that would affect your trust in researchers and the medical community?

Milgram’s Obedience Study

Not all ethically important research involved physical harm or the deliberate exploitation of participants. In the early 1960s, psychologist Stanley Milgram conducted a series of studies to examine whether ordinary people would obey an authority figure, even when doing so appeared to harm another person (Milgram, 1963). Participants believed they were administering increasingly painful electric shocks to another individual whenever an incorrect answer was given on a memory task. In reality, no shocks were delivered - the “learner” was a research confederate, playing a role that relied on deception.

Milgram found that many participants continued administering what they believed were dangerous shocks when instructed to do so by the experimenter. The findings provided important insights into the power of authority and have helped psychologists understand events such as the Holocaust and other historical atrocities in which ordinary people obeyed harmful orders.

Unlike the Nazi experiments and the Tuskegee study, Milgram’s participants were debriefed and suffered no lasting physical harm. Nevertheless, many psychologists questioned whether exposing participants to such intense psychological distress was ethically justified. Milgram’s work illustrates that ethical decisions are not always clear-cut. Researchers must often weigh the potential scientific benefits of a study against the possible risks to participants.

Although these historical cases differed in important ways, they shared a common lesson: scientific research must be guided by ethical principles that protect participants while allowing important discoveries to be made. In response to these and other events, researchers, professional organizations, and governments developed ethical guidelines that continue to shape psychological research today. We’ll review these next.

Pause and Reflect: If you had served on an ethics review board in the 1960s, would you have approved Milgram’s study? Why or why not? What changes, if any, would you require before allowing the study to proceed?

A Framework for Thinking about Research Ethics

Table 4.1 presents a framework for thinking through the ethical issues involved in psychological research. The rows of Table 4.1 represent four general moral principles that apply to scientific research: weighing risks against benefits, acting responsibly and with integrity, seeking justice, and respecting people’s rights and dignity. (These principles are adapted from those in the American Psychological Association [APA] Ethics Code.) The columns of Table 4.1 represent three groups of people that are affected by scientific research: the research participants, the scientific community, and society more generally. The idea is that a thorough consideration of the ethics of any research project must take into account how each of the four moral principles applies to each of the three groups of people.

Table 4.1 A Framework for Thinking about Ethical Issues in Scientific Research^[1]

The Four Ethical Principles of Research

Principle 1 - Scientific research must weigh risks against benefits

Scientific research in psychology is ethical only if its potential benefits outweigh its potential risks. Among the risks to participants are that a treatment might fail to help or even be harmful, a procedure might result in physical or psychological harm, or their right to privacy might be violated. Among the potential benefits are receiving helpful treatment, learning about psychology, experiencing the satisfaction of contributing to scientific knowledge, and receiving money or course credit for participating.

Scientific research can have risks and benefits to the scientific community and to society too. A risk to science is that if a research question is uninteresting or a study is poorly designed, then the time, money, and effort spent on that research could have been spent on more productive research. A risk to society is that research results could be misunderstood or misapplied with harmful consequences. For example, a now-discredited study that falsely linked the measles, mumps, and rubella (MMR) vaccine to autism greatly contributed to vaccine hesitancy and reduced vaccination rates, despite overwhelming scientific evidence showing no such link.

In practice, researchers must carefully consider these risks and benefits before a study begins. Institutional Review Boards (IRBs) review proposed studies to help ensure that participants are protected and that the potential benefits of the research justify any risks.

Principle 2 - Acting Responsibly and With Integrity

Researchers must act responsibly and with integrity. This means carrying out their research in a thorough and competent manner, meeting professional obligations, and being truthful throughout the research process. Acting with integrity is essential because it promotes trust, which is the foundation of ethical research. Participants must be able to trust that researchers are being honest about what a study involves, will keep their promises (such as maintaining confidentiality), and will conduct their research in ways that maximize benefits and minimize risks.

Integrity also requires researchers to collect, analyze, and report their findings honestly. Fabricating data, falsifying results, or plagiarizing the work of others undermines public trust and damages the scientific process.

Principle 3 - Seeking Justice

Researchers must conduct their research in a just manner. They should treat their participants fairly, for example, by giving them adequate compensation for their participation and making sure that benefits and risks are distributed across all participants. For example, in a study of a new and potentially beneficial psychotherapy, some participants might receive the psychotherapy while others serve as a control group that receives no treatment. If the psychotherapy turns out to be effective, it would be fair to offer it to participants in the control group when the study ends.

Justice also means ensuring that the burdens and benefits of research are shared fairly. Researchers should avoid exploiting vulnerable populations simply because they are more accessible or less able to refuse participation.

Principle 4 - Respecting People’s Rights and Dignity

Researchers must respect people’s rights and dignity as human beings. One element of this is respecting participants’ autonomy—their right to make their own decisions and choose whether to participate in research free from coercion or undue pressure. This principle is reflected in the practice of informed consent, which requires researchers to explain a study and obtain participants’ voluntary agreement before they take part.

Another element of respecting people’s rights and dignity is respecting their privacy—their right to decide what information about them is shared with others. This means that researchers must maintain confidentiality, which is essentially an agreement not to disclose participants’ personal information without their consent or some appropriate legal authorization.

These four principles provide the ethical foundation for psychological research. In practice, however, ethical decisions are not always simple. Researchers must often balance competing priorities, such as advancing scientific knowledge while protecting participants from harm. To help researchers make these decisions consistently, professional organizations and governments have developed formal ethics codes and oversight procedures.

From Principles to Practice: Ethics Codes

The four ethical principles of weighing risks against benefits, acting with integrity, seeking justice, and respecting people’s rights and dignity provide a useful framework for thinking about ethical research. However, principles alone do not tell researchers exactly what to do in every situation. To help researchers make consistent ethical decisions, governments and professional organizations have developed formal ethics codes and oversight procedures. These guidelines help protect participants while allowing important scientific discoveries to continue.

Historical Overview

The Nuremberg Code

The first major code of research ethics was the Nuremberg Code, developed in 1947 following the trials of Nazi physicians who conducted horrific experiments on concentration camp prisoners during World War II. The Nuremberg Code established that participation in research must be voluntary and based on informed consent. It also emphasized that researchers should minimize unnecessary physical and psychological harm, ensure that the potential benefits of research outweigh its risks, and allow participants to withdraw from a study at any time. Although originally developed in response to the Nazi medical experiments, the Nuremberg Code became the foundation for modern research ethics and influenced many of the ethical guidelines that followed.

The Declaration of Helsinki

In 1964, the World Medical Association expanded upon the Nuremberg Code by adopting the Declaration of Helsinki. This document emphasized that research involving human participants should follow a carefully written research protocol and be reviewed by an independent committee before data collection begins. These ideas helped establish the practice of independent ethical review that is now common in research institutions around the world.

The Belmont Report

In the United States, public concern following the Tuskegee Syphilis Study and other unethical research led to the publication of the Belmont Report in 1979. The Belmont Report became the foundation of modern research ethics in the United States and identified three core principles that should guide research involving human participants:

1. Respect for Persons: Individuals should be treated as autonomous decision makers and should voluntarily choose whether to participate in research. This principle is the basis for informed consent.
2. Beneficence: Researchers should maximize the potential benefits of research while minimizing possible risks and harm to participants.
3. Justice: The benefits and burdens of research should be distributed fairly, and vulnerable populations should not be exploited simply because they are more accessible or less able to refuse participation.

Although the wording differs slightly, these principles closely parallel the ethical principles discussed earlier in this module. Together, they continue to guide the ethical conduct of research in psychology and many other scientific disciplines.

Institutional Review Boards

Today, most colleges, universities, hospitals, and research institutions have an Institutional Review Board (IRB), a committee responsible for reviewing research involving human participants before data collection begins. Importantly, the IRB committee consists of a diverse group of individuals with different backgrounds, areas of expertise, and perspectives. Typically, an IRB includes scientists and nonscientists, as well as at least one member who is not affiliated with the institution. This diversity in membership helps ensure that research proposals are evaluated from multiple viewpoints, making it more likely that potential ethical concerns will be recognized before a study begins. Furthermore, because everyone is susceptible to cognitive biases, a diverse committee is also less likely than a single reviewer or a homogeneous group to overlook important ethical concerns because of shared assumptions or blind spots.

Researchers submit a detailed research proposal describing the purpose of the study, the procedures, potential risks and benefits, and the steps they will take to protect participants. The IRB evaluates whether the study meets ethical standards and may approve the study, require modifications, or determine that it cannot be conducted as proposed.

APA Ethics Code

Although the Nuremberg Code, Declaration of Helsinki, and Belmont Report provide broad ethical guidance, psychologists in the United States also follow the American Psychological Association (APA) Ethical Principles of Psychologists and Code of Conduct. This document provides detailed standards for psychologists in research, teaching, clinical practice, and other professional activities. For researchers, the most relevant section is Standard 8: Research and Publication, which outlines ethical responsibilities before, during, and after a study.

Several parts of the APA Ethics Code are especially important for beginning researchers. In the following sections, we will examine informed consent, deception, debriefing, research with nonhuman animals, and scholarly integrity. Together, these standards help ensure that psychological research is conducted ethically, responsibly, and with respect for participants.

Informed Consent

Standards 8.02 to 8.05 are about informed consent. Again, informed consent means obtaining and documenting people’s agreement to participate in a study, having informed them of everything that might reasonably be expected to affect their decision. This includes details of the procedure, the risks and benefits of the research, the fact that they have the right to decline to participate or to withdraw from the study, the consequences of doing so, and any legal limits to confidentiality. For example, some states require researchers who learn of child abuse or other crimes to report this information to authorities.

Although the process of obtaining informed consent often involves having participants read and sign a consent form, it is important to understand that this is not all it is. Although having participants read and sign a consent form might be enough when they are competent adults with the necessary ability and motivation, many participants do not actually read consent forms or read them but do not understand them. For example, participants often mistake consent forms for legal documents and mistakenly believe that by signing them they give up their right to sue the researcher. Even with competent adults, therefore, it is good practice to tell participants about the risks and benefits, demonstrate the procedure, ask them if they have questions, and remind them of their right to withdraw at any time—in addition to having them read and sign a consent form.

Note also that there are situations in which informed consent is not necessary. These include situations in which the research is not expected to cause any harm and the procedure is straightforward or the study is conducted in the context of people’s ordinary activities. For example, if you wanted to sit outside a public building and observe whether people hold the door open for people behind them, you would not need to obtain their informed consent. Similarly, if a college instructor wanted to compare two legitimate teaching methods across two sections of his research methods course, he would not need to obtain informed consent from his students.

Deception

Deception of participants in psychological research can take a variety of forms: misinforming participants about the purpose of a study, using confederates, using phony equipment like Milgram’s shock generator, and presenting participants with false feedback about their performance (e.g., telling them they did poorly on a test when they actually did well). Deception also includes not informing participants of the full design or true purpose of the research even if they are not actively misinformed. For example, a study on incidental learning—learning without conscious effort—might involve having participants read through a list of words in preparation for a “memory test” later. Although participants are likely to assume that the memory test will require them to recall the words, it might instead require them to recall the contents of the room or the appearance of the research assistant.

Some researchers have argued that deception of participants is rarely if ever ethically justified. Among their arguments are that it prevents participants from giving truly informed consent, fails to respect their dignity as human beings, has the potential to upset them, makes them distrustful and therefore less honest in their responding, and damages the reputation of researchers in the field (Baumrind, 1985).

Note, however, that the APA Ethics Code takes a more moderate approach—allowing deception when the benefits of the study outweigh the risks, participants cannot reasonably be expected to be harmed, the research question cannot be answered without the use of deception, and participants are informed about the deception as soon as possible. This approach acknowledges that not all forms of deception are equally bad. Compare, for example, Milgram’s study in which he deceived his participants in several significant ways that resulted in their experiencing severe psychological stress with an incidental learning study in which a “memory test” turns out to be slightly different from what participants were expecting. It also acknowledges that some scientifically and socially important research questions can be difficult or impossible to answer without deceiving participants. Knowing that a study concerns the extent to which they obey authority, act aggressively toward a peer, or help a stranger is likely to change the way people behave so that the results no longer generalize to the real world.

Debriefing

Standard 8.08 is about debriefing. This is the process of informing participants as soon as possible of the purpose of the study, revealing any deception, and correcting any other misconceptions they might have as a result of participating. Debriefing also involves minimizing harm that might have occurred. For example, an experiment on the effects of being in a sad mood on memory might involve inducing a sad mood in participants by having them think sad thoughts, watch a sad video, or listen to sad music. Debriefing would be the time to return participants’ moods to normal by having them think happy thoughts, watch a happy video, or listen to happy music.

Nonhuman Animal Subjects

Standard 8.09 is about the humane treatment and care of nonhuman animal subjects. Although most contemporary research in psychology does not involve nonhuman animal subjects, a significant minority of it does—especially in the study of learning and conditioning, behavioral neuroscience, and the development of drug and surgical therapies for psychological disorders.

The use of nonhuman animal subjects in psychological research is like the use of deception in that there are those who argue that it is rarely, if ever, ethically acceptable (Bowd & Shapiro, 1993). Clearly, nonhuman animals are incapable of giving informed consent. Yet they can be subjected to numerous procedures that are likely to cause them suffering. They can be confined, deprived of food and water, subjected to pain, operated on, and ultimately euthanized. (Of course, they can also be observed benignly in natural or zoolike settings.) Others point out that psychological research on nonhuman animals has resulted in many important benefits to humans, including the development of behavioral therapies for many disorders, more effective pain control methods, and antipsychotic drugs (Miller, 1985). It has also resulted in benefits to nonhuman animals, including alternatives to shooting and poisoning as means of controlling them.

As with deception, the APA acknowledges that the benefits of research on nonhuman animals can outweigh the costs, in which case it is ethically acceptable. However, researchers must use alternative methods when they can. When they cannot, they must acquire and care for their subjects humanely and minimize the harm to them. For more information on the APA’s position on nonhuman animal subjects, see the website of the APA’s Committee on Animal Research and Ethics.

Scholarly Integrity

Standards 8.10 to 8.15 are about scholarly integrity. These include the obvious points that researchers must not fabricate data or plagiarize. Plagiarism means using others’ words or ideas without proper acknowledgment. Proper acknowledgment generally means indicating direct quotations with quotation marks and providing a citation to the source of any quotation or idea used.

According to the APA Ethics Code, faculty advisers should discuss publication credit—who will be an author and the order of authors—with their student collaborators as early as possible in the research process.

The remaining standards make some less obvious but equally important points. Researchers should not publish the same data a second time as though it were new, they should share their data with other researchers, and as peer reviewers they should keep the unpublished research they review confidential. Note that the authors’ names on published research—and the order in which those names appear—should reflect the importance of each person’s contribution to the research. It would be unethical, for example, to include as an author someone who had made only minor contributions to the research (e.g., analyzing some of the data) or for a faculty member to make himself or herself the first author on research that was largely conducted by a student.

Pause and Reflect: A researcher finds that the results do not support the original hypothesis. The researcher is tempted to leave out a few participants whose responses seem unusual because doing so would make the results statistically significant. Is this ethical? Why or why not? What should the researcher do instead?

Putting Ethics Into Practice

In this section, we look at some practical advice for conducting ethical research in psychology. Again, it is important to remember that ethical issues arise well before you begin to collect data and continue to arise through publication and beyond.

Know and Accept Your Ethical Responsibilities

As the American Psychological Association (APA) Ethics Code notes in its introduction, “Lack of awareness or misunderstanding of an ethical standard is not itself a defense to a charge of unethical conduct.” This is why the very first thing that you must do as a new researcher is know and accept your ethical responsibilities. At a minimum, this means reading and understanding the relevant standards of the APA Ethics Code, distinguishing minimal risk from at-risk research, and knowing the specific policies and procedures of your institution—including how to prepare and submit a research protocol for institutional review board (IRB) review. If you are conducting research as a course requirement, there may be specific course standards, policies, and procedures. If any standard, policy, or procedure is unclear—or you are unsure what to do about an ethical issue that arises—you must seek clarification. You can do this by reviewing the relevant ethics codes, reading about how similar issues have been resolved by others, or consulting with more experienced researchers, your IRB, or your course instructor. Ultimately, you as the researcher must take responsibility for the ethics of the research you conduct.

Identify and Minimize Risks

As you design your study, you must identify and minimize risks to participants. Start by listing all the risks, including risks of physical and psychological harm and violations of confidentiality. Remember that it is easy for researchers to see risks as less serious than participants do or even to overlook them completely. For example, one student researcher wanted to test people’s sensitivity to violent images by showing them gruesome photographs of crime and accident scenes. Because she was an emergency medical technician, however, she greatly underestimated how disturbing these images were to most people. Remember too that some risks might apply only to some participants. For example, while most people would have no problem completing a survey about their fear of various crimes, those who have been a victim of one of those crimes might become upset. This is why you should seek input from a variety of people, including your research collaborators, more experienced researchers, and even from nonresearchers who might be better able to take the perspective of a participant.

Once you have identified the risks, you can often reduce or eliminate many of them. One way is to modify the research design. For example, you might be able to shorten or simplify the procedure to prevent boredom and frustration. You might be able to replace upsetting or offensive stimulus materials (e.g., graphic accident scene photos) with less upsetting or offensive ones (e.g., milder photos of the sort people are likely to see in the newspaper). A good example of modifying a research design is a 2009 replication of Milgram’s study conducted by Jerry Burger. Instead of allowing his participants to continue administering shocks up to the 450-V maximum, the researcher always stopped the procedure when they were about to administer the 150-V shock (Burger, 2009). This made sense because in Milgram’s study (a) participants’ severe negative reactions occurred after this point and (b) most participants who administered the 150-V shock continued all the way to the 450-V maximum. Thus the researcher was able to compare his results directly with Milgram’s at every point up to the 150-V shock and also was able to estimate how many of his participants would have continued to the maximum—but without subjecting them to the severe stress that Milgram did. (The results, by the way, were that these contemporary participants were just as obedient as Milgram’s were.)

A second way to minimize risks is to use a pre-screening procedure to identify and eliminate participants who are at high risk. You can do this in part through the informed consent process. For example, you can warn participants that a survey includes questions about their fear of crime and remind them that they are free to withdraw if they think this might upset them. Pre-screening can also involve collecting data to identify and eliminate participants. For example, Burger used an extensive pre-screening procedure involving multiple questionnaires and an interview with a clinical psychologist to identify and eliminate participants with physical or psychological problems that put them at high risk.

A third way to minimize risks is to take active steps to maintain confidentiality. You should keep signed consent forms separately from any data that you collect and in such a way that no individual’s name can be linked to his or her data. In addition, beyond people’s sex, age, and ethnicity, you should only collect personal information that you actually need to answer your research question. If people’s sexual orientation or ethnicity is not clearly relevant to your research question, for example, then do not ask them about it. Be aware also that certain data collection procedures can lead to unintentional violations of confidentiality. When participants respond to an oral survey in a shopping mall or complete a questionnaire in a classroom setting, it is possible that their responses will be overheard or seen by others. If the responses are personal, it is better to administer the survey or questionnaire individually in private or to use other techniques to prevent the unintentional sharing of personal information.

Identify and Minimize Deception

Remember that deception can take a variety of forms, not all of which involve actively misleading participants. It is also deceptive to allow participants to make incorrect assumptions (e.g., about what will be on a “memory test”) or simply withhold information about the full design or purpose of the study. This is called passive deception. Overall, it is best to identify and minimize all forms of deception.

Remember that according to the APA Ethics Code, deception is ethically acceptable only if there is no way to answer your research question without it. Therefore, if your research design includes any form of active deception, you should consider whether it is truly necessary. Imagine, for example, that you want to know whether the age of college professors affects students’ expectations about their teaching ability. You could do this by telling participants that you will show them photos of college professors and ask them to rate each one’s teaching ability. But if the photos are not really of college professors but of your own family members and friends, then this would be deception. This deception could easily be eliminated, however, by telling participants instead to imagine that the photos are of college professors and to rate them as if they were.

In general, it is considered acceptable to wait until debriefing before you reveal your research question as long as you describe the procedure, risks, and benefits during the informed consent process. For example, you would not have to tell participants that you wanted to know whether the age of college professors affects people’s expectations about them until the study was over. Not only is this information unlikely to affect people’s decision about whether or not to participate in the study, but it has the potential to invalidate the results. Participants who know that age is the independent variable might rate the older and younger “professors” differently because they think you want them to. Alternatively, they might be careful to rate them the same so that they do not appear prejudiced. But even this extremely mild form of deception can be minimized by informing participants—orally, in writing, or both—that although you have accurately described the procedure, risks, and benefits, you will wait to reveal the research question until afterward. In essence, participants give their consent to be deceived or to have information withheld from them until later.

Weigh the Risks Against the Benefits

Once the risks of the research have been identified and minimized, you need to weigh them against the benefits. This requires identifying all the benefits. Remember to consider benefits to the participants, to science, and to society. If you are a student researcher, remember that one of the benefits is the knowledge you will gain about how to conduct scientific research in psychology—knowledge you can then use to complete your studies and succeed in graduate school or in your career.

If the research poses minimal risk—no more than in people’s daily lives or routine physical or psychological examinations—then even a small benefit to participants, science, or society is generally considered enough to justify it. If it poses more than minimal risk, then there should be more benefits. If the research has the potential to upset some participants, for example, then it becomes more important that the study be well designed and answer a scientifically interesting research question or have clear practical implications. It would be unethical to subject people to pain, fear, or embarrassment for no better reason than to satisfy one’s personal curiosity. In general, psychological research that has the potential to cause harm that is more than minor or lasts for more than a short time is rarely considered justified by its benefits. Consider, for example, that Milgram’s study—as interesting and important as the results were—would be considered unethical by today’s standards.

Create Informed Consent and Debriefing Procedures

Once you have settled on a research design, you need to create your informed consent and debriefing procedures. Start by deciding whether informed consent is necessary according to APA Standard 8.05. If informed consent is necessary, there are several things you should do. First, when you recruit participants—whether it is through word of mouth, posted advertisements, or a participant pool—provide them with as much information about the study as you can. This will allow those who might find the study objectionable to avoid it. Second, prepare a script or set of “talking points” to help you explain the study to your participants in simple everyday language. This should include a description of the procedure, the risks and benefits, and their right to withdraw at any time. Third, create an informed consent form that covers all the points in Standard 8.02a that participants can read and sign after you have described the study to them. Your university, department, or course instructor may have a sample consent form that you can adapt for your own study. If not, an Internet search will turn up several samples. Remember that if appropriate, both the oral and written parts of the informed consent process should include the fact that you are keeping some information about the design or purpose of the study from them but that you will reveal it during debriefing.

Debriefing is similar to informed consent in that you cannot necessarily expect participants to read and understand written debriefing forms. So again it is best to write a script or set of talking points with the goal of being able to explain the study in simple everyday language. During debriefing, you should reveal the research question and full design of the study. For example, if participants are tested under only one condition, then you should explain what happened in the other conditions. If you deceived your participants, you should reveal this as soon as possible, apologize for the deception, explain why it was necessary, and correct any misconceptions that participants might have as a result. Debriefing is also a good time to provide additional benefits to research participants by giving them relevant practical information or referrals to other sources of help. For example, in a study of attitudes toward domestic abuse, you could provide pamphlets about domestic abuse and referral information to the university counseling center for those who might want it.

Remember to schedule plenty of time for the informed consent and debriefing processes. They cannot be effective if you have to rush through them.

Get Approval

The next step is to get institutional approval for your research based on the specific policies and procedures at your institution or for your course. This will generally require writing a protocol that describes the purpose of the study, the research design and procedure, the risks and benefits, the steps taken to minimize risks, and the informed consent and debriefing procedures. Do not think of the institutional approval process as merely an obstacle to overcome but as an opportunity to think through the ethics of your research and to consult with others who are likely to have more experience or different perspectives than you. If the IRB has questions or concerns about your research, address them promptly and in good faith. This might even mean making further modifications to your research design and procedure before resubmitting your protocol.

Follow Through

Your concern with ethics should not end when your study receives institutional approval. It now becomes important to stick to the protocol you submitted or to seek additional approval for anything other than a minor change. During the research, you should monitor your participants for unanticipated reactions and seek feedback from them during debriefing. One criticism of Milgram’s study is that although he did not know ahead of time that his participants would have such severe negative reactions, he certainly knew after he had tested the first several participants and should have made adjustments at that point (Baumrind, 1985). Be alert also for potential violations of confidentiality. Keep the consent forms and the data safe and separate from each other and make sure that no one, intentionally or unintentionally, has access to any participant’s personal information.

Finally, you must maintain your integrity through the publication process and beyond. Address publication credit—who will be authors on the research and the order of authors—with your collaborators early and avoid plagiarism in your writing. Remember that your scientific goal is to learn about the way the world actually is and that your scientific duty is to report on your results honestly and accurately. So do not be tempted to fabricate data or alter your results in any way. Besides, unexpected results are often as interesting, or more so, than expected ones.

Pause and Reflect: Think back to the three historical cases discussed at the beginning of this chapter: the Nazi medical experiments, the Tuskegee Syphilis Study, and Milgram's obedience study. Which ethical principle—or combination of principles—was most clearly violated in each case?

Media Attributions

^[1] A Framework for Thinking about Ethical Issues in Scientific Research. Adapted from Research Ethics by M. Ukeye, in Critical Research Methods in Psychology, by S.D’Costa, M.Ukeye, M. O’Neil, & R. Anguiano, Critical research methods in psychology. Saint Mary’s College of California. Licensed under CC BY-NC-SA 4.0.

Text Attributions

Dudley, M. (2019). Research methods in psychology. OER Commons. https://oercommons.org/authoring/51456-research-methods-in-psychology. Licensed under CC BY-NC-SA 4.0. Modified by current authors.

References

Baumrind, D. (1985). Research using intentional deception: Ethical issues revisited. American Psychologist, 40(2), 165-174. https://doi.org/10.1037/0003-066X.40.2.165

Bowd, A.D. & Shapiro, K.J. (1993). The case against laboratory animal research in psychology. Journal of Social Issues, 49(1), 133-142.

Burger, J.M. (2009). Replicating Milgram: Would people still obey today? American Psychologist, 64(1), 1 – 11. https://doi.org/10.1037/a0010932

Milgram, S. (1963). Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67, 371-378. https://psycnet.apa.org/doi/10.1037/h0040525

Miller, N.E. (1985). The value of behavioral research on animals. American Psychologist, 40(4), 423-440. https://psycnet.apa.org/doi/10.1037/0003-066X.40.4.423

Reverby, S.M. (2009). Examining Tuskegee: The infamous syphilis study and its legacy. Chapel Hill, NC: University of North Carolina Press.

Module 2: Getting Good Data - Measurement and Sampling

Chapter 5: Psychological Measurement

Researchers Tara MacDonald and Alanna Martineau were interested in the effect of female university students’ moods on their intentions to have unprotected sexual intercourse (MacDonald & Martineau, 2002). In a carefully designed empirical study, they found that being in a negative mood increased intentions to have unprotected sex—but only for students who were low in self-esteem. Although there are many challenges involved in conducting a study like this, one of the primary ones is the measurement of the relevant variables. In this study, the researchers needed to know whether each of their participants had high or low self-esteem, which of course required measuring their self-esteem. They also needed to be sure that their attempt to put people into a negative mood (by having them think negative thoughts) was successful, which required measuring their moods. Finally, they needed to see whether self-esteem and mood were related to participants’ intentions to have unprotected sexual intercourse, which required measuring these intentions.

To students who are just getting started in psychological research, the challenge of measuring such variables might seem insurmountable. Is it really possible to measure things as intangible as self-esteem, mood, or an intention to do something? The answer is a resounding yes, and in this chapter we look closely at the nature of the variables that psychologists study and how they can be measured. We also look at some practical issues in psychological measurement.

What Is Measurement?

Measurement is the assignment of scores to individuals so that the scores represent some characteristic of the individuals. This very general definition is consistent with the kinds of measurement that everyone is familiar with—for example, weighing oneself by stepping onto a bathroom scale, or checking the internal temperature of a roasting turkey by inserting a meat thermometer. It is also consistent with measurement throughout the sciences. In physics, for example, one might measure the potential energy of an object in Earth’s gravitational field by finding its mass and height (which of course requires measuring those variables) and then multiplying them together along with the gravitational acceleration of Earth (9.8 m/s²). The result of this procedure is a score that represents the object’s potential energy.

Of course this general definition of measurement is consistent with measurement in psychology too. (Psychological measurement is often referred to as psychometrics.) Imagine, for example, that a cognitive psychologist wants to measure a person’s working memory capacity—his or her ability to hold in mind and think about several pieces of information all at the same time. To do this, she might use a backward digit span task, where she reads a list of two digits to the person and asks him or her to repeat them in reverse order. She then repeats this several times, increasing the length of the list by one digit each time, until the person makes an error. The length of the longest list for which the person responds correctly is the score and represents his or her working memory capacity. Or imagine a clinical psychologist who is interested in how depressed a person is. He administers the Beck Depression Inventory, which is a 21-item self-report questionnaire in which the person rates the extent to which he or she has felt sad, lost energy, and experienced other symptoms of depression over the past 2 weeks. The sum of these 21 ratings is the score and represents his or her current level of depression.

The important point here is that measurement does not require any particular instruments or procedures. It does not require placing individuals or objects on bathroom scales, holding rulers up to them, or inserting thermometers into them. What it does require is some systematic procedure for assigning scores to individuals or objects so that those scores represent the characteristic of interest.

Psychological Constructs

Many variables studied by psychologists are straightforward and simple to measure. These include age, height, weight, and birth order. You can ask people how old they are and be reasonably sure that they know and will tell you. Although people might not know or want to tell you how much they weigh, you can have them step onto a bathroom scale. Other variables studied by psychologists—perhaps the majority—are not so straightforward or simple to measure. We cannot accurately assess people’s level of intelligence by looking at them, and we certainly cannot put their self-esteem on a bathroom scale. These kinds of variables are called constructs (pronounced CON-structs) and include personality traits (e.g., extroversion), emotional states (e.g., fear), attitudes (e.g., toward taxes), and abilities (e.g., athleticism).

Psychological constructs often cannot be observed directly. One reason is that they often represent tendencies to think, feel, or act in certain ways. For example, to say that a particular college student is highly extroverted (see below) does not necessarily mean that she is behaving in an extroverted way right now. In fact, she might be sitting quietly by herself, reading a book. Instead, it means that she has a general tendency to behave in extroverted ways (talking, laughing, etc.) across a variety of situations. Another reason psychological constructs cannot always be observed directly is that they often involve internal processes. Fear, for example, involves the activation of certain central and peripheral nervous system structures, along with certain kinds of thoughts, feelings, and behaviors—none of which is necessarily obvious to an outside observer. Notice also that neither extroversion nor fear “reduces to” any particular thought, feeling, act, or physiological structure or process. Instead, each is a kind of summary of a complex set of behaviors and internal processes.

The Big Five

The Big Five is a set of five broad dimensions that capture much of the variation in human personality. Each of the Big Five can even be defined in terms of six more specific constructs called “facets” (Costa & McCrae, 1992). Table 5.1 describes the Big Five dimensions and facets.

Table 5.1 The Big 5 Personality Dimensions^[1]

The conceptual definition of a psychological construct describes the behaviors and internal processes that make up that construct, along with how it relates to other variables. For example, a conceptual definition of neuroticism (another one of the Big Five) is people’s tendency to experience negative emotions such as anxiety, anger, and sadness across a variety of situations. This definition might also include that it has a strong genetic component, remains fairly stable over time, and is positively correlated with the tendency to experience pain and other physical symptoms.

Students sometimes wonder why, when researchers want to understand a construct like self-esteem or neuroticism, they do not simply look it up in the dictionary. One reason is that many scientific constructs do not have counterparts in everyday language (e.g., working memory capacity). More importantly, researchers develop definitions that are more detailed, more precise, and more accurate than the informal definitions found in a dictionary. As we will see, they do this by proposing conceptual definitions, testing them empirically, and revising them as necessary. Sometimes they throw them out altogether. This is why the research literature often includes different conceptual definitions of the same construct. In some cases, an older conceptual definition has been replaced by a newer one that works better. In others, researchers are still in the process of deciding which of various conceptual definitions is the best.

Operational Definitions

An operational definition is a definition of a variable in terms of precisely how it is to be measured (or manipulated depending on the type of variable you are utilizing and goals of your study; see Module 5 for more information on manipulated variables). These measures generally fall into one of three broad categories. Self-report measures are those in which participants report on their own thoughts, feelings, and actions, as with the Rosenberg Self-Esteem Scale. Behavioral measures are those in which some aspect of participants’ behavior is observed and recorded. This is an extremely broad category that includes the observation of people’s behavior both in highly structured laboratory tasks and in more natural settings. A good example of the former would be measuring working memory capacity using the backward digit span task. A good example of the latter is a famous operational definition of physical aggression from researcher Albert Bandura and his colleagues (Bandura et al., 1961). They let each of several children play for 20 minutes in a room that contained a clown-shaped punching bag called a Bobo doll. They filmed each child and counted the number of acts of physical aggression he or she committed. These included hitting the doll with a mallet, punching it, and kicking it. Their operational definition, then, was the number of these specifically defined acts that the child committed in the 20-minute period. Finally, physiological measures are those that involve recording any of a wide variety of physiological processes, including heart rate and blood pressure, galvanic skin response (an indicator of sweating and physiological arousal), hormone levels, and electrical activity and blood flow in the brain.

For any given variable or construct, there will be multiple operational definitions. Stress is a good example. A rough conceptual definition is that stress is an adaptive response to a perceived danger or threat that involves physiological, cognitive, affective, and behavioral components. But researchers have operationally defined it in several ways. The Social Readjustment Rating Scale (Holmes & Rahe, 1967) is a self-report questionnaire on which people identify stressful events that they have experienced in the past year and assigns points for each one depending on its severity. For example, a man who has been divorced (73 points), changed jobs (36 points), and had a change in sleeping habits (16 points) in the past year would have a total score of 125. The Daily Hassles and Uplifts Scale (Delongis et al., 1982) is similar but focuses on everyday stressors like misplacing things and being concerned about one’s weight. The Perceived Stress Scale (Cohen et al., 1983) is another self-report measure that focuses on people’s feelings of stress (e.g., “How often have you felt nervous and stressed?”). Researchers have also operationally defined stress in terms of several physiological variables including blood pressure and levels of the stress hormone cortisol.

When psychologists use multiple operational definitions of the same construct—either within a study or across studies—they are using converging operations. The idea is that the various operational definitions are “converging” on the same construct. When scores based on several different operational definitions are closely related to each other and produce similar patterns of results, this constitutes good evidence that the construct is being measured effectively and that it is useful. The various measures of stress, for example, are all correlated with each other and have all been shown to be correlated with other variables such as immune system functioning (also measured in a variety of ways) (Segerstrom & Miller, 2004). This is what ultimately allows researchers to draw useful general conclusions, such as “stress is negatively correlated with immune system functioning,” as opposed to more specific and less useful ones, such as “people’s scores on the Perceived Stress Scale are negatively correlated with their white blood counts.”

Levels of Measurement

The psychologist S. S. Stevens suggested that scores can be assigned to individuals in a way that communicates more or less quantitative information about the variable of interest (Stevens, 1946). For example, the officials at a 100-m race could simply rank order the runners as they crossed the finish line (first, second, etc.), or they could time each runner to the nearest tenth of a second using a stopwatch (11.5 s, 12.1 s, etc.). In either case, they would be measuring the runners’ times by systematically assigning scores to represent those times. But while the rank ordering procedure communicates the fact that the second-place runner took longer to finish than the first-place finisher, the stopwatch procedure also communicates how much longer the second-place finisher took. Stevens suggested four different levels of measurement (which he called “scales of measurement”) that correspond to four types of information that can be communicated by a set of scores, and the statistical procedures that can be used with the information.

Nominal Scales

The nominal level of measurement is used for categorical variables and involves assigning scores that are category labels. Category labels communicate whether any two individuals are the same or different in terms of the variable being measured. For example, if you ask your participants about their marital status, you are engaged in nominal-level measurement. Or if you ask your participants to indicate which of several ethnicities they identify themselves with, you are again engaged in nominal-level measurement. The essential point about nominal scales is that they do not imply any ordering among the responses. For example, when classifying people according to their favorite color, there is no sense in which green is placed “ahead of” blue. Responses are merely categorized. Nominal scales thus embody the lowest level of measurement.

Ordinal Scales

The remaining three levels of measurement are used for quantitative variables. The ordinal level of measurement involves assigning scores so that they represent the rank order of the individuals. Ranks communicate not only whether any two individuals are the same or different in terms of the variable being measured but also whether one individual is higher or lower on that variable. For example, a researcher wishing to measure consumers’ satisfaction with their microwave ovens might ask them to specify their feelings as either “very dissatisfied,” “somewhat dissatisfied,” “somewhat satisfied,” or “very satisfied.” The items in this scale are ordered, ranging from least to most satisfied. This is what distinguishes ordinal from nominal scales. Unlike nominal scales, ordinal scales allow comparisons of the degree to which two individuals rate the variable. For example, our satisfaction ordering makes it meaningful to assert that one person is more satisfied than another with their microwave ovens. Such an assertion reflects the first person’s use of a verbal label that comes later in the list than the label chosen by the second person.

On the other hand, ordinal scales fail to capture important information that will be present in the other levels of measurement we examine. In particular, the difference between two levels of an ordinal scale cannot be assumed to be the same as the difference between two other levels (just like you cannot assume that the gap between the runners in first and second place is equal to the gap between the runners in second and third place). In our satisfaction scale, for example, the difference between the responses “very dissatisfied” and “somewhat dissatisfied” is probably not equivalent to the difference between “somewhat dissatisfied” and “somewhat satisfied.” Nothing in our measurement procedure allows us to determine whether the two differences reflect the same difference in psychological satisfaction. Statisticians express this point by saying that the differences between adjacent scale values do not necessarily represent equal intervals on the underlying scale giving rise to the measurements. (In our case, the underlying scale is the true feeling of satisfaction, which we are trying to measure.)

Interval Scales

The interval level of measurement involves assigning scores using numerical scales in which intervals have the same interpretation throughout. As an example, consider either the Fahrenheit or Celsius temperature scales. The difference between 30 degrees and 40 degrees represents the same temperature difference as the difference between 80 degrees and 90 degrees. This is because each 10-degree interval has the same physical meaning (in terms of the kinetic energy of molecules).

Interval scales are not perfect, however. In particular, they do not have a true zero point even if one of the scaled values happens to carry the name “zero.” The Fahrenheit scale illustrates the issue. Zero degrees Fahrenheit does not represent the complete absence of temperature (the absence of any molecular kinetic energy). In reality, the label “zero” is applied to its temperature for quite accidental reasons connected to the history of temperature measurement. In psychology, the intelligence quotient (IQ) is often considered to be measured at the interval level. While it is technically possible to receive a score of 0 on an IQ test, such a score would not indicate the complete absence of IQ. Moreover, a person with an IQ score of 140 does not have twice the IQ of a person with a score of 70. However, the difference between IQ scores of 80 and 100 is the same as the difference between IQ scores of 120 and 140.

Ratio Scales

Finally, the ratio level of measurement involves assigning scores in such a way that there is a true zero point that represents the complete absence of the quantity. Height measured in meters and weight measured in kilograms are good examples. So are counts of discrete objects or events, such as the number of siblings one has or the number of questions a student answers correctly on an exam. You can think of a ratio scale as the three earlier scales rolled up in one. Like a nominal scale, it provides a name or category for each object (the numbers serve as labels). Like an ordinal scale, the objects are ordered (in terms of the ordering of the numbers). Like an interval scale, the same difference at two places on the scale has the same meaning. However, in addition, the same ratio at two places on the scale also carries the same meaning (see Table 5.2).

The Fahrenheit scale for temperature has an arbitrary zero point and is therefore not a ratio scale. However, zero on the Kelvin scale is absolute zero. This makes the Kelvin scale a ratio scale. For example, if one temperature is twice as high as another, as measured on the Kelvin scale, then it has twice the kinetic energy of the other temperature.

Another example of a ratio scale is the amount of money you have in your pocket right now (25 cents, 50 cents, etc.). Money is measured on a ratio scale because, in addition to having the properties of an interval scale, it has a true zero point: if you have zero money, this actually implies the absence of money. Since money has a true zero point, it makes sense to say that someone with 50 cents has twice as much money as someone with 25 cents.

Stevens’s levels of measurement are important for at least two reasons. First, they emphasize the generality of the concept of measurement. Although people do not normally think of categorizing or ranking individuals as measurement, in fact, they are as long as they are done so that they represent some characteristic of the individuals. Second, the levels of measurement can serve as a rough guide to the statistical procedures that can be used with the data and the conclusions that can be drawn from them. With nominal-level measurement, for example, the only available measure of central tendency is the mode. With ordinal-level measurement, the median or mode can be used as indicators of central tendency. Interval and ratio-level measurement are typically considered the most desirable because they permit for any indicators of central tendency to be computed (i.e., mean, median, or mode). Also, ratio-level measurement is the only level that allows meaningful statements about ratios of scores. Once again, one cannot say that someone with an IQ of 140 is twice as intelligent as someone with an IQ of 70 because IQ is measured at the interval level, but one can say that someone with six siblings has twice as many as someone with three because the number of siblings is measured at the ratio level.

Table 5.2 Summary of Levels of Measurement^[2]

Please watch: Variable Measurement Scales (Statistics Lectures)

Please watch: What Are Scales of Measurement? (Nominal, Ordinal, Interval, Ratio) (Daniel Storage)

Reliability

Measurement involves assigning scores to individuals so that they represent some characteristic of the individuals. But how do researchers know that the scores actually represent the characteristic, especially when it is a construct like intelligence, self-esteem, depression, or working memory capacity? The answer is that they conduct research using the measure to confirm that the scores make sense based on their understanding of the construct being measured. This is an extremely important point. Psychologists do not simply assume that their measures work. Instead, they collect data to demonstrate that they work. If their research does not demonstrate that a measure works, they stop using it.

As an informal example, imagine that you have been dieting for a month. Your clothes seem to be fitting more loosely, and several friends have asked if you have lost weight. If at this point your bathroom scale indicated that you had lost 10 pounds, this would make sense and you would continue to use the scale. But if it indicated that you had gained 10 pounds, you would rightly conclude that it was broken and either fix it or get rid of it. In evaluating a measurement method, psychologists consider two general dimensions that fall under construct validity: reliability and validity.

Reliability refers to the consistency of a measure. Psychologists consider three types of consistency: over time (test-retest reliability), across items (internal consistency), and across different researchers (inter-rater reliability).

Test-Retest Reliability

When researchers measure a construct that they assume to be consistent across time, then the scores they obtain should also be consistent across time. Test-retest reliability is the extent to which this is actually the case. For example, intelligence is generally thought to be consistent across time. A person who is highly intelligent today will be highly intelligent next week. This means that any good measure of intelligence should produce roughly the same scores for this individual next week as it does today. Clearly, a measure that produces highly inconsistent scores over time cannot be a very good measure of a construct that is supposed to be consistent.

Assessing test-retest reliability requires using the measure on a group of people at one time, using it again on the same group of people at a later time, and then looking at the test-retest correlation between the two sets of scores. In general, a test-retest correlation of +.80 or greater is considered to indicate good reliability.

Figure 5.2 Test-Retest Correlation Between Two Sets of Scores of Several College Students on the Rosenberg Self-Esteem Scale, Given Two Times a Week Apart^[3]

Again, high test-retest correlations make sense when the construct being measured is assumed to be consistent over time, which is the case for intelligence, self-esteem, and the Big Five personality dimensions. But other constructs are not assumed to be stable over time. The very nature of mood, for example, is that it changes. So a measure of mood that produced a low test-retest correlation over a period of a month would not be a cause for concern.

Internal Consistency

A second kind of reliability is internal consistency, which is the consistency of people’s responses across the items on a multiple-item measure. In general, all the items on such measures are supposed to reflect the same underlying construct, so people’s scores on those items should be correlated with each other. On the Rosenberg Self-Esteem Scale, people who agree that they are a person of worth should tend to agree that they have a number of good qualities. If people’s responses to the different items are not correlated with each other, then it would no longer make sense to claim that they are all measuring the same underlying construct. This is as true for behavioral and physiological measures as for self-report measures. For example, people might make a series of bets in a simulated game of roulette as a measure of their level of risk seeking. This measure would be internally consistent to the extent that individual participants’ bets were consistently high or low across trials.

Like test-retest reliability, internal consistency can only be assessed by collecting and analyzing data. One approach is to look at a split-half correlation. This involves splitting the items into two sets, such as the first and second halves of the items or the even- and odd-numbered items. Then a score is computed for each set of items, and the relationship between the two sets of scores is examined. A split-half correlation of +.80 or greater is generally considered good internal consistency.

Figure 5.2 Split-Half Correlation Between Several College Students’ Scores on the Even-Numbered Items and Their Scores on the Odd-Numbered Items of the Rosenberg Self-Esteem Scale^[4]

Perhaps the most common measure of internal consistency used by researchers in psychology is a statistic called Cronbach’s α (the Greek letter alpha). Conceptually, α is the mean of all possible split-half correlations for a set of items. For example, there are 252 ways to split a set of 10 items into two sets of five. Cronbach’s α would be the mean of the 252 split-half correlations. Note that this is not how α is actually computed, but it is a correct way of interpreting the meaning of this statistic. Again, a value of +.80 or greater is generally taken to indicate good internal consistency.

Please watch: Cronbach’s alpha (Simply Explained)

Inter-rater Reliability

Many behavioral measures involve significant judgment on the part of an observer or a rater. Inter-rater reliability is the extent to which different observers are consistent in their judgments. For example, if you were interested in measuring college students’ social skills, you could make video recordings of them as they interacted with another student whom they are meeting for the first time. Then you could have two or more observers watch the videos and rate each student’s level of social skills. To the extent that each participant does in fact have some level of social skills that can be detected by an attentive observer, different observers’ ratings should be highly correlated with each other. If they were not, then those ratings could not be an accurate representation of participants’ social skills. Inter-rater reliability is often assessed using Cronbach’s α when the judgments are quantitative or an analogous statistic called Cohen’s κ (the Greek letter kappa) when they are categorical.

Please watch: Forms of Reliability in Research and Statistics (Daniel Storage)

Validity

Validity is the extent to which the scores from a measure represent the variable they are intended to. But how do researchers make this judgment? We have already considered one factor that they take into account—reliability. When a measure has good test-retest reliability and internal consistency, researchers should be more confident that the scores represent what they are supposed to. There has to be more to it, however, because a measure can be extremely reliable but have no validity whatsoever. As an absurd example, imagine someone who believes that people’s index finger length reflects their self-esteem and therefore tries to measure self-esteem by holding a ruler up to people’s index fingers. Although this measure would have extremely good test-retest reliability, it would have absolutely no validity. The fact that one person’s index finger is a centimeter longer than another’s would indicate nothing about which one had higher self-esteem.

Textbook presentations of validity usually divide it into several distinct “types.” But a good way to interpret these types is that they are other kinds of evidence—in addition to reliability—that should be taken into account when judging the validity of a measure.

Face Validity

Face validity is the extent to which a measurement method appears “on its face” to measure the construct of interest. Most people would expect a self-esteem questionnaire to include items about whether they see themselves as a person of worth and whether they think they have good qualities. So a questionnaire that included these kinds of items would have good face validity. The finger-length method of measuring self-esteem, on the other hand, seems to have nothing to do with self-esteem and therefore has poor face validity. Although face validity can be assessed quantitatively—for example, by having a large sample of people rate a measure in terms of whether it appears to measure what it is intended to—it is usually assessed informally.

Face validity is at best a very weak kind of evidence that a measurement method is measuring what it is supposed to. One reason is that it is based on people’s intuitions about human behavior, which are frequently wrong. It is also the case that many established measures in psychology work quite well despite lacking face validity. The Minnesota Multiphasic Personality Inventory (MMPI) measures many personality characteristics and disorders by having people decide whether each of over 567 different statements applies to them—where many of the statements do not have any obvious relation to the construct that they measure.

Content Validity

Content validity is the extent to which a measure “covers” the construct of interest. For example, if a researcher conceptually defines test anxiety as involving both sympathetic nervous system activation (leading to nervous feelings) and negative thoughts, then his measure of test anxiety should include items about both nervous feelings and negative thoughts. Or consider that attitudes are usually defined as involving thoughts, feelings, and actions toward something. By this conceptual definition, a person has a positive attitude toward exercise to the extent that he or she thinks positive thoughts about exercising, feels good about exercising, and actually exercises. So to have good content validity, a measure of people’s attitudes toward exercise would have to reflect all three of these aspects. Like face validity, content validity is not usually assessed quantitatively. Instead, it is assessed by carefully checking the measurement method against the conceptual definition of the construct.

While face and content validity are good first steps at assessing if a measure is accurately assessing the construct of interest, they are nonetheless subjective in nature and typically considered non-empirical forms for assessing validity. The next three kinds of validity described below are considered empirical or more objective assessments of validity, and therefore tend to be more desirable for researchers.

Criterion Validity – Concurrent, Predictive, and Convergent

Criterion validity is the extent to which people’s scores on a measure are correlated with other variables (known as criteria) that one would expect them to be correlated with. For example, people’s scores on a new measure of test anxiety should be negatively correlated with their performance on an important school exam. If it were found that people’s scores were in fact negatively correlated with their exam performance, then this would be a piece of evidence that these scores really represent people’s test anxiety. But if it were found that people scored equally well on the exam regardless of their test anxiety scores, then this would cast doubt on the validity of the measure.

A criterion can be any variable that one has reason to think should be correlated with the construct being measured, and there will usually be many of them. For example, one would expect test anxiety scores to be negatively correlated with exam performance and course grades and positively correlated with general anxiety and with blood pressure during an exam. Or imagine that a researcher develops a new measure of physical risk taking. People’s scores on this measure should be correlated with their participation in “extreme” activities such as snowboarding and rock climbing, the number of speeding tickets they have received, and even the number of broken bones they have had over the years. When the criterion is measured at the same time as the construct, criterion validity is referred to as concurrent validity; however, when the criterion is measured at some point in the future (after the construct has been measured), it is referred to as predictive validity (because scores on the measure have “predicted” a future outcome).

Criteria can also include other measures of the same construct. For example, one would expect new measures of test anxiety or physical risk taking to be positively correlated with existing measures of the same constructs. This is known as convergent validity.

Assessing criterion validity requires collecting data using the measure. Researchers John Cacioppo and Richard Petty did this when they created their self-report Need for Cognition Scale to measure how much people value and engage in thinking (Cacioppo & Petty, 1982). In a series of studies, they showed that college faculty scored higher than assembly-line workers, that people’s scores were positively correlated with their scores on a standardized academic achievement test, and that their scores were negatively correlated with their scores on a measure of dogmatism (which represents a tendency toward obedience). In the years since it was created, the Need for Cognition Scale has been used in literally hundreds of studies and has been shown to be correlated with a wide variety of other variables, including the effectiveness of an advertisement, interest in politics, and juror decisions (Petty et al., 2009).

Discriminant Validity

Discriminant validity (also referred to as divergent validity) is the extent to which scores on a measure are not correlated with measures of variables that are conceptually distinct. For example, self-esteem is a general attitude toward the self that is fairly stable over time. It is not the same as mood, which is how good or bad one happens to be feeling right now. So people’s scores on a new measure of self-esteem should not be very highly correlated with their moods. If the new measure of self-esteem were highly correlated with a measure of mood, it could be argued that the new measure is not really measuring self-esteem; it is measuring mood instead.

When they created the Need for Cognition Scale, Cacioppo and Petty also provided evidence of discriminant validity by showing that people’s scores were not correlated with certain other variables. For example, they found only a weak correlation between people’s need for cognition and a measure of their cognitive style—the extent to which they tend to think analytically by breaking ideas into smaller parts or holistically in terms of “the big picture.” They also found no correlation between people’s need for cognition and measures of their test anxiety and their tendency to respond in socially desirable ways. All these low correlations provide evidence that the measure is reflecting a conceptually distinct construct.

Please watch: Forms of Validity in Research and Statistics (Daniel Storage)

Illustrating Reliability and Validity of a Measurement

Imagine that a construct we want to measure is the bullseye (the center) of a target, as illustrated by the bullseye in each of the diagrams below. Then imagine that we have a measure, such as a survey, a questionnaire, a laboratory task, a brain imaging device, or any of a myriad types of measurements we use in psychological science.

Now imagine that each time we use our measurement is like shooting an arrow toward the target. Because our goal is to measure each construct reliably and validly, we want our arrow shots to be as consistent (reliable) and accurate (valid) as possible. We want them to consistently (reliably) and accurately (validly) hit the target’s bullseye.

In the diagrams below (Figure 5.3), our arrow shots are illustrated with back dots. The larger the dots, the more of our arrows — our measurements — landed on that spot. Each diagram illustrates a relation between reliability and validity.

Figure 5.3 Summarizing the Connection Between Reliability and Validity in Psychological Measurement^[5]

Practical Strategies for Psychological Measurement

So far in this chapter, we have considered several basic ideas about the nature of psychological constructs and their measurement. But now imagine that you are in the position of actually having to measure a psychological construct for a research project. How should you proceed? Broadly speaking, there are four steps in the measurement process: (a) conceptually defining the construct, (b) operationally defining the construct, (c) implementing the measure, and (d) evaluating the measure. In this section, we will look at each of these steps in turn. It is important to note that creation of a psychological measure can be a long process and in some cases can even serve as its own research project when validating it on a national level.

Conceptually Defining the Construct

Having a clear and complete conceptual definition of a construct is a prerequisite for good measurement. For one thing, it allows you to make sound decisions about exactly how to measure the construct. If you had only a vague idea that you wanted to measure people’s “memory,” for example, you would have no way to choose whether you should have them remember a list of vocabulary words, a set of photographs, a newly learned skill, an experience from long ago, or have them remember to perform a task at a later time. Because psychologists now conceptualize memory as a set of semi-independent systems, you would have to be more precise about what you mean by “memory.” If you are interested in long-term episodic memory (memory for previous experiences), then having participants remember a list of words that they learned last week would make sense, but having them try to remember to execute a task in the future would not. In general, there is no substitute for reading the research literature on a construct and paying close attention to how others have defined it.

Operationally Defining the Construct

Once you have a conceptual definition of the construct you are interested in studying, it is time to operationally define the construct. Recall an operational definition is a definition of the variable in terms of precisely how it is to be measured. Since most variables are relatively abstract concepts that cannot be directly observed (e.g., stress), and observation is at the heart of the scientific method, conceptual definitions must be transformed into something that can be directly observed and measured. Most variables can be operationally defined in many different ways. For example, stress can be operationally defined as people’s scores on a stress scale such as the Perceived Stress Scale (Cohen et al., 1983), cortisol concentrations in their saliva, or the number of stressful life events they have recently experienced. As described below, operationally defining your variable(s) of interest may involve using an existing measure or creating your own measure.

Using an Existing Measure

It is usually a good idea to use an existing measure that has been used successfully in previous research. Among the advantages are that (a) you save the time and trouble of creating your own, (b) there is already some evidence that the measure is valid (if it has been used successfully), and (c) your results can more easily be compared with and combined with previous results. In fact, if there already exists a reliable and valid measure of a construct, other researchers might expect you to use it unless you have a good and clearly stated reason for not doing so.

If you choose to use an existing measure, you may still have to choose among several alternatives. You might choose the most common one, the one with the best evidence of reliability and validity, the one that best measures a particular aspect of a construct that you are interested in (e.g., a physiological measure of stress if you are most interested in its underlying physiology), or even the one that would be easiest to use. For example, the Ten-Item Personality Inventory (TIPI) is a self-report questionnaire that measures all the Big Five personality dimensions with just 10 items (Gosling et al., 2003). It is not as reliable or valid as longer and more comprehensive measures, but a researcher might choose to use it when testing time is severely limited.

When an existing measure was created primarily for use in scientific research, it is usually described in detail in a published research article and is free to use in your own research—with a proper citation. You might find that later researchers who use the same measure describe it only briefly but provide a reference to the original article, in which case, you would have to get the details from the original article. The American Psychological Association also publishes the Directory of Unpublished Experimental Measures and PsycTESTS, which are extensive catalogs/collections of measures that have been used in previous research. Many existing measures—especially those that have applications in clinical psychology—are proprietary. This means that a publisher owns the rights to them and that you would have to purchase them. These include many standard intelligence tests, the Beck Depression Inventory, and the Minnesota Multiphasic Personality Inventory (MMPI). Details about many of these measures and how to obtain them can be found in other reference books, including Tests in Print and the Mental Measurements Yearbook. There is a good chance you can find these reference books in your university library.

Creating Your Own Measure

Instead of using an existing measure, you might want to create your own. Perhaps there is no existing measure of the construct you are interested in, or existing ones are too difficult or time-consuming to use. Or perhaps you want to use a new measure specifically to see whether it works in the same way as existing measures—that is, to evaluate convergent validity. In this section, we consider some general issues in creating new measures that apply equally to self-report, behavioral, and physiological measures.

First, be aware that most new measures in psychology are really variations of existing measures, so you should still look to the research literature for ideas. Perhaps you can modify an existing questionnaire, create a paper-and-pencil version of a measure that is normally computerized (or vice versa), or adapt a measure that has traditionally been used for another purpose. For example, the famous Stroop task (Stroop, 1935)—in which people quickly name the colors that various color words are printed in—has been adapted for the study of social anxiety. People high in social anxiety are slower at color naming when the words have negative social connotations such as “stupid” (Amir et al., 2002).

When you create a new measure, you should strive for simplicity. Remember that your participants are not as interested in your research as you are and that they will vary widely in their ability to understand and carry out whatever task you give them. You should create a set of clear instructions using simple language that you can present in writing or read aloud (or both). It is also a good idea to include one or more practice items so that participants can become familiar with the task and to build in an opportunity for them to ask questions before continuing. It is also best to keep the measure brief to avoid boring or frustrating your participants to the point that their responses start to become less reliable and valid.

The need for brevity, however, needs to be weighed against the fact that it is nearly always better for a measure to include multiple items rather than a single item. There are two reasons for this. One is a matter of content validity. Multiple items are often required to cover a construct adequately. The other is a matter of reliability. People’s responses to single items can be influenced by all sorts of irrelevant factors—misunderstanding the particular item, a momentary distraction, or a simple error such as checking the wrong response option. But when several responses are summed or averaged, the effects of these irrelevant factors tend to cancel each other out to produce more reliable scores. Remember, however, that multiple items must be structured in a way that allows them to be combined into a single overall score by summing or averaging. To measure “financial responsibility,” a student might ask people about their annual income, obtain their credit score, and have them rate how “thrifty” they are—but there is no obvious way to combine these responses into an overall score. To create a true multiple-item measure, the student might instead ask people to rate the degree to which 10 statements about financial responsibility describe them on the same five-point scale.

Finally, the very best way to assure yourself that your measure has clear instructions, includes sufficient practice, and is an appropriate length is to test several people. Observe them as they complete the task, time them, and ask them afterward to comment on how easy or difficult it was, whether the instructions were clear, and anything else you might be wondering about. Obviously, it is better to discover problems with a measure before beginning any large-scale data collection.

Implementing the Measure

You will want to implement any measure in a way that maximizes its reliability and validity. In most cases, it is best to test everyone under similar conditions that, ideally, are quiet and free of distractions. Participants are often tested in groups because it is efficient, but be aware that it can create distractions that reduce the reliability and validity of the measure. As always, it is good to use previous research as a guide. If others have successfully tested people in groups using a particular measure, then you should consider doing it too.

Be aware also that people can react in a variety of ways to being measured, which reduces the reliability and validity of the scores. Although some disagreeable participants might intentionally respond in ways meant to disrupt a study, participant reactivity is more likely to take the opposite form. Agreeable participants might respond in ways they believe they are expected to. Some participants might engage in socially desirable responding, doing or saying things because they think it is the socially appropriate thing. For example, people with low self-esteem agree that they feel they are a person of worth, not because they really feel this way, but because they believe this is the socially appropriate response and do not want to look bad in the eyes of the researcher. Additionally, research studies can have built-in demand characteristics: subtle cues that reveal how the researcher expects participants to behave. For example, a participant whose attitude toward exercise is measured immediately after she is asked to read a passage about the dangers of heart disease might reasonably conclude that the passage was meant to improve her attitude. As a result, she might respond more favorably because she believes she is expected to by the researcher. Finally, your own expectations (as a researcher) can bias participants’ behaviors in unintended ways.

There are several precautions you can take to minimize these kinds of reactivity. One is to make the procedure as clear and brief as possible so that participants are not tempted to vent their frustrations on your results. Another is to guarantee participants’ anonymity and make clear to them that you are doing so. If you are testing them in groups, be sure that they are seated far enough apart that they cannot see each other’s responses. Give them all the same type of writing tool so that they cannot be identified by, for example, the pink glitter pen that they used. You can even allow them to seal completed questionnaires into individual envelopes or put them into a drop box where they immediately become mixed with others’ questionnaires. Although informed consent requires telling participants what they will be doing, it does not require revealing your hypothesis or other information that might suggest to participants how you expect them to respond. A questionnaire designed to measure financial responsibility need not be titled “Are You Financially Responsible?” It could be titled “Money Questionnaire” or have no title at all. Finally, the effects of your expectations can be minimized by arranging to have the measure administered by a helper who is “blind” or unaware of its intent or of any hypothesis being tested. Regardless of whether this is possible, you should standardize all interactions between researchers and participants—for example, by always reading the same set of instructions word for word.

Evaluating the Measure

Once you have used your measure on a sample of people and have a set of scores, you are in a position to evaluate it more thoroughly in terms of reliability and validity. Even if the measure has been used extensively by other researchers and has already shown evidence of reliability and validity, you should not assume that it worked as expected for your particular sample and under your particular testing conditions. Regardless, you now have additional evidence bearing on the reliability and validity of the measure, and it would make sense to add that evidence to the research literature.

In most research designs, it is not possible to assess test-retest reliability because participants are tested at only one time. For a new measure, you might design a study specifically to assess its test-retest reliability by testing the same set of participants at two separate times. In other cases, a study designed to answer a different question still allows for the assessment of test-retest reliability. For example, a psychology instructor might measure his students’ attitude toward critical thinking using the same measure at the beginning and end of the semester to see if there is any change. Even if there is no change, he could still look at the correlation between students’ scores at the two times to assess the measure’s test-retest reliability. It is also customary to assess internal consistency for any multiple-item measure—usually by looking at a split-half correlation or Cronbach’s α.

Criterion validity can be assessed in various ways. For example, if your study included more than one measure of the same construct or measures of conceptually distinct constructs, then you should look at the correlations among these measures to be sure that they fit your expectations. Note also that a successful experimental manipulation also provides evidence of criterion validity. Recall from the beginning of this chapter that MacDonald and Martineau manipulated participants’ moods by having them think either positive or negative thoughts, and after the manipulation, their mood measure showed a distinct difference between the two groups. This simultaneously provided evidence that their mood manipulation worked and that their mood measure was valid.

But what if your newly collected data casts doubt on the reliability or validity of your measure? The short answer is that you have to ask why. It could be that there is something wrong with your measure or how you administered it. It could be that there is something wrong with your conceptual definition. It could be that your experimental manipulation failed. For example, if a mood measure showed no difference between people who you instructed to think positive versus negative thoughts, maybe it is because the participants did not actually think the thoughts they were supposed to, or that the thoughts did not actually affect their moods. In short, it is “back to the drawing board” to revise the measure, revise the conceptual definition, or try a new manipulation.

Summary

• Measurement is the assignment of scores to individuals so that the scores represent some characteristic of the individuals. Psychological measurement can be achieved in a wide variety of ways, including self-report, behavioral, and physiological measures.
• Psychological constructs such as intelligence, self-esteem, and depression are variables that are not directly observable because they represent behavioral tendencies or complex patterns of behavior and internal processes. An important goal of scientific research is to conceptually define psychological constructs in ways that accurately describe them.
• For any conceptual definition of a construct, there will be many different operational definitions or ways of measuring it. The use of multiple operational definitions, or converging operations, is a common strategy in psychological research.
• Variables can be measured at four different levels—nominal, ordinal, interval, and ratio—that communicate increasing amounts of quantitative information. The level of measurement affects the kinds of statistics you can use and conclusions you can draw from your data.
• Psychological researchers do not simply assume that their measures work. Instead, they conduct research to show that they work. If they cannot show that they work, they stop using them.
• There are two distinct criteria by which researchers evaluate their measures: reliability and validity. Reliability is consistency across time (test-retest reliability), across items (internal consistency), and across researchers (inter-rater reliability). Validity is the extent to which the scores actually represent the variable they are intended to.
• Validity is a judgment based on various types of evidence. The relevant evidence includes the measure’s reliability, whether it covers the construct of interest, and whether the scores it produces are correlated with other variables they are expected to be correlated with and not correlated with variables that are conceptually distinct.
• Good measurement begins with a clear conceptual definition of the construct to be measured. This is accomplished both by clear and detailed thinking and by a review of the research literature.
• You often have the option of using an existing measure or creating a new measure. You should make this decision based on the availability of existing measures and their adequacy for your purposes.
• Several simple steps can be taken in creating new measures and in implementing both existing and new measures that can help maximize reliability and validity.
• Once you have used a measure, you should reevaluate its reliability and validity based on your new data. Remember that the assessment of reliability and validity is an ongoing process.

Media Attributions

^[1-4] Jhangiani, R.S., Chiang, I-C.A., Cuttler, C., & Leighton, D.C. (2019). Research methods in psychology. 4th edition. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

^[5] Gernsbacher, M.A. (n.d.). Open Access Active-Learning Research Methods Course. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Text Attributions

D’Costa, S., Ukeye, M., O’Neil, M., & Anguiano, R. (no date). Critical Research Methods in Psychology. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Dudley, M. (2019). Research Methods in Psychology. OER Commons. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Gernsbacher, M.A. (n.d.). Open Access Active-Learning Research Methods Course. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Jhangiani, R.S., Chiang, I-C.A., Cuttler, C., & Leighton, D.C. (2019). Research methods in psychology. 4th edition. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

References

Amir, N., Freshman, M., & Foa, E. (2002). Enhanced Stroop interference for threat in social phobia. Journal of Anxiety Disorders, 16, 1–9.

Bandura, A., Ross, D., & Ross, S. A. (1961). Transmission of aggression through imitation of aggressive models. Journal of Abnormal and Social Psychology, 63, 575–582.

Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality and Social Psychology, 42, 116–131.

Cohen, S., Kamarck, T., & Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24, 386-396.

Costa, P. T., Jr., & McCrae, R. R. (1992). Normal personality assessment in clinical practice: The NEO Personality Inventory. Psychological Assessment, 4, 5–13.

Delongis, A., Coyne, J. C., Dakof, G., Folkman, S., & Lazarus, R. S. (1982). Relationships of daily hassles, uplifts, and major life events to health status. Health Psychology, 1, 119-136.

Gosling, S. D., Rentfrow, P. J., & Swann, W. B., Jr. (2003). A very brief measure of the Big Five personality domains. Journal of Research in Personality, 37, 504–528.

Holmes, T. H., & Rahe, R. H. (1967). The Social Readjustment Rating Scale. Journal of Psychosomatic Research, 11, 213-218.

MacDonald, T. K., & Martineau, A. M. (2002). Self-esteem, mood, and intentions to use condoms: When does low self-esteem lead to risky health behaviors? Journal of Experimental Social Psychology, 38, 299–306.

Petty, R. E, Briñol, P., Loersch, C., & McCaslin, M. J. (2009). The need for cognition. In M. R. Leary & R. H. Hoyle (Eds.), Handbook of individual differences in social behavior (pp. 318–329). New York, NY: Guilford Press.

Rosenberg, M. (1989). Society and the adolescent self-image (rev. ed.). Middletown, CT: Wesleyan University Press.

Segerstrom, S. E., & Miller, G. E. (2004). Psychological stress and the human immune system: A meta-analytic study of 30 years of inquiry. Psychological Bulletin, 130, 601–630.

Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103, 677–680.

Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of Experimental Psychology, 18, 643–662.

Chapter 6: Sampling

Populations and Samples

We are usually interested in understanding a specific group of people. This group is known as the population of interest, or simply the population. The population is the collection of all people who have some characteristic in common; it can be as broad as “all people” if we have a very general research question about human psychology, or it can be extremely narrow, such as “all freshmen psychology majors at Midwestern public universities” if we have a specific group in mind.

In statistics, we often rely on a sample—that is, a small subset of a larger set of data—to draw inferences about the larger set. The larger set is known as the population from which the sample is drawn. Let’s say you have been hired by the National Election Commission to examine how the American people feel about the fairness of the voting procedures in the U.S. Whom will you ask? It is not practical to ask every single American how he or she feels about the fairness of the voting procedures. Instead, we query a relatively small number of Americans and draw inferences about the entire country from their responses. The Americans actually queried constitute our sample of the larger population of all Americans.

A sample is typically a small subset of the population. In the case of voting attitudes, we would sample a few thousand Americans drawn from the hundreds of millions that make up the country. In choosing a sample, it is therefore crucial that it not over-represent one kind of citizen at the expense of others. For example, something would be wrong with our sample if it happened to be made up entirely of Florida residents. If the sample held only Floridians, it could not be used to infer the attitudes of other Americans. The same problem would arise if the sample were composed only of Republicans. Inferences from statistics are based on the assumption that sampling is representative of the population. If the sample is not representative, then the possibility of sampling bias occurs. Sampling bias means that our conclusions apply only to our sample and are not generalizable to the full population.

To solidify your understanding of the importance of representative sampling, consider the following examples. Try to identify the population of interest, and then reflect on whether the sample is likely to yield the information desired.

Probability versus Nonprobability Sampling

Essentially all psychological research involves sampling—selecting a sample to study from the population of interest. Sampling falls into two broad categories. Probability sampling occurs when the researcher can specify the probability that each member of the population will be selected for the sample. It gives the likelihood of the sample being representative of the population.

Nonprobability sampling occurs when the researcher cannot specify these probabilities. Nonprobability sampling cannot assure the representativeness of a sample as well as probability sampling. However, it tends to be simpler and cheaper for researchers to conduct, and thus psychological research more often involves nonprobability sampling techniques. For example, convenience sampling—studying individuals who happen to be nearby and willing to participate—is a very common form of nonprobability sampling used in psychological research. University participant pools (such as through SONA) are examples of convenience sampling. Another form of nonprobability sampling is snowball sampling, in which existing research participants help recruit additional participants for the study. Quota sampling, in which subgroups in the sample are recruited to be proportional to those subgroups in the population, is also common. For example, a researcher might want their sample to have proportional quotas across religious groups and set pre-determined quotas across those groups (e.g., Christian, Jewish, Muslim, etc.). Another nonprobabily sampling technique is purposive sampling, in which only a certain type of people are sampled to fit the researcher’s research question based on specific, pre-determined criteria. As an example, if a researcher is studying the psychological impact of transitioning to remote work, they might not be interested in sampling from the general population. Instead, the researcher might set out purposeful criteria of who to sample from, such as IT managers and remote employees who recently transitioned to remote work. Finally, self-selection sampling is when individuals choose to take part in the research on their own accord, without being approached by the researcher directly. To clarify, self-selection sampling does not mean the researcher participates in their own study (a common misconception).

Survey researchers, however, are much more likely to use some form of probability sampling. This tendency is because the goal of most survey research is to make accurate estimates about what is true in a particular population, and these estimates are most accurate when based on a probability sample. For example, it is important for researchers to base their estimates of election outcomes—which are often decided by only a few percentage points—on probability samples of likely registered voters. However, probability sampling is more difficult to implement than nonprobability sampling. Therefore, some researchers may choose to use one of the nonprobability sampling techniques described since they tend to be easier to implement.

Researchers may also use nonprobability sampling techniques if generalizability is not a top priority. Generalizability is the extent to which the results of a study can be accurately applied to the population and is closely related to external validity (discussed in Module 5). If a researcher is trying to test an association or causal claim (establishing the relation between two variables of interest), then generalizability (and the utilization of probability sampling techniques) is usually a lower priority. In these cases, nonprobability sampling techniques may be preferred. However, if a researcher is trying to test a frequency claim (e.g., election outcomes), then generalizability is essential, making a probability sampling technique appropriate.

Probability Sampling Techniques

Compared with nonprobability sampling, probability sampling requires a very clear specification of the population, which of course depends on the research questions to be answered. The population might be all registered voters in the state of Arkansas, all American consumers who have purchased a car in the past year, women in the United States over 40 years old who have received a mammogram in the past decade, or all the alumni of a particular university. Once the population has been specified, probability sampling requires a sampling frame. This is essentially a list of all the members of the population from which to select the respondents. Sampling frames can come from a variety of sources, including telephone directories, lists of registered voters, and hospital or insurance records. In some cases, a map can serve as a sampling frame, allowing for the selection of cities, streets, or households.

Simple Random Sampling

There are a variety of different probability sampling methods. Simple random sampling is done in such a way that each individual in the population has an equal probability of being selected for the sample. It is a reliable method of obtaining information because every single member of a population is chosen randomly, merely by chance. This could involve putting the names of all individuals in the sampling frame into a hat, mixing them up, and then drawing out the number needed for the sample. Given that most sampling frames take the form of computer files, random sampling is more likely to involve computerized sorting or selection of respondents. A common approach in telephone surveys is random-digit dialing, in which a computer randomly generates phone numbers from among the possible phone numbers within a given geographic area. A specific advantage of simple random sampling is that it is the most straightforward method of probability sampling. A disadvantage is that you may not find enough individuals with your characteristic of interest, especially if that characteristic is uncommon.

Sometimes it is not feasible to build a sample using simple random sampling. To see the problem, consider the fact that both Dallas and Houston competed to be hosts of the 2012 Olympics. Imagine that you had been hired to assess whether most Texans preferred Houston to Dallas as the host, or the reverse. Given the impracticality of obtaining the opinion of every single Texan, you had to construct a sample of the Texas population. But notice how difficult it would have been to proceed by simple random sampling. For example, how would you have contacted those individuals who didn’t vote and didn’t have a phone? Even among people you found in the telephone book, how could you have identified those who had just relocated to another state (and had no reason to inform you of their move)? What would you have done about the fact that, since the beginning of the study, an additional 4,212 people took up residence in the state of Texas? As you can see, it is sometimes very difficult to develop a truly random procedure. For this reason, other kinds of sampling techniques have been devised.

Stratified Sampling

A common alternative to simple random sampling is stratified random sampling, in which the population is divided into different subgroups or “strata” (usually based on demographic characteristics) and then a random sample is taken from each “stratum.” Proportionate stratified random sampling can be used to select a sample in which the proportion of respondents in each of various subgroups matches the proportion in the population. For example, because about 12.5% of the US population is Black, stratified random sampling can be used to ensure that a survey of 1,000 American adults includes about 125 Black respondents.

Figure 6.1 Examples of Strat in a Population to be Sampled Via Stratified Random Sampling^[1]

Disproportionate stratified random sampling (also referred to as oversampling) can also be used to sample extra respondents from particularly small subgroups—allowing valid conclusions to be drawn about those subgroups. For example, because Asian Americans make up a fairly small percentage of the US population (about 4.5%), a simple random sample of 1,000 American adults might include too few Asian Americans to draw any conclusions about them as distinct from any other subgroup. If representation is important to the research question, however, then disproportionate stratified random sampling could be used to ensure that enough Asian American respondents are included in the sample to draw valid conclusions about Asian Americans as a whole.

Cluster Sampling

Yet another type of probability sampling is cluster sampling, in which larger clusters of individuals are randomly sampled. Demographic characteristics, such as race/ethnicity, gender, age, and zip code can be used to identify a cluster. Cluster sampling can be more efficient than simple random sampling, especially where a study takes place over a wide geographic region. Researchers can go a step further and engage in multistage sampling by randomly sampling individuals within each cluster. For example, to select a sample of small-town residents in the United States, a researcher might randomly select several small towns and then randomly select several individuals within each town. Cluster and multistage sampling are especially useful for surveys that involve face-to-face interviewing because it minimizes the amount of traveling that the interviewers must do. For example, instead of traveling to 200 small towns to interview 200 residents, a research team could travel to 10 small towns and interview 20 residents of each. The U.S. Census Bureau uses multistage sampling by first taking a simple random sample of counties in each state, then taking another simple random sample of households in each county, and collecting data on those households.

Please watch: Sampling Methods and Bias with Surveys (Crash Course Statistics #10)

Please watch: Sampling: Simple Random, Convenience, systematic, cluster, stratified - Statistics Help

Sample Size and Population Size

How large does a survey sample need to be? In general, this depends on two factors. One is the level of confidence in the result that the researcher wants. The larger the sample, the closer any statistic based on that sample will tend to be to the corresponding value in the population. The other factor is the budget of the study. Larger samples provide greater confidence, but they take more time, effort, and money to obtain. Taking these two factors into account, most survey research uses sample sizes that range from about 100 to about 1,000. Conducting a power analysis prior to launching the survey helps to guide the researcher in making this trade-off.

Statistical Power and Sample Size

Samples size and statistical power are two closely connected topics. The statistical power of a research design is the probability of rejecting the null hypothesis given the sample size and expected relation strength. For example, the statistical power of a study with 50 participants and an expected Pearson’s r of +.30 in the population is .59. That is, there is a 59% chance of rejecting the null hypothesis if indeed the population correlation is +.30. Statistical power is the complement of the probability of committing a Type II error. So in this example, the probability of committing a Type II error would be 1 − .59 = .41. Clearly, researchers should be interested in the power of their research designs if they want to avoid making Type II errors. In particular, they should make sure their research design has adequate power before collecting data. A common guideline is that a power of .80 is adequate. This guideline means that there is an 80% chance of rejecting the null hypothesis for the expected relationship strength.

The topic of how to compute power for various research designs and null hypothesis tests is beyond the scope of this book. However, there are online tools that allow you to do this by entering your sample size, expected relation strength, and α level for various hypothesis tests (see “Computing Power Online”). In addition, Table 6.1 shows the sample size needed to achieve a power of .80 for weak, medium, and strong relations for a two-tailed independent samples t-test and for a two-tailed test of Pearson’s r. Notice that weak relations require very large samples to provide adequate statistical power.

Table 6.1 Sample Size Needed to Achieve Statistical Power of .80 for Different Expected Relation Strengths for an Independent Samples t-Test and a Test of Pearson’s r ^[2]

What should you do if you discover that your research design does not have adequate power? Imagine, for example, that you are conducting a between-subjects experiment with 20 participants in each of two conditions and that you expect a medium difference (d = .50) in the population. The statistical power of this design is only .34. That is, even if there is a medium difference in the population, there is only about a one in three chance of rejecting the null hypothesis and about a two in three chance of committing a Type II error. Given the time and effort involved in conducting the study, this probably seems like an unacceptably low chance of rejecting the null hypothesis and an unacceptably high chance of committing a Type II error.

Given that statistical power depends primarily on relation strength and sample size, there are essentially two steps you can take to increase statistical power: increase the strength of the relation or increase the sample size. Increasing the strength of the relation can sometimes be accomplished by using a stronger manipulation or by more carefully controlling extraneous variables to reduce the amount of noise in the data (e.g., by using a within-subjects design rather than a between-subjects design). The usual strategy, however, is to increase the sample size. For any expected relation strength, there will always be some sample large enough to achieve adequate power.

Sampling Bias

Probability sampling was developed in large part to address the issue of sampling bias. Sampling bias occurs when a sample is selected in such a way that it is not representative of the entire population and therefore produces inaccurate results.

Sampling bias is an important consideration for researchers who seek to engage in social justice aligned research practices. Historically, many research samples consisted of White men but the findings from these studies were applied to individuals across racial groups. One key example was the creation of the body mass index (BMI) and its connection to health access. The original research to develop BMI was done in Europe with White men and women. Further research was done in the US in the 1970s but the sample size was still not racially diverse. Nevertheless, the findings were implemented across many racial and ethnic groups, globally. The implications of BMI have led to difficulties with health and insurance access for many racially marginalized communities. It highlights the importance of considering our sampling procedures to ensure that the generalizability of the findings are accurate.

There is one form of sampling bias that even careful random sampling is subject to. It is almost never the case that everyone selected for the sample actually responds to the survey. Some may have died or moved away, and others may decline to participate because they are too busy, are not interested in the survey topic, or do not participate in surveys on principle. If these survey non-responders differ from survey responders in systematic ways, then this can produce non-response bias. For example, in a mail survey on alcohol consumption, researcher Vivienne Lahaut and colleagues found that only about half the sample responded after the initial contact and two follow-up reminders (Lahaut et al., 2002). The danger here is that the half who responded might have different patterns of alcohol consumption than the half who did not, which could lead to inaccurate conclusions on the part of the researchers. So to test for non-response bias, the researchers later made unannounced visits to the homes of a subset of the non-responders—coming back up to five times if they did not find them at home. They found that the original non-responders included an especially high proportion of abstainers (nondrinkers), which meant that their estimates of alcohol consumption based only on the original responders were too high.

Although there are methods for statistically correcting for non-response bias, they are based on assumptions about the non-responders—for example, that they are more similar to late responders than to early responders—which may not be correct. For this reason, the best approach to minimizing non-response bias is to minimize the number of non-responders—that is, to maximize the response rate. There is a large research literature on the factors that affect survey response rates (Groves et al., 2004). In general, in-person interviews have the highest response rates, followed by telephone surveys, and then mail and Internet surveys. Among the other factors that increase response rates are sending potential respondents a short prenotification message informing them that they will be asked to participate in a survey in the near future and sending simple follow-up reminders to non-responders after a few weeks. The perceived length and complexity of the survey also makes a difference, which is why it is important to keep survey questionnaires as short, simple, and on topic as possible. Finally, offering an incentive—especially cash—is a reliable way to increase response rates. However, ethically, there are limits to offering incentives that may be so large as to be considered coercive.

Summary

• Research usually involves probability sampling, in which each member of the population has a known probability of being selected for the sample. Types of probability sampling include simple random sampling, stratified random sampling, and cluster sampling.
• Sampling bias occurs when a sample is selected in such a way that it is not representative of the population and therefore produces inaccurate results. The most pervasive form of sampling bias is non-response bias, which occurs when people who do not respond to the survey differ in important ways from people who do respond.
• The best way to minimize non-response bias is to maximize the response rate by prenotifying respondents, sending them reminders, constructing questionnaires that are short and easy to complete, and offering incentives.

Media Attributions

^[1] D’Costa, S., Ukeye, M., O’Neil, M., & Anguiano, R. (no date). Critical Research Methods in Psychology. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

^[2] Jhangiani, R.S., Chiang, I-C.A., Cuttler, C., & Leighton, D.C. (2019). Research methods in psychology. 4th edition. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Text Attributions

D’Costa, S., Ukeye, M., O’Neil, M., & Anguiano, R. (no date). Critical Research Methods in Psychology. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Dudley, M. (2019). Research Methods in Psychology. OER Commons. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Jhangiani, R.S., Chiang, I-C.A., Cuttler, C., & Leighton, D.C. (2019). Research methods in psychology. 4th edition. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

References

Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2004). Survey methodology. Hoboken, NJ: Wiley.

Lahaut, V. M. H. C. J., Jansen, H. A. M., van de Mheen, D., & Garretsen, H. F. L. (2002). Non-response bias in a sample survey on alcohol consumption. Alcohol and Alcoholism, 37, 256–260.

Module 3: Descriptive Statistics – A Refresher

Chapter 7: Descriptive and Inferential Statistics

What Are Statistics?

Statistics include numerical facts and figures. For instance:

• The largest earthquake measured 9.2 on the Richter scale.
• Men are at least 10 times more likely than women to commit murder.
• One in every eight South Africans is HIV positive.
• By the year 2050, there will be 12 people aged 65 and over for every new baby born.

The study of statistics involves math and relies upon calculations of numbers. But it also relies heavily on how the numbers are chosen and how the statistics are interpreted. For example, consider the following three scenarios and the interpretations based on the presented statistics. You will find that the numbers may be right, but the interpretation may be wrong. Try to identify a major flaw with each interpretation before we describe it.

1. A new advertisement for Ben & Jerry’s ice cream introduced in late May of last year resulted in a 30% increase in ice cream sales for the following three months. Thus, the advertisement was effective.

Major flaw: Ice cream consumption generally increases in the months of June, July, and August regardless of advertisements. This effect is called a history effect and leads people to interpret outcomes as the result of one variable when another variable (in this case, one having to do with the passage of time) is actually responsible.

1. The more churches in a city, the more crime there is. Thus, churches lead to crime.

Major flaw: Both increased churches and increased crime rates can be explained by larger populations. In bigger cities, there are both more churches and more crime. This is an example of the third-variable problem. Namely, a third variable can cause both situations; however, people erroneously believe that there is a causal relationship between the two primary variables rather than recognize that a third variable can cause both.

1. Seventy-five percent more interracial marriages are occurring this year than 25 years ago. Thus, our society now accepts interracial marriages.

Major flaw: We do not have all the information we need. What is the rate at which marriages are occurring? Suppose only 1% of marriages 25 years ago were interracial and so now 1.75% of marriages are interracial (1.75 is 75% higher than 1). This latter number is hardly evidence suggesting the acceptance of interracial marriages. In addition, the statistic provided does not rule out the possibility that the number of interracial marriages has seen dramatic fluctuations over the years and this year is not the highest. Again, there is simply not enough information to understand fully the impact of the statistics. As a whole, these examples show that statistics are not only facts and figures; they are something more than that. In the broadest sense, “statistics” refers to a range of techniques and procedures for analyzing, interpreting, displaying, and making decisions based on data. Statistics is the language of science and data. The ability to understand and communicate using statistics enables researchers from different labs, different languages, and different fields to articulate to one another exactly what they have found in their work. It is an objective, precise, and powerful tool in science and in everyday life.

Please watch: What Is Statistics: Crash Course Statistics #1

What a Statistics Course Is Not

Many psychology students dread the idea of taking a statistics course, and more than a few have changed majors upon learning that it is a requirement. That is because many students view statistics as only a math class, which is only partially true. While many of you will not believe this or agree with it, statistics in psychology isn’t just about math.

Although math is a central component of it, statistics is a broader way of organizing, interpreting, and communicating information in an objective manner. Indeed, great care has been taken to eliminate as much math from this course as possible (students who do not believe this are welcome to ask the professor what matrix algebra is). Statistics is a way of viewing reality as it exists around us in a way that we otherwise could not.

Why Do We Study Statistics?

To the surprise of many students, statistics is a fairly significant part of a psychological education. To the surprise of no-one, statistics is very rarely the favorite part of one’s psychological education. After all, if you really loved the idea of doing statistics, you would probably be enrolled in a statistics class right now, not a psychology class. So, not surprisingly, a pretty large proportion of psychology students probably are not happy about the fact that psychology has so much statistics in it. Maybe you are one of them (I see you, nodding your head vigorously in agreement!). In view of this, the right place to start might be to answer the most common questions that people have about statistics (commonly shortened to stats).

A big part of this issue at hand relates to the very idea of statistics. What are they? What are they there for? And why are scientists so obsessed with them? These are all good questions, when you think about it. Let us start with the last one. As a group, scientists seem to be fixated on running statistical tests on everything. In fact, we use statistics so often that we sometimes forget to explain to people why we do. It is a kind of article of faith among scientists – and especially social scientists – that your findings cannot be trusted until you have done some stats. Undergraduate students might be forgiven for thinking that psychology professors are completely mad, because no-one takes the time to answer one very simple question:

Why Do Statistics? Why Not Just Use “Common Sense”?

The best answer is a really simple one: We do not trust ourselves enough. We recognize that we are human and, therefore, susceptible to all of the biases, temptations, and frailties from which humans suffer. Statistics are basically a safeguard. Using “common sense” (who knows what that is really, anyway?) to evaluate evidence means trusting gut instincts, relying on verbal arguments, on using the raw power of human reason to come up with the right answer. Most scientists do not think this approach is likely to work or be very accurate, and scientists like accuracy and evidence.

Virtually every student of the behavioral sciences takes some form of statistics class. This is because statistics are how we communicate in science. It serves as the link between a research idea and usable conclusions. Without statistics, we would be unable to interpret the massive amounts of information contained in data. Even small datasets contain hundreds—if not thousands—of numbers, each representing a specific observation we made. Without a way to organize these numbers into a more interpretable form, we would be lost, having wasted the time and money of our participants, ourselves, and the communities we serve.

Beyond its use in science, however, there is a more personal reason to study statistics. Like most people, you probably feel that it is important to “take control of your life.” But, what does this mean? Partly, it means being able to properly evaluate the data and claims that bombard you every day. If you cannot distinguish good from faulty reasoning, then you are vulnerable to manipulation and to decisions that are not in your best interest. Statistics provides tools that you need in order to react intelligently to information you hear or read. In this sense, statistics is one of the most important things that you can study.

To be more specific, here are some claims that we have heard. (We are not saying that each one of these claims is true!)

• Four out of five dentists recommend Dentyne.
• Almost 85% of lung cancers in men and 45% in women are tobacco-related.
• Condoms are effective 94% of the time.
• People tend to be more persuasive when they look others directly in the eye and speak loudly and quickly.
• Women make 75 cents to every dollar a man makes when they work the same job.
• A surprising new study shows that eating egg whites can increase life span.
• People predict that it is very unlikely there will ever be another baseball player with a batting average over .400.
• There is an 80% chance that in a room full of 30 people, at least two people will share the same birthday.
• 79.48% of all statistics are made up on the spot.

All of these claims are statistical in character. We suspect that some of them sound familiar; if not, we bet that you have heard other claims like them. Notice how diverse the examples are. They come from psychology, health, law, sports, business, etc. Indeed, data (and data interpretation) show up in discourse from virtually every facet of contemporary life.

We hope that the discussion above helped explain why science in general is so focused on statistics. However, we are guessing that you have a lot more questions about what role statistics play in psychology, and specifically why psychology classes always devote so many lectures to stats. So, here is an attempt to answer a few of them…

Why Does Psychology Have So Many Statistics?

To be perfectly honest, many different reasons exist, some of which are better than others. The most important reason is that psychology is a statistical science. What I mean by that is that the “things” that we study are people. Real, complicated, gloriously messy, infuriatingly perverse people. The “things” of physics include objects like electrons, and while there are all sorts of complexities that arise in physics, electrons do not have minds of their own. They do not have opinions, they do not differ from each other in weird and arbitrary ways, they do not get bored in the middle of an experiment and scroll on their phones, and they do not get angry at the experimenter and then deliberately try to sabotage the data set (yes, this occasionally happens!). Most social sciences are desperately reliant on statistics. Not because we are bad experimenters, but because we have picked a harder problem to solve. We teach you stats because you really, really need it.

Can’t Someone Else Besides Me Do the Statistics?

To some extent, perhaps, but not completely. It is true that you do not need to become a fully trained statistician just to do psychology, but you do need to reach a certain level of statistical tolerance and competence. We have found four reasons that every psychology student ought to be able to understand and do basic statistics:

• First, if you want to be good at doing research, then you need to be able to understand psychological literature, right? Here is the catch: Almost every paper in psychological literature reports the results of statistical analyses. So, if you really want to understand psychology, you need to be able to understand what other people did with their data, and that means deciphering a certain amount of statistics.
• Second, if you want to conduct research: Statistics is deeply intertwined with research design. If you want to be good at designing psychological studies, you need to at the very least understand the basics of stats.
• Thirdly, a big practical problem exists with being dependent on other people to do all your statistics: Statistical analysis is expensive. If you ever get bored and want to look up how much the Australian government charges for university fees, you would notice something interesting: Statistics is designated as a “national priority” category, and so the fees are much, much lower than for any other area of study. There is a massive shortage of statisticians out there. So, if you were a psychological researcher, the laws of supply and demand are not exactly on your side here! As a result, in almost any real-life situation where you want to do psychological research, the cruel facts will be that you do not have enough money to afford a statistician. The economics of the situation mean that you have to be pretty self-sufficient.
• Lastly, a lot of these reasons generalize beyond researchers. If you want to be a practicing psychologist and stay on top of the field to do right by your clients, it helps to be able to read the scientific literature, which relies pretty heavily on statistics.

I Don’t Care About Jobs, Research, or Counseling Work. Do I Still Need Statistics?

Yes, we think it should matter to you, too. Statistics should matter to you personally in the same way that statistics should matter to everyone collectively. We live in the 21st century, and data are everywhere. Some would say science is under attack, fake news is rampant, AI systems often hallucinate, and our social media feeds us echo chambers of things it already thinks we want to hear. So, we can live blindly, or we can live smartly. Frankly, given the world in which we live these days, a basic knowledge of statistics is pretty much a survival tool!

Please watch Why Study Statistics in Psychology?

Statistics are often presented in an effort to add credibility to an argument or advice. You can see this by paying attention to television advertisements. Many of the numbers thrown about in this way do not represent careful statistical analysis. They can be misleading and push you into decisions that you might find cause to regret. For these reasons, learning about statistics is a long step toward taking control of your life. (It is not, of course, the only step needed to do so.) The purpose of this course, beyond preparing you for a career in psychology, is to help you learn statistical essentials. It will make you into an intelligent consumer of statistical claims.

Please watch How Statistics can be Misleading - Mark Liddell

You can take the first step right away. To be an intelligent consumer of statistics, your first reflex must be to understand and question the statistics you encounter. The British Prime Minister Benjamin Disraeli is quoted by Mark Twain as having said, “There are three kinds of lies—lies, damned lies, and statistics.” This quote reminds us why it is so important to understand statistics. So, let us invite you to reform your statistical habits from now on. No longer will you blindly accept numbers or findings. Instead, you will begin to think about the numbers, their sources, and most importantly, the procedures used to generate them.

The above section puts an emphasis on defending ourselves against fraudulent claims wrapped up as statistics, but let us also look at a more positive note. Just as important as detecting the deceptive use of statistics is the appreciation of the proper use of statistics. You must also learn to recognize statistical evidence that supports a stated conclusion. Statistics are all around you: sometimes used well, sometimes not. We must learn how to distinguish between the two cases. In doing so, statistics will likely be the course you use most in your day-to-day life, even if you do not ever run a formal analysis or do a research project again.

For those of you who are intending to go into some kind of helping profession with your psychology degree, think about it: Would you want your counselor to “go with their gut” and tell you whatever they feel like is best for you, or would you want a treatment or therapy that scientific studies have found to be helpful for clients with your condition? Would you want your insurance company to charge you whatever they want or give you a fair rate for your car insurance, based on your driving record and personal characteristics? If you wondered if your child had autism or ADHD, would you want your school counselor or school psychologist to use opinion or data and testing to make that diagnosis? Would you want your social worker or case manager to use an approach that has been shown to help people like you and those you care about and to help you find resources? If so, you can see the value of statistics in everyday professions as well, beyond the research scientists that make a career out of producing scientific manuscripts regarding surveys and experiment results.

Descriptive Statistics

Descriptive statistics are numbers that are used to summarize and describe data. The word “data” refers to the information that has been collected from an experiment, a survey, a historical record, etc. (By the way, data are plural. One piece of information is called a datum.) If we are analyzing birth certificates, for example, a descriptive statistic might be the percentage of certificates issued in New York State, or the average age of the mother. Any other number we choose to compute also counts as a descriptive statistic for the data from which the statistic is computed. Several descriptive statistics are often used at one time to give a full picture of the data.

Descriptive statistics are just that: descriptive. They describe and reflect what has happened. They do not involve generalizing or interpreting beyond the data at hand. Making conclusions from our data to another set of cases or individuals is the business of inferential statistics, which we will be studying in another chapter. Here, we focus on simply descriptive statistics.

Some descriptive statistics are shown in Table 7.1. The table shows the average salaries for various occupations in the United States in 1999. Descriptive statistics like these offer insight into American society. It is interesting to note, for example, that the people who educate our children and who protect our citizens are paid a great deal less than people who take care of our feet or our teeth.

Table 7.1 Average Salaries for Various U.S. Occupations in 1999^[1]

For more descriptive statistics, consider Table 7.2. It shows the number of unmarried men per 100 unmarried women in U.S. metro areas in 1990. From this table we see that men outnumber women most in Jacksonville, North Carolina, and women outnumber men most in Sarasota, Florida. You can see that descriptive statistics can be useful if you were looking for an opposite-sex partner! (These data come from the Information Please Almanac.)

Table 7.2 Number of Unmarried Men per 100 Unmarried Women in U.S. Metro Areas in 1990 (Note: Unmarried includes never–married, widowed, and divorced persons, 15 years or older)^[2]

These descriptive statistics may make us ponder why the numbers are so disparate in these cities. One potential explanation, for instance, as to why there are more women in Florida than men may involve the facts that older adults tend to move down to the Sarasota region and that women tend to outlive men. Thus, more women might live in Sarasota than men. However, in the absence of proper data, this is only speculation.

You probably know that descriptive statistics are central to the world of sports. Every sporting event produces numerous statistics, such as the shooting percentage of players on a basketball team. For the Olympic marathon (a foot race of 26.2 miles), we possess data that cover more than a century of competition. (The first modern Olympics took place in 1896.) Table 7.3 and Table 7.4 show the winning times for women and men, respectively. (Women have only been allowed to compete since 1984.)

Table 7.3 Women’s Winning Olympic Marathon Times, 1984–2004^[3]

Table 7.4 Men’s Winning Olympic Marathon Times, 1896–2004^[4]

There are many descriptive statistics that we can compute from the data in these tables. To gain insight into the improvement in speed over the years, let us divide the men’s times into two pieces, namely, the first 13 races (up to 1952) and the second 13 (starting from 1956). The mean winning time for the first 13 races is 2 hours, 44 minutes, and 22 seconds (written 2:44:22). The mean winning time for the second 13 races is 2:13:18. This is quite a difference (over half an hour). Does this prove that the fastest men are running faster? Or is the difference just due to chance, no more than what often emerges from chance differences in performance from year to year? We cannot answer this question with descriptive statistics alone. All we can affirm is that the two means are “suggestive.”

Examining Table 7.3 and Table 7.4 leads to many other questions. We note that Takahashi (the lead female runner in 2000) would have beaten the male runner in 1956 and all male runners in the first 12 marathons. This fact leads us to ask whether the gender gap will close or remain constant. When we look at the times within each gender, we also wonder how far they will decrease (if at all) in the next century of the Olympics. Might we one day witness a marathon winner finishing in under two hours? The study of statistics can help people make reasonable guesses about the answers to these questions.

It is also important to differentiate what we use to describe populations vs. what we use to describe samples. A population is described by a parameter; the parameter is the true value of the descriptive in the population, but one that we can never know for sure. For example, the Bureau of Labor Statistics once reported that the average hourly wage of chefs is $23.87. However, even if this number were computed using information from every single chef in the United States (making it a parameter), it would quickly become slightly off as one chef retires and a new chef enters the job market. Additionally, as noted above, there is virtually no way to collect data from every single person in a population. In order to understand a variable, we estimate the population parameter using a sample statistic. Here, the term statistic refers to the specific number we compute from the data (e.g., the average), not the field of statistics. A sample statistic is an estimate of the true population parameter, and if our sample represents the population well, then the statistic is considered to be a good and reasonable estimator of the parameter.

Even the best sample will be somewhat off from the full population, earlier referred to as sampling bias, and as a result, there will always be a tiny discrepancy between the parameter and the statistic we use to estimate it. This difference is known as sampling error, and, as we will see throughout the course, understanding sampling error is the key to understanding statistics. Every observation we make about a variable, be it a full research study or observing an individual’s behavior, is incapable of being completely representative of all possibilities for that variable. Knowing where to draw the line between an unusual observation and a true difference is what statistics as a field of study is all about.

Inferential Statistics

Descriptive statistics are wonderful at telling us what our data look like. However, what we often want to understand is how our data behave. What variables are related to other variables? Under what conditions will the value of a variable change? Are two groups different from each other, and if so, are people within each group different or similar? These are the questions answered by inferential statistics, and inferential statistics are how we generalize from our sample back up to our population or how we test ideas and questions about the world (otherwise known as hypotheses).

Module 4 is all about inferential statistics, the formal analyses and tests we run to make conclusions about our data and the world. For example, we will learn how to use an r statistic to look at the relationship between two factors. We will learn how to use a t statistic to determine whether people change over time or differ when enrolled in an intervention or experimental condition. We will also use an F statistic to determine if we can predict future values on a variable based on current known values of a variable. There are many types of inferential statistics, each allowing us insight into a different behavior of the data we collect. This course will only touch on a small subset (or a sample) of them, but the principles we learn along the way will make it easier to learn new tests, as most inferential statistics follow the same structure and format.

Mathematical Notation

As noted earlier, statistics is not just about math. It does, however, rely on logic and on math as a tool for logic. Many statistical formulas involve summing numbers. Fortunately, there is a convenient notation for expressing summation. This section covers the basics of this summation notation, for those whose professors might require hand calculations in their course.

Say we have a variable X that represents the weights (in grams) of 4 grapes:

Table 7.5 Weights of Grapes^[5]

We label the weight of Grape 1 as  of Grape 2 as , etc. The following formula means to sum up the weights of the four grapes:

The Greek letter Σ (sigma) indicates summation. The “i = 1” at the bottom indicates that the summation is to start with X1, and the 4 at the top indicates that the summation will end with X4. The “Xi” indicates that X is the variable to be summed as i goes from 1 to 4. Therefore:

As another example, the symbol

indicates that only the first 3 scores are to be summed. The index variable i goes from 1 to 3.

When all the scores of a variable (such as i) are to be summed, it is often convenient to use the following abbreviated notation:

Thus, when no values of i are shown, it means to sum all the values of X.

Many formulas involve squaring numbers before they are summed. This is indicated as:

Notice that:

because the expression on the left means to sum up all the values of X and then square the sum (19² = 361), whereas the expression on the right means to square the numbers and then sum the squares (90.54, as shown).

Some formulas involve the sum of cross products (multiplying). Below are the data for variables X and Y. The cross products (XY) are shown in the third column. The sum of the cross products is 3 + 4 + 21 = 28.

Table 7.6 The Sum of Cross-Products (Multiplying X and Y)^[6]

In summation notation, this is written as:

Wrapping it Up

Hopefully, this section has helped you see the importance of statistics as an important tool for scientific thinking for psychology students and practitioners, even if you are not a fan of math and research is not your jam. Statistics help us answer questions and make important decisions, both in our jobs and in everyday life.

Additional Resources

For tips and demonstrations on setting up and running statistical analyses in SPSS, please see Virginia Wickline: SPSS Statistics Helps (publishing dates vary).

Media Attributions

^[1-6] (CC BY-SA 4.0 by Cote et al., 2021)

Text Attributions

Cote, L. R., Gordon, R. G. Randell, C. E., Schmitt, J., & Marvin, H. (2021). Introduction to statistics in the psychological sciences. Pressbooks. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Navarro, D. J., & Foxcroft, D. R. (2025). Learning statistics with jamovi: A tutorial for beginners in statistical analysis. Open Book Publishers. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

References

CrashCourse (2018, January 24). What is statistics: Crash course statistics #1. YouTube. https://www.youtube.com/watch?v=sxQaBpKfDRk

Liddell, M. (2016, January 14). How statistics can be misleading - Mark Liddell. TED-Ed. https://www.youtube.com/watch?v=sxYrzzy3cq8

Storage, D. (2019, June 17). Why study statistics in psychology? YouTube. https://www.youtube.com/watch?v=nQ_5ta7_jyE

Chapter 8: Describing Data Using Distributions and Graphs

Before we can understand our analyses, we must first understand our data. The first step in doing this is using tables, charts, graphs, plots, and other visual tools to see what our data look like. We will not go over every possible graph out there in this chapter, but we will highlight some of the most common ones. Note that the words chart, graph, and plot can be used interchangeably, and we tend to call these figures. Figures can include any kind of image, but all kinds of graphs would be a kind of figure. Figures are different than tables, which are data collected in rows and columns of information. Tables are not figures, and figures are not tables. Tables and figures both show data, but they are different visual ways of doing so.

Graphing Categorical Variables

When Apple Computer introduced the iMac computer in August 1998, the company wanted to learn whether the iMac was expanding Apple’s market share. Was the iMac just attracting previous Macintosh owners? Or was it purchased by newcomers to the computer market and by previous Windows users who were switching over? To find out, 500 iMac customers were interviewed. Each customer was categorized as a previous Macintosh owner, a previous Windows owner, or a new computer purchaser.

This section examines graphical methods for displaying the results of the interviews. We will learn some general lessons about how to graph data that fall into a small number of categories (otherwise known as categorical data or qualitative data). The key point about the categorical data in the present section is that they do not come with a pre-established ordering (the way numbers are ordered). For example, there is no natural sense in which the category of previous Windows users comes before or after the category of previous Macintosh users. Categorical data are like bins for sorting trash. They are assigned a number, but the numbering is arbitrary.

Figure 8.1 Bins for recycling different kinds of materials^[1]

A later section will consider how to graph numerical data in which each observation is represented by a number in some range (otherwise known as continuous data or quantitative data). Continuous data, for example, could include something like height. People of one height are naturally ordered with respect to people of a different height, from shortest to tallest.

Figure 8.2 Original members of the legendary rock band, Kiss, ordered from shortest to tallest^[2]

Frequency Distributions

Frequency means how often something happens. It is a count of the total number of responses for a given condition or choice. When we collect data, we often want an idea of who said or did what, and we want to show those responses as a batch. In other words, we want to create a frequency distribution that shows each of the responses and how many individuals represent each option. Frequency distributions can be shown in two ways: tables (with rows and columns of data) or figures (graphs or charts).

Frequency Tables

All of the graphical methods shown in this section are first derived from frequency tables. Table 8.1 shows a frequency table for the results of the iMac study; it shows the frequencies of the various response categories. It also shows the relative frequency, which is the proportion of responses in each category. For example, the relative frequency (.17) for “none” is 85 (people that picked “none”) divided by 500 (the total number of respondents): 85/500 = .17.

Table 8.1 Frequency Table for the iMac Data^[3]

Pie Charts

The pie chart in Figure 8.3 shows the results of the iMac study. In a pie chart, each category is represented by a slice of the pie. The area of the slice is proportional to the percentage of responses in the category. This is simply the relative frequency multiplied by 100. Although most iMac purchasers were Macintosh owners (71%), Apple was encouraged by the 12% of purchasers who were former Windows users, and by the 17% of purchasers who were buying a computer for the first time.

Figure 8.3 Pie chart of iMac purchases illustrating frequencies of previous computer ownership: 71% of purchasers owned a Macintosh before buying their iMac^[4]

Pie charts are effective for displaying the relative frequencies of a small number of categories. They are not recommended, however, when you have a large number of categories. Pie charts can also be confusing when they are used to compare the outcomes of two different surveys or experiments. In an influential book on the use of graphs, Edward Tufte (1983) asserted, “The only worse design than a pie chart is several of them” (p. 178).

Here is another important point about pie charts. If they are based on a small number of observations, it can be misleading to label the pie slices with percentages. For example, if just 5 people had been interviewed by Apple Computers, and 3 were former Windows users, it would be misleading to display a pie chart with the Windows slice showing 60%. With so few people interviewed, such a large percentage of Windows users might easily have occurred since chance can cause large errors with small samples. In this case, it is better to alert the user of the pie chart to the actual numbers involved. The slices should therefore be labeled with the actual frequencies observed (e.g., 3) instead of with percentages.

Bar Charts

Bar charts can also be used to represent frequencies of different categories. Think of bar charts like separate bins: In which bin do you put your trash versus recycling versus composting? Or how can you sort your laundry into bins to wash it more effectively, with like items in each load (e.g., lights, darks, towels, delicates, dog stuff)? A bar chart of the iMac purchases is shown in Figure 8.4. Notice that there are gaps between the bars because each group is unique. Frequencies are shown on the y-axis down the side (also known as the ordinate) and the type of computer previously owned is shown on the x-axis across the bottom (also known as the abscissa). Typically, the y-axis shows the number of observations (frequency) in each category rather than the percentage of observations in each category as is typical in pie charts.

Figure 8.4 Bar chart of iMac purchases as a function of previous computer ownership^[5]

Comparing Distributions

Often, we need to compare the results of different surveys, or of different conditions within the same overall survey. In this case, we are comparing the distributions or collections of responses between the surveys or conditions. Bar charts are also excellent for illustrating differences between two distributions. Figure 8.5 shows the number of people playing card games at the Yahoo web site on a Sunday and on a Wednesday in the spring of 2001. We see that there were more players overall on Wednesday compared to Sunday. The number of people playing Pinochle was nonetheless the same on these two days. In contrast, there were about twice as many people playing Hearts on Wednesday as on Sunday. Facts like these emerge clearly from a well-designed bar chart.

The bars in Figure 8.5 are oriented horizontally rather than vertically. The horizontal format is useful with many categories because there is more room for the category labels. There will be more about bar charts when we consider numerical quantities later in this chapter.

Figure 8.5 A bar chart of the number of people playing different card games on Sunday and Wednesday^[6]

Please watch Charts Are Like Pasta - Data Visualization Part 1: Crash Course Statistics #5

Some Graphical Mistakes to Avoid

First mistake: Don’t get too fancy! Sometimes, more is less. In many instances, people add features to graphs that do not help to convey their information. For example, three-dimensional bar charts such as the one shown in Figure 8.6 are not necessary and are usually not as effective as their two-dimensional counterparts.

Figure 8.6 A three-dimensional version of Figure 8.4. Charts like this are less effective^[7]

Here is another way that fanciness can lead to trouble. Instead of plain bars, it is tempting to substitute meaningful or intriguing images. For example, Figure 8.7 presents the iMac data using pictures of computers. The heights of the pictures accurately represent the number of buyers, yet Figure 8.7 is misleading because the viewer’s attention will be captured by areas. The areas can exaggerate the size differences between the groups. In terms of percentages, the ratio of previous Macintosh owners to previous Windows owners is about 6 to 1. But the ratio of the two areas in Figure 8.7 is about 35 to 1. A biased person wishing to hide the fact that many Windows owners purchased iMacs would be tempted to use Figure 8.7 instead of Figure 8.4!

Figure 8.7 A redrawing of Figure 8.4 with a lie factor greater than 8^[8]

Tufte (1983) coined the term lie factor to refer to the ratio of the size of the effect shown in a graph to the size of the effect shown in the data. He suggests that lie factors greater than 1.05 or less than 0.95 produce unacceptable distortion.

Another distortion in bar charts results from setting the baseline to a value other than zero. The baseline is the bottom of the y-axis, representing the least number of cases that could have occurred in a category. Normally, but not always, this number should be zero. Figure 8.8 shows the iMac data with a baseline of 50 on the y-axis instead of zero. Once again, the differences in areas suggest a different story than the true differences in percentages. The number of Windows-switchers seems minuscule compared to its true value of 12%. In everyday life, watch out for graphs that do not start their baseline at zero.

Figure 8.8 A redrawing of Figure 8.4 with a baseline of 50^[9]

Finally, we note that it is a serious mistake to use a line graph when the x-axis contains categorical variables. A line graph is essentially a bar graph with the tops of the bars represented by points joined by lines (the rest of the bar is suppressed). Figure 8.9 inappropriately shows a line graph of the card game data from Yahoo that was presented in Figure 8.5. The drawback to Figure 8.9 is that it gives the false impression that the games are naturally ordered in a numerical way when, in fact, they are ordered alphabetically.

Figure 8.9 A line graph used inappropriately to depict the number of people playing different card games on Sunday and Wednesday^[10]

Please watch: How to Spot a Misleading Graph – Lea Gaslowitz

Summary

Pie charts and bar charts can both be effective methods of portraying categorical data. Bar charts are better when there are more than just a few categories and for comparing two or more distributions. Be careful to avoid creating misleading graphs.

Graphing Continuous Variables

Continuous variables are measured on a numeric scale. Height, weight, response time, subjective rating of pain, temperature, and score on an exam are all examples of continuous variables. Continuous variables are distinguished from categorical variables (sometimes also called nominal variables), such as favorite color, religion, city of birth, and favorite sport, in which there is no ordering or measuring involved, even though a number is assigned arbitrarily.

Many types of graphs can be used to portray distributions of continuous variables. The upcoming sections cover the following types of graphs: (1) stem-and-leaf displays, (2) histograms, (3) box plots, (4) bar charts, (5) line graphs, and (6) scatter plots (discussed in the section on correlation, also sometimes spelled scatterplots). Some graph types, such as stem-and-leaf displays, are best-suited for small to moderate amounts of data, whereas others, such as histograms, are best-suited for large amounts of data. Graph types such as box plots or bar charts are good as frequency distributions or for depicting differences between distributions. Scatterplots are used to show the relationship between two variables.

Stem-and-Leaf Displays

A stem-and-leaf display is a graphical method of displaying data that is particularly useful when data are not too numerous. In this section, we will explain how to construct and interpret this kind of graph.

As usual, we will start with an example. Consider Figure 8.10, which shows the number of touchdown passes (TD passes) thrown by each of the 31 teams in the National Football League during the 2000 season.

Figure 8.10 Number of touchdown passes^[11]

A stem-and-leaf display of the data is shown in Figure 8.11. The left portion of Figure 8.11 contains the stems. They are the numbers 3, 2, 1, and 0, arranged as a column to the left of the bars. Think of these numbers as 10s digits. A stem of 3, for example, can be used to represent the 10s digit in any of the numbers from 30 to 39. The numbers to the right of the bar are leaves, and they represent the 1s digits. Every leaf in the graph therefore stands for the result of adding the leaf to 10 times its stem.

Figure 8.11 Stem-and-leaf display of the number of touchdown passes^[12]

To make this clear, let us examine Figure 8.11 more closely. In the top row, the four leaves to the right of stem 3 are 2, 3, 3, and 7. Combined with the stem, these leaves represent the numbers 32, 33, 33, and 37, which are the numbers of TD passes for the first four teams in Figure 8.10. The next row has a stem of 2 and 12 leaves. Together, they represent 12 data points, namely, two occurrences of 20 TD passes, three occurrences of 21 TD passes, three occurrences of 22 TD passes, one occurrence of 23 TD passes, two occurrences of 28 TD passes, and one occurrence of 29 TD passes. We leave it to you to figure out what the third row represents. The fourth row has a stem of 0 and two leaves. It stands for the last two entries in Figure 8.10, namely 9 TD passes and 6 TD passes. (The latter two numbers may be thought of as 09 and 06.)

One purpose of a stem-and-leaf display is to clarify the shape of the distribution. You can see many facts about TD passes more easily in Figure 8.11 than in Figure 8.10. For example, by looking at the stems and the shape of the plot, you can tell that most of the teams had between 10 and 29 passing TDs, with a few having more and a few having less. The precise numbers of TD passes can be determined by examining the leaves.

There are two things about the football data that make them easy to graph with stems and leaves. First, the data are limited to whole numbers that can be represented with a one-digit stem and a one-digit leaf. Second, all the numbers are positive. If the data include numbers with three or more digits or contain decimals, they can be rounded to two-digit accuracy. Negative values are also easily handled.

Although stem-and-leaf displays are unwieldy for large datasets, they are often useful for datasets with up to 200 observations. Whether your data can be suitably represented by a stem-and-leaf display depends on whether they can be rounded without loss of important information. Also, all values must be two successive digits. Deciding what kind of graph is best suited to displaying your data thus requires good judgment. Statistics is not just recipes!

Box Plots

Box plots (sometimes also called box-and-whisker plots) are useful for identifying extreme scores and for comparing distributions. We will explain box plots with the help of data from a small experiment. Students in Introductory Statistics were presented with a page containing 30 colored rectangles. Their task was to name the colors as quickly as possible. Their times (in seconds) were recorded. We will compare the scores for the 16 men and 31 women who participated in the experiment by making separate box plots for each gender. Such a display is said to involve parallel box plots. The data for the women in our sample are shown in Figure 8.12.

Figure 8.12 Women’s times, raw data^[13]

There are several steps in constructing a box plot. The first relies on the 25th, 50th, and 75th percentiles in the distribution of scores. Figure 8.13 shows how these three statistics are used. For each gender we draw a box extending from the 25th percentile to the 75th percentile. The 50th percentile (otherwise known as the median, see Module 4) is drawn inside the box. Therefore, the bottom of each box is the 25th percentile, the top is the 75th percentile, and the line in the middle is the 50th percentile. Together, this box is otherwise known as the interquartile range, see Module 4).

Figure 8.13 The first step in creating box plots^[14]

For the data reflecting the women’s times, the 25th percentile is 17, the 50th percentile is 19, and the 75th percentile is 20. For the men (whose data are not shown), the 25th percentile is 19, the 50th percentile is 22.5, and the 75th percentile is 25.5.

Continuing with the box plots, we put “whiskers” above and below each box to give additional information about the spread of data. Whiskers are vertical lines that end in a horizontal stroke. Whiskers are drawn at 1.5 times the interquartile range, which is the 75^th percentile minus the 25^th percentile.

Figure 8.14 The box plots with the whiskers drawn^[15]

Although whiskers are not drawn all the way to outside or far-out values, we still wish to represent them in our box plots. This is achieved by adding additional marks beyond the whiskers. Specifically, outside values are indicated by small circles (⁰), and far out values are indicated by asterisks (*). In our data, there are no far-out values and just one outside value. This outside value of 29 is for the women and is shown in Figure 8.15.

Figure 8.15 The box plots with the outside value shown^[16]

Box plots provide basic information about a distribution. For example, a distribution with a that is heavier on the upper end would have a longer whisker in the positive direction than in the negative direction. Box plots are good at portraying extreme values and are especially good at showing differences between distributions. However, many of the details of a distribution are not revealed in a box plot; to examine these details one should create a histogram and/or a stem-and-leaf display.

Bar Charts

In the section on categorical variables, we saw how bar charts could be used to illustrate the frequencies of different categories. For example, as we saw earlier in this chapter, the bar chart shown in Figure 8.4 shows how many purchasers of iMac computers were previous Macintosh users, previous Windows users, and new computer purchasers.

In this section we show how bar charts can be used with two variables to present other kinds of continuous information, not just frequency counts. The bar chart in Figure 8.16 shows the percent increases in the Dow Jones, Standard & Poor 500 (S&P), and Nasdaq stock indexes from May 24, 2000, to May 24, 2001. Notice that both the S&P and the Nasdaq had “negative increases” which means that they decreased in value. In this bar chart, the y-axis is not frequency but rather another variable, the percentage increase.

Figure 8.16 Percent increase in three stock indexes from May 24, 2000, to May 24, 2001^[17]

Bar charts are particularly effective for showing change over time. Figure 8.17 for example, shows the percent increase in the Consumer Price Index (CPI) over four three-month periods. The fluctuation in inflation is apparent in the graph.

Figure 8.17 Percent change in the Consumer Price Index (CPI) over time. Each bar represents percent increase for the three months ending at the date indicated^[18]

Bar charts are often used to compare the means of different experimental conditions. Figure 8.18 shows the mean time it took one person to move the cursor to either a small target or a large target. On average, more time was required for small targets than for large ones.

Figure 8.18. Bar chart showing the means for the two conditions.^[19]

Line Graphs

A line graph is like a bar graph but with the tops of the bars represented by points joined by lines (and the rest of the bar is suppressed). For example, Figure 8.17, which was presented in the section on bar charts, shows changes in the Consumer Price Index (CPI) over time. A line graph of these same data is shown in Figure 8.19. Although the figures are similar, the line graph emphasizes the change from period to period.

Figure 8.19 A line graph of the percent change in the CPI over time. Each point represents percent increase for the three months ending at the date indicated^[20]

Line graphs are appropriate only when both the x– and y-axes display continuous, ordered (rather than categorical) variables. Although bar charts can also be used in this situation, line graphs are generally better at comparing changes over time.  Figure 8.20, for example, shows percent increases and decreases in five components of the CPI. The figure makes it easy to see that medical costs had a steadier progression than the other components. Although you could create an analogous bar chart, its interpretation would not be as easy.

Figure 8.20 A line graph of the percent change in five components of the CPI over time^[21]

Let us stress again that it is misleading to use a line graph when the x-axis contains merely categorical variables. As we saw earlier in this chapter, Figure 8.9, inappropriately shows a line graph of the card game data from Yahoo, discussed in the section on qualitative variables. The defect in Figure 8.9 is that it gives the false impression that the games are naturally ordered in a numerical way, whereas Figure 8.20 is appropriate because we are looking at differences across time.

Histograms

A histogram is a graphical method for displaying the shape of a distribution of continuous scores (interval or ratio data). It is particularly useful when there are a large number of observations. We begin with an example consisting of the scores of 642 students on a psychology test. The test consists of 197 items, each graded as “correct” or “incorrect.” The students’ scores, therefore, ranged from 46 to 167.

The first step is to create a frequency table. Unfortunately, a simple frequency table would be too big, containing over 100 rows. To simplify the table, we group scores together as shown in Table 8.2, called a grouped frequency table.

Table 8.2 Grouped Frequency Distribution of Psychology Test Scores^[22]

To create this table, the range of scores was broken into bins or intervals, called class intervals. The first interval is from 39.5 to 49.5, the second from 49.5 to 59.5, etc. Next, the number of scores falling into each interval was counted to obtain the class frequencies. There are 3 scores in the first interval, 10 in the second, etc.

Class intervals of width 10 provide enough detail about the distribution to be revealing without making the graph too “choppy.” More information on choosing the widths of class intervals is presented later in this section. Placing the limits of the class intervals midway between two numbers (e.g., 49.5) ensures that every score will fall in an interval rather than on the boundary between intervals.

In a histogram, the class frequencies are represented by bars. The height of each bar corresponds to its class frequency. Although a histogram uses bars, it is not called a bar chart because it shows continuous rather than categorical data, so the bars are connected, rather than distinct groups. A histogram of these psychology test score data is shown in Figure 8.21.

The histogram makes it plain that most of the scores are in the middle of the distribution, with fewer scores in the extremes. You can also see that the distribution is not symmetric: the scores extend farther to the right than they do to the left. The distribution is therefore said to be skewed. (We’ll have more to say about shapes of distributions below.)

Figure 8.21 Histogram of scores on a psychology test^[23]

In our example, the observations are whole numbers. Histograms can also be used when the scores are measured on a more continuous scale with decimals such as the length of time (in milliseconds) required to perform a task. In this case, there is no need to worry about fence sitters since they are improbable. (It would be quite a coincidence for a task to require exactly 7 seconds, measured to the nearest thousandth of a second.) We are therefore free to choose whole numbers as boundaries for our class intervals, for example, 4000, 5000, etc. The class frequency is then the number of observations that are greater than or equal to the lower bound, and strictly less than the upper bound. For example, one interval might hold times from 4000 to 4999 milliseconds. Using whole numbers as boundaries avoids a cluttered appearance and is the practice of many computer programs that create histograms. Note also that some computer programs label the middle of each interval rather than the end points.

Histograms can be based on relative frequencies instead of actual frequencies. Histograms based on relative frequencies show the proportion of scores in each interval rather than the number of scores. In this case, the y-axis runs from 0 to 1 (or somewhere in between if there are no extreme proportions). You can change a histogram based on frequencies to one based on relative frequencies by (a) dividing each class frequency by the total number of observations, and then (b) plotting the quotients on the y-axis (labeled as proportion).

There is more to be said about the widths of the class intervals, sometimes called bin widths. Your choice of bin width determines the number of class intervals. This decision, along with the choice of starting point for the first interval, affects the shape of the histogram. The best advice is to experiment with different choices of width, and to choose a histogram according to how well it communicates the shape of the distribution.

Please watch: Plots, Outliers, and Justin Timberlake: Data Visualization Part 2: Crash Course Statistics #6

Shapes of Continuous Distributions

Finally, it is useful to discuss how we describe the shapes of distributions, which we will revisit in future chapters to learn how different shapes affect our numerical descriptors of data and distributions.

The primary characteristic we are concerned about when assessing the shape of a distribution is whether the distribution is symmetrical or skewed. A symmetrical distribution, as the name suggests, when cut down the center forms two mirror images. Consider the two sides of butterfly wings, which look almost identical. Although in practice we will never get a perfectly symmetrical distribution, we would like our data to be as close to symmetrical as possible, otherwise known as approximately normal, for reasons we delve into in future sections. Many types of distributions are symmetrical, but by far the most common and pertinent distribution at this point is the normal distribution, shown in Figure 8.22. Notice that although the symmetry is not absolutely perfect (for instance, the bar just to the right of the center is taller than the one just to the left), the two sides are roughly the same shape. The normal distribution has a single peak, known as the center, and two ends that extend out equally, forming what is known as a bell shape or bell curve.

Figure 8.22 A symmetrical distribution^[24]

Symmetrical distributions can also have multiple peaks. Figure 8.23 shows a bimodal distribution, named for the two peaks that lie roughly symmetrically on either side of the center point (think of a Bactrian camel, with two humps rather than one). As we will see in future chapters, this is not a particularly desirable characteristic of our data, and, worse, this is a relatively difficult characteristic to detect numerically. Thus, it is important to visualize your data by looking at the distribution before moving ahead with any formal data analyses, which means summarizing and answering questions with your data.

Figure 8.23 A bimodal distribution^[25]

Skew and Kurtosis

There are two more descriptive statistics that are sometimes reported in the psychological literature: skew and kurtosis. In practice, neither one is used anywhere near as frequently as the measures of central tendency and variability. Distributions that are not symmetrical also come in many forms, more than can be described here. The most common asymmetry to be encountered is referred to as skew, in which one of the two tails (ends) of the distribution is disproportionately longer than the other. This property can affect the value of the averages we use in our analyses and make them an inaccurate representation of our data, which causes many problems. Skew can either be positive (right skew, to the right) or negative (left skew, to the left), based on which tail is longer. It is very easy to get the two confused at first; many students want to describe the skew by where the bulk of the data (larger portion of the histogram, known as the body) is placed, but the correct determination is based on which tail is longer. You can think of the tail as an arrow or ski slope; whichever direction the arrow or slope is pointing is the direction of the skew. Figure 8.24 shows positive (right) and negative (left) skew, respectively.

Figure 8.24 Positively skewed (A) and negatively skewed (B) distributions^[26]

As another example, Figure 8.25 illustrates that if the data tend to have a lot of extreme small values (i.e., the lower tail is “longer” than the upper tail) and not so many extremely large values (left panel), the data are negatively skewed, with a tail stretching to the left. On the other hand, if there are more extremely large values than extremely small ones (right panel), the data are positively skewed, with a tail stretching to the right. A symmetric distribution has a skewness of 0. The skewness value for a positively skewed distribution is positive, and a negative value for a negatively skewed distribution.

Figure 8.25 Distributions with negative skew (left), no skew (middle), and positive skew (right)^[27]

The final measure we will review in distributions is kurtosis of a data set. Put simply, kurtosis is a measure of how thin or thick the tails (ends) of a distribution are, as illustrated in Figure 8.26.

Figure 8.26 Different shapes of distributions including leptokurtic (A), mesokurtic (B), platykurtic (C) and uniform^[28]

The first distribution (A, in black) has fat tails with more extreme scores, so the kurtosis is positive, and we say that the data is leptokurtic. Often, these distributions also appear to have a tall peak, so you might notice they look like they are leaping up. The second distribution (B, in red) is mesokurtic, with a kurtosis value almost exactly 0 and tails are neither thin nor fat. Think of a mesa in the desert or mesa in Spanish, which means “table”. The third distribution (C, in green) is platykurtic, meaning that the distribution is fairly evenly spread out and has thin tails (very few extreme scores), so the kurtosis value is negative. Think of a plate or a platypus tail – very flat. Finally, the fourth distribution (D, in blue) is uniform. This is a special kind of platykurtic distribution that is really evenly spread out, with no tails. This is summarized in Table 8.3.

Table 8.3 Thin to Fat Tails to Illustrate Kurtosis^[29]

Please watch: Randomness: Crash Course Statistics #17 (which covers skew and kurtosis)

Summary

Hopefully, this section has helped you see the importance of tables and graphs for visualizing what is happening in a sample of data. It is important to be able to ‘read’ tables and graphs to test claims that others are making about what is happening or what is good for us, and it is good for scientists to use table and graph representations to communicate their findings with their audience swiftly and effectively.

Additional Video Helps & Resources

For tips and demonstrations on setting up and running visual data analyses in SPSS, please see Virginia Wickline: SPSS Statistics Helps (publishing dates vary).

Media Attributions

^[1] (Posted by Terrence Ong via Wikimedia Commons licensed under CC BY -SA 3.0.)

^[2] (Public domain in the U.S., without copyright notice. Posted by Fma 12 via Wikimedia Commons.)

^[3] (CC BY-SA 4.0 by Cote et al., 2021)

^[4] (“Mac Pie Chart” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[5] (“Mac Bar Chart” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[6] (“Card Game Bar Chart” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[7] (“Mac Bar Chart 3D” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[8] (“Mac Bar Chart Lie Factor” by Judy Schmitt is licensed under CC BY-NC-SA 4.0. “Apple iMac G3 (1998)” by albaco/Flickr is licensed under CC BY-NC-SA 2.0; image was brightened and background was removed.)

^[9](“Mac Bar Chart Baseline 50” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[10] Figure 8.9. (“Line Chart Inappropriately Used” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[11] (“Touchdown Passes Raw Data” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[12] (“Touchdown Passes Stem and Leaf” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[13] (“Women’s Times Raw Data” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[14] (“Box Plot First Step” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[15] (“Box Plot Whiskers” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[16] (“Box Plot Outside Value” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[17] (“Percent Increase in Stock Indexes” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[18] (“Percent Change in CPI” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[19] (“Means of Two Conditions” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[20] (“Percent Change in CPI Line Graph” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[21] (“Percent Change in CPI x5 Line Graph” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[22] (CC BY-SA 4.0 by Cote et al., 2021)

^[23] (“Psychology Test Scores Histogram” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[24] (“Symmetrical Distribution” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[25] (“Bimodal Distribution” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[26] (“Skewed Distributions” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[27] (CC BY-SA 4.0 by Navarro & Foxcroft, 2025)

^[28] (CC BY 3.0 by Hunsvotti via Wikimedia Commons)

^[29] (CC BY-SA 4.0 by Navarro & Foxcroft, 2025)

Text Attributions

Cote, L. R., Gordon, R. G. Randell, C. E., Schmitt, J., & Marvin, H. (2021). Introduction to statistics in the psychological sciences. Pressbooks. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Navarro, D. J., & Foxcroft, D. R. (2025). Learning statistics with jamovi: A tutorial for beginners in statistical analysis. Open Book Publishers. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

References

CrashCourse (2018, May 23). Randomness: Crash course statistics #17. https://www.youtube.com/watch?v=jL9en6NvQfk

CrashCourse (2018, February 21). Charts are like pasta - data visualization part 1: Crash course statistics #5. https://www.youtube.com/watch?v=hEWY6kkBdpo

CrashCourse (2018, February 28). Plots, outliers, and Justin Timberlake: Data visualization part 2: Crash course statistics #6. https://www.youtube.com/watch?v=HMkllhBI91Y

Gaslowitz, L. (2017, July 6). How to spot a misleading graph – Lea Gaslowitz. TED-Ed. https://www.youtube.com/watch?v=E91bGT9BjYk

Tufte, E. R. (1983). The visual display of quantitative information. Graphics Press.

Chapter 9: Central Tendency and Variability

Measures of Central Tendency

In most situations, the first thing that a researcher will want to calculate for a sample of data is a measure of central tendency. That is, they would like to know something about what the most typical response is. The three most commonly used measures of central tendency are the mean, median, and mode.

The mean of a set of observations is another way of saying the average. Add all of the values up, and then divide by the total number of values. If the number of books that a group of five kindergartners read last month were 56, 31, 56, 8, and 32, the mean of these observations would be: (56+31+56+8+32) / 5 = 183 / 5 = 36.60. Of course, this definition of the mean probably is not new to you. Averages (i.e., means) are used so often in everyday life that this is pretty familiar stuff.

Figure 9.1 Row of mathematics books^[1]

The second measure of central tendency that is often used is the median. The median of a set of observations is just the middle value, like the median on the highway separates the two lanes of traffic. As before, let us imagine how many books a group of kindergarteners read: 56, 31, 56, 8 and 32. To figure out the median, sort these numbers into ascending order: 8, 31, 32, 56, 56. Once the numbers are ordered, it is easy to see that the median value of these five observations is 32 since that is the middle score in the sorted list (we have put it in bold to make it even more obvious). Easy enough, right? However, what should we do if we are interested in six kindergarteners rather than five? If a sixth kindergartener read 14 books, our sorted list is now 8, 14, 31, 32, 56, 56. There are two middle numbers, 31 and 32. The median is defined as the average of those two numbers, which is 31.5. It would be very tedious to do this by hand if you had lots of numbers.

Figure 9.2 Highway with a median that divides the road in half^[2]

The mode of a sample is very simple. It is the value that occurs most frequently. Things that are fashionable are “in the mode” and pie with ice cream (also very popular) is called pie á la mode. One last point to make regarding the mode. While the mode is most often calculated when data are nominal, because means and medians are useless for those sorts of variables, there are some situations in which one really does want to know the mode of an ordinal, interval, or ratio scale variable. Consider this scenario: A friend of yours is offering a bet and they pick a football game at random. Without knowing who is playing, you have to guess the exact winning margin. If you guess correctly, you win $50. If you do not, you lose $1. There are no consolation prizes for “almost” getting the right answer. You have to guess exactly the right margin. For this bet, the mean and the median are completely useless to you. It is the mode that you should bet on. Or, what if you were asking how old people are in your statistics classes and there were two most popular answers, 19 and 22? In that case, having a bimodal distribution would be another good reason to focus on the mode(s) for interval or ratio data.

Figure 9.3 Apple pie à la mode - so yummy and popular!^[3]

Figure 9.4 Tokyo Fashion Week 2010 collections. The mode is so popular, everyone will want it^[4]

Please watch: Math Antics – Mean, Median and Mode.

Measures of Variability

The statistics that we have discussed so far all relate to central tendency. That is, they all talk about which value(s) are most typical or common in the data. However, central tendency is not the only type of summary statistic that we want to calculate. The second thing that we really want is a measure of the variability of the data, that is, how spread out the data are. Metaphorically, do we have a small puddle or a large puddle coming out from the center? In other words, whatever we have chosen for our measure of central tendency (mean, median, or mode), we also want to know how far away from this the observed values tend to be.

The range for a given variable is very simple. It is the biggest value minus the smallest value. Although the range is the simplest way to quantify the notion of variability, it is also one of the worst because it is the least precise. If the data set has some extreme (possibly untrustworthy) values in it, we would like our summary statistics to not be unduly influenced by these cases. For example, let us say we asked nine people how many cookies they ate yesterday and we got these responses: 0, 1, 2, 3, 4, 5, 6, 10, 100. In a variable containing an extreme outlier, it is clear that the range is not very robust. This variable has a range of 100, but if the outlier were removed, it would have a range of only 10. One has to use the range if they have selected the mode for central tendency, but for continuous data, we probably want a more representative answer.

The interquartile range (IQR) is like the range, but instead of the difference between the biggest and smallest value, the difference between the 25th percentile and the 75th percentile is calculated. If you do not already know what a percentile is, the 10th percentile of a data set is the given number (let’s say, x) such that 10% of the data is less than x. In fact, we have already come across this idea indirectly. The median of a data set is the value that is at the 50th percentile!

While it is clear how to interpret the range, how to interpret the IQR is a little less obvious. The simplest way to think about it is like this: The interquartile range is the range spanned by the middle half or middle 50% of the scores. That is, one quarter of the data falls below the 25th percentile and one quarter of the data is above the 75th percentile, leaving the middle half of the data lying in between the two, so the IQR is the range covered by that middle half. If one has opted for the median as the best measure of central tendency, one would select the IQR as the best measure of variability.

Another kind of variability that we would use with the mean is called the variance, which is the average squared deviation (distance) of each score from the mean. The variance of a data set 𝑋 is sometimes written as Var(𝑋), but it is more commonly denoted 𝑠2 (the reason for this will become clearer shortly).

How about a concrete example? Say our friend, Jamisa, was a basketball player, and we looked at how many points she scored in her last five games. If we calculate the mean, she is averaging 36.6 points per game (great job, Jamisa!). We would end up with the information shown in Table 9.1.

Table 9.1 Measures of Variability for Jamisa’s Last Five Basketball Games^[5]

That last column contains all of our squared deviations, so all we have to do is average them, dividing by the sample size (N). If we do that by hand, using a calculator, we end up with a variance of 324.64.

If we were using this to estimate a population value, rather than just describing the sample, we would divide by (N-1). There is a subtle distinction between describing a sample and making guesses about the population from which the sample came. Regardless of whether one is describing a sample or drawing inferences about the population, the mean is calculated exactly the same way. Not so for the variance, or the standard deviation, or for many other measures. What was outlined to you initially (i.e., take the actual average, and thus divide by 𝑁) assumes that you literally intend to calculate the variance of the sample. Most of the time, however, researchers are not terribly interested in the sample in and of itself. Rather, the sample exists to tell us something about the world. If so, you are actually starting to move away from calculating a sample statistic (gathered from the data you collected) and towards the idea of estimating a population parameter (an estimate for the population at large).

This section so far may have read a bit like a mystery novel. We have calculated the variance and described the “𝑁 −1” thing to estimate a population, but we need to review the single most important thing: how to interpret the variance. Descriptive statistics are supposed to describe things, after all, and right now the variance might seem like a gibberish number. Unfortunately, the reason why we have not given you the human-friendly interpretation of the variance is that there really is not one. This is the most serious problem with the variance. Although it has some elegant mathematical properties that suggest that it really is a fundamental quantity for expressing variation, it is completely useless if you want to communicate with an actual human. Variances are completely uninterpretable in terms of the original variable! All the numbers have been squared, so they do not mean anything practical anymore. This is a huge issue. For instance, according to Table 9.1, the margin in game 1 was “376.36 points-squared higher than the average margin.” This is exactly as goofy as it sounds, and so when we calculate a variance of 324.64, we are in the same situation. If you have watched a good bit of basketball, you will notice at no time has anyone ever referred to “points squared.” It is not a real unit of measurement.

Okay, suppose that you like the idea of using the variance, but since you are a human and not a robot or math fanatic, you would like to have a measure that is expressed in the same units as the data itself (i.e., points scored, not points squared). What should you do? The solution to the problem is to take the square root of the variance, known as the standard deviation, also called the root mean squared deviation, or RMSD. This solves our problem fairly neatly. While nobody has a clue what “a variance of 324.64 points-squared” really means, it is much easier to understand “a standard deviation of 18.01 points” since it is expressed in the original units. It is traditional to refer to the standard deviation of a sample of data as s, though “sd” and “std dev.” are also used at times, and you might see the Greek letter sigma (s) if we are talking populations. If you are using a statistical program to analyze your data, it almost always calculates a version that divides by 𝑁 − 1 rather than 𝑁.

Interpreting standard deviation is slightly more complex. Because the standard deviation is derived from the variance, and the variance is a quantity that has little to no meaning that makes practical sense, the standard deviation does not have a simple interpretation. As a consequence, most of us just rely on a simple rule of thumb. In general, if the data are normally distributed, one should expect 68% of the data to fall within 1 standard deviation of the mean, 95% of the data to fall within 2 standard deviations of the mean, and 99.7% of the data to fall within 3 standard deviations of the mean. This rule tends to work pretty well most of the time, but it is not exact. It is actually calculated based on an assumption that the histogram is symmetric and “bell shaped”.

Figure 9.5 A symmetrical bell curve showing that 68.2% of scores fall within one standard deviation (SD), and 95.4% within 2 SDs, and 99.6% within 3 SDs^[6]

Here is a quick summary about variability:

• Range. Describes the full spread of the data. It is very vulnerable to outliers, and as a consequence, it is not often used unless one is complementing the mode for nominal data or there are good reasons to care about the extremes in the data.
• Interquartile range. Indicates where the middle half of the data sits. It is pretty robust and complements the median nicely. This is used a lot.
• Variance. The average squared deviation from the mean. It is mathematically elegant and is probably the “right” way to describe variation around the mean, but it is challenging because it does not use the same units as the data. Almost never used except as a mathematical tool, when it is buried within other statistical calculations.
• Standard deviation. This is the square root of the variance. It is fairly elegant mathematically, and it is expressed in the same units as the data so it can be understood and interpreted pretty well. In situations where the mean is the measure of central tendency, this is the default. This is by far the most popular measure of variation. In short, the IQR and the standard deviation are easily the two most common measures used to report the variability of the data.

Please watch: Measures of Variability (Range, Standard Deviation, Variance).

Mean, Median, or Mode – Which to Choose?

Knowing how to calculate mean, median, and mode is only a part of the story. One also needs to understand what each one is saying about the data, and what that implies for when each one should be used. This is illustrated in Figure 9.6.

Figure 9.6 An illustration of the difference of how the mean and the median should be interpreted ^[7]

The mean is basically the ‘center of gravity’ of the data set. If you imagine that the histogram of the data is a solid object, then the point on which you could balance it (as if on a see-saw or old-fashioned scale) is the mean. In contrast, the median is the middle observation, with half of the scores smaller than it and half of the observations larger than it.

The mean is the balance point of the data set, whereas the median is the dividing point in the data, and the mode is the most popular score or scores. What this implies, as far as which one should be used, depends a little on what type of data one has and what they are trying to achieve. Consider your best measure of central tendency and variability as dancing partners: The mode always goes with the range, the median always goes with the IQR, and the mean always goes with the standard deviation. Do not try to make them dance with other partners!

Figure 9.7 Like dancing partners, the mode always goes with the range, the median always goes with the IQR, and the mean always goes with the standard deviation^[8]

As a rough guide:

• If the data are on a nominal scale, statisticians say the mean and the median cannot or should not be used. Both the mean and the median rely on the idea that the numbers assigned to values are meaningful and have order. If the numbering scheme is arbitrary like it is for nominal (categorical) data, then the mode must be used. As mentioned previously, the mode would also be the best choice if the data are bimodal.
• If the data are on an ordinal scale, use the median instead of the mean. The median only makes use of the order information in the data (i.e., which numbers are bigger) but does not depend on the precise numbers involved. That is exactly the situation that applies when data are on an ordinal scale. The mean, on the other hand, makes use of the precise numeric values assigned to the observations, so it is not really appropriate for ordinal data.
• For data on an interval and ratio scale, either the median or mean is generally acceptable. Which one you pick depends a bit on what you are trying to achieve. The mean has the advantage that it uses all the information in the data (which is useful when not much data exists). However, the mean is very sensitive to extreme, outlying values called outliers. Extreme scores are two or more standard deviations from the mean, and outliers are three or more standard deviations. Think of outliers as outlaws in an old Western movie: They kind of do their own thing, and they often cause chaos in a group of data.

Figure 9.8 A mural of an outlaw on his horse. Watch out for outliers in your data set – they can wreak havoc^[9]

Expanding on that last part a little, one consequence is that there are systematic differences between the mean and the median or mode when the histogram is asymmetric (has abnormal skew and kurtosis, as discussed in a previous chapter). This is illustrated in Figure 9.6 above. Notice that the median (right-hand side) is located closer to the body of the histogram, whereas the mean (left-hand side) gets dragged towards the tail (where the extreme values are).

To give a concrete example, suppose Bob (income $50,000), Li (income $60,000) and Keisha (income $65,000) are sitting at a table. The average income at the table is $58,333 and the median income is $60,000. Then José sits down with them (income $100,000,000). The average income has now jumped to $25,043,750, but the median rises only to $62,500. If you are interested in looking at the overall income for the people at the table, the mean might be the right answer. However, if you are really interested in the typical income at the table, the median would be a much better, more realistic, and (some would say) more truthful choice.

Another demonstration is illustrated in Figure 9.9. If a data set is normally distributed, the mean, median, and mode will all be at (or relatively at) the same place in the distribution, so it does not matter which one you pick (although we would go with the mean if we can). When a data set is positively skewed by a large outlier or outliers to the right (like the high-income example above), the mean is pulled toward those extreme scores, making the mean higher than the median and mode, making the median the most reliable indicator of what is typical for the dataset. If the data set is negatively skewed because there is an outlier or outliers in the low end of the distribution (a very low score), the mean will be lower in value than the median and mode, again, making the median the most reliable indicator of what is typical for the group.

Figure 9.9 How the mean, median, and mode line up on skewed and normal distributions^[10]

Please watch: Why Averages Lie To You.

A Metaphor and a Real-Life Example

When the chapter author’s daughter was in elementary school, the highlight for the end-of-year celebration was field day, with the very last event being a tug-of-war competition for each grade. Being in a very small school, they kept children of like abilities together, so my daughter always had the same classmates. In her class was a child they called “Big Mike” (not his real name). Big Mike was four feet tall at age 8 – no joke – over a foot taller than anyone else in his class and matching height with his teachers by fourth grade. Guess who won the tug-of-war every year? My daughter’s class! Well, let’s be honest…Big Mike. In this case, Big Mike was like the outlier, pulling the flag from the middle to his team’s side every time. This is just what an outlier does to the mean – it pulls it off center in the distribution. Following the metaphor, Big Mike masked the strength of typical children because he overpowered them. An outlier does the same thing to the mean.

Figure 9.10 Junior high students in China play tug-of-war. Like an outlier, someone super strong will pull the flag (the mean) from the middle to their side to win the game^[11]

To try to get an even better sense of why you need to pay attention to the differences between the mean and the median, consider also this real-life example. This is an article by Michael Janda from the ABC news website from 24 September, 2010:

Senior Commonwealth Bank executives have travelled the world in the past couple of weeks with a presentation showing how Australian house prices, and the key price to income ratios, compare favourably with similar countries. “Housing affordability has actually been going sideways for the last five to six years,” said Craig James, the chief economist of the bank’s trading arm, CommSec.

This probably comes as a huge surprise to anyone with a mortgage, or who wants a mortgage, or pays rent, or is not completely oblivious to what has been going on in the housing market over recent decades. Back to the article:

CBA has waged its war against what it believes are housing doomsayers with graphs, numbers and international comparisons. In its presentation, the bank rejects arguments that Australia’s housing is relatively expensive compared to incomes. It says Australia’s house price to household income ratio of 5.6 in the major cities, and 4.3 nationwide, is comparable to many other developed nations. It says San Francisco and New York have ratios of 7, Auckland’s is 6.7, and Vancouver comes in at 9.3.

More excellent news! Except, the article goes on to make the observation that:

Many analysts say that has led the bank to use misleading figures and comparisons. If you go to page four of CBA’s presentation and read the source information at the bottom of the graph and table, you would notice there is an additional source on the international comparison– Demographia. However, if the Commonwealth Bank had also used Demographia’s analysis of Australia’s house price to income ratio, it would have come up with a figure closer to 9 rather than 5.6 or 4.3.

That is a rather serious discrepancy. One group of people say 9, another says 4-5. Should we just split the difference and say the truth lies somewhere in between? Absolutely not! This is a situation where there is a right answer and a wrong answer. Demographia is correct, and the Commonwealth Bank is wrong. As the article points out:

[An] obvious problem with the Commonwealth Bank’s domestic price to income figures is they compare average incomes with median house prices (unlike the Demographia figures that compare median incomes to median prices). The median is the mid-point, effectively cutting out the highs and lows, and that means the average is generally higher when it comes to incomes and asset prices, because it includes the earnings of Australia’s wealthiest people. To put it another way: the Commonwealth Bank’s figures count Ralph Norris’ multi-million dollar pay packet on the income side, but not his (no doubt) very expensive house in the property price figures, thus understating the house price to income ratio for middle-income Australians.

Notice: The way that Demographia calculated the ratio is correct. The way that the Bank did it is incorrect. As for why an extremely quantitatively sophisticated organization such as a major bank made such an elementary mistake, well…we cannot say for sure. However, the article itself does happen to mention the following facts, which may or may not be relevant:

[As] Australia’s largest home lender, the Commonwealth Bank has one of the biggest vested interests in house prices rising. It effectively owns a massive swathe of Australian housing as security for its home loans as well as many small business loans.

Sometimes, it is good to be skeptical of the source of information, as organizations can have agendas, just like individuals.

Summary

This section has helped illustrate why both what is typical (central tendency) and how spread out the data are (variability) are important in understanding what is happening in a data sample. Each measure of central tendency has its variability partner when reporting data. One must be careful in considering which measure of central tendency and variability are best to report, considering scale of measurement and shape of the distribution when making their choice; the average is not always best just because it is the average. In fact, sometimes the average can be misleading.

Additional Video Helps & Resources

For tips and demonstrations on setting up and running descriptive statistical analyses in SPSS, please see Virginia Wickline: SPSS Statistics Helps (publishing dates vary).

Media Attributions

^[1] (CC BY-SA 4.0 by AKibombo via Wikimedia Commons)

^[2] (CC0 1.0 Universal by Vknyshov via Wikimedia Commons)

^[3] (CC BY 3.0 by Dwight Burdette via Wikimedia Commons)

^[4] (CC BY-SA 2.0 by Tokyographer via Wikimedia Commons)

^[5] (CC BY-SA4.0 by Navarro & Foxcroft, modified by current authors.)

^[6] (CC BY 2.5 by Jeremy Kemp via Wikimedia Commons)

^[7] (CC BY-SA 4.0 by Navarro & Foxcroft, 2025)

^[8] (CC BY-SA4.0 by Todd Combs via Wikimedia Commons)

^[9] (CC BY-SA 3.0 by Franco Baresi via Wikimedia Commons)

^[10] (CC BY-SA4.0 by Diva Jain via Wikimedia Commons)

^[11] (CC BY-SA 2.0 by 褒忠國中 雲端網 (WC-QHS) via Wikimedia Commons)

Text Attributions

Navarro, D. J., & Foxcroft, D. R. (2025). Learning statistics with jamovi: A tutorial for beginners in statistical analysis. Open Book Publishers. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

References (including Video Attributions)

Foundations of Math (2026, January 14). Why averages lie to you. https://www.youtube.com/watch?v=8LXynB2mPls

MathAntics (2017, March 3). Math antics – Mean, median and mode. https://www.youtube.com/watch?v=B1HEzNTGeZ4

Storage, D. (2019, June 18). Measures of variability (range, standard deviation, variance). https://www.youtube.com/watch?v=s7WTQ0H0Acc

Chapter 10: Cronbach’s alpha (Reliability Analysis)

Cronbach’s alpha (a): A Measure of Internal Consistency

If you are using a survey measure in your study, you will want to see how well the factors are measured using a scale that combines the observed variables. In statistics, we use reliability analysis to provide information about how consistently a scale measures a psychological construct (see Module 2), which is a variable we are interested in studying. Many kinds of reliability exist to make sure a measure is consistent. Internal consistency is what we are concerned with here, which refers to the consistency across all the individual items that make up a measurement scale. In other words, we want to make sure that all the variables play nice together, that they all correlate with one another. We can calculate a statistic that tells us how internally consistent these items are in measuring the underlying construct.

One popular statistic used to check the internal consistency of a scale is Cronbach’s alpha (Cronbach, 1951). Cronbach’s alpha is a measure of equivalence (whether different sets of scale items would give the same measurement outcomes). Equivalence is tested by dividing the scale items into two groups (a “split-half”) and seeing whether analysis of the two parts gives comparable results. Of course, there are many ways a set of items could be split, but if all possible splits are made then it is possible to produce a statistic that reflects the overall pattern of split-half coefficients. Cronbach’s alpha (a) is such a statistic: a function of all the split-half coefficients for a scale. If a set of items that measure a construct (e.g., an Extraversion scale) has an a = .80, then the proportion of error variance in the scale is 0.20. In other words, a scale with an a = .80 includes approximately 20% error.

Cronbach’s a: Let’s Cook!

Why do we need a measure to be internally consistent? Let’s use the metaphor of cooking. People cook so they can eat. Some people really like to cook. Almost everyone really likes to eat. Many cultures around the world enjoy eating some form of chicken soup. Take a second and imagine a big steaming cup of chicken noodle soup like your Mama or Daddy (or whomever cooked for you) used to make.

Figure 10.1 Cup of chicken noodle soup with carrots and green beans. Yum!^[1]

Although chicken noodle soup is a worldwide phenomenon, not everyone makes chicken noodle soup the same way. There are a thousand different recipes for a thousand different chefs. Imagine your ingredients. What elements would you need to make good chicken soup?

Chicken? Yes

Noodles? Yes

Broth? Yes

Carrots? Yes

Garlic? Yes

Parsley? Yes

Onion? Yes

Salt? Yes

Pepper? Yes

Turmeric? Yes

Sardines? Yes

Chocolate? Yes

Figure 10.2 Sardines grilled and served with lemon^[2]

Figure 10.3 Dark chocolate and mint chocolate chips^[3]

Whoa…whoa…whoa…hold up. Sardines?!? Chocolate?!? We probably had you nodding and maybe even your stomach growling until these last two ingredients. Maybe you like chocolate, but it does not belong in chicken noodle soup. Maybe you hate sardines. If you put spoiler items in your recipe, it might ruin it, and you could need to throw it out and start over. For some other ingredients, if they are missing, you still have a recipe, but one that is not as good. Yet other ingredients are an absolute must. Do you have soup if you do not add broth? No, you have a casserole or stir fry instead.

That is really the big picture of what Cronbach’s a does: It makes sure we have a good ‘recipe’ for our survey measure. It makes sure all the items correlate with one another. Enough ingredients, not too few or too many. Nothing irrelevant. Nothing that makes you retch or start over. Ready to serve for a dinner party. Ready to use in our research study and trust our outcomes.

Cronbach’s a: How Reliable is ‘Enough’ or ‘Too Much’?

Cronbach’s a is expressed as a number somewhere between 0 and 1. Those are its mathematical limits.

If you ever get a negative number for Cronbach’s a, this indicates that you seriously messed something up. The smoke alarm is going off. The pan is charred. The dog won’t even eat it. Start over. Sometimes the items are just plain wrong and do not belong, like if your measure of self-esteem also included people’s age or how much they like peanut butter. Sometimes if you get a negative number, you have an item that is relevant, but it is backwards and needs to be reverse scored, which is where you turn a scale from, say, 1 to 5 instead of 5 to 1. For example, if you were doing a measure of attitudes toward mask-wearing for COVID, where a score of 1 = Strongly Disagree and 5 = Strongly Agree. If you had an item that said, “Wearing masks in public is really annoying,” it is relevant to your idea, but it is written in the opposite of what you are looking for. Someone who scores highly on this item does not like wearing masks. In this case, instead of throwing the item out, you would reverse it so a high score becomes a low score…then it will play nicely (mathematically) with your other items.

We also want to know when our measure is good enough to use in research. If we were cooking, we would want to know when our recipe is good enough to serve to guests for dinner. Tavakol and Dennick (2011) note that mathematicians differ a little on what stands for ‘good enough,’ with estimates somewhere between .70 and .95. Across authors we have seen, an a of .70 is generally considered ‘sufficient’ or ‘good’ for research purposes. Below that, and it will typically be called ‘questionable’, with the lower the number goes shifting to ‘poor’ or ‘unreliable.’ In other words, an a below .70 calls into question whether you have good data and, therefore, might not really be measuring what you think you are in your research, no matter what you are trying to find, study, or test. Maybe you should start over and get another measure. If you have already run your project, this could lead to researchers not to trust your whole study, and you could have trouble getting your research published or funded (if you were seeking grants). Thus, it is important to make sure when you are planning your study that you have good survey measures that are reliable. It is usually better, therefore, to use measures that have been created and checked by other authors in their studies, rather than making up questions on your own. If you do make up your own questions, the burden of proof is on you – you need to show that you have made a good batch (or recipe) of items that go well together and make sense.

An alpha level can also be too high. Tavakol and Dennick (2011) recommend .90, other authors say .95. If a is too high, then we are being redundant, like saying “add noodles,” “add noodles,” “add noodles” when we are cooking. We already said to add noodles, so we do not need to say it again. If you ask participants the same or nearly identical question over and over on a survey, they will tend to get irritated and less cooperative, because you are wasting their time.

Cronbach’s a: Some Caveats

When Cronbach’s a is high, it is a good starting point, but it is not necessarily a measure of unidimensionality (i.e., an indicator that a scale is measuring a single factor or construct rather than multiple related constructs). Scales that are multidimensional will cause alpha to be under-estimated (low) if not assessed separately for each dimension, but a high value for alpha is not always an indicator of unidimensionality. So, a = .80 does not mean that 80% of a single underlying construct is accounted for. It could be that the 80% comes from more than one underlying construct. In other words, a strong Cronbach’s a score is necessary but not sufficient for determining unidimensionality. If Cronbach’s a is low, the measure cannot be unidimensional. The items do not go well together. If the Cronbach’s a is high, the measure might be unidimensional, but it is not necessarily so.

Revisiting our chicken noodle soup measure for illustration. First, the sardines. You’re cooking. If you are rolling right along, building your chicken noodle soup, and throw in an ingredient you do not like that does not go well with the others (say sardines or perhaps rotten tomatoes), it will ruin the recipe. Cronbach’s a will be low, and you will need to start all over, taking the stinker ingredient (your bad survey item) out.

If we have all of our ingredients, which are things that we like, and we also add another thing that we like (chocolate), it will not necessarily ruin the statistic. Instead of measuring how well everything tastes together, we might just be measuring “things I like to eat,” irrespective of whether we like to eat them together, in this recipe. So even if the numbers play nice, we also need to read and review the items to make sure they fit together logically. In this case, reliability is also a necessary but not sufficient condition for validity (accuracy). If something is going to be measured right (valid), it must be consistent (reliable)…but just because it is reliable does not make it valid.

Another illustration that reliability is not validity can be seen in the following recipe. You decide to make chicken noodle soup, so you throw together ice cream, hot fudge sauce, peanut butter sauce, sprinkles, whipped cream, and a cherry. Voilá! Chicken noodle soup! By now you are looking pretty confused and saying, “That is not chicken noodle soup…that is an ice cream sundae.” And right you are! But mathematically, you would still have a strong Cronbach’s a, because all of these items go together well. They are reliable but not valid items for a chicken noodle soup recipe.

Cronbach’s a can also be affected by the length of the survey (Tavakol & Dennick 2011). If there are too few items, Cronbach’s a will not be as strong. (Consider if you just made soup with chicken, noodles, and broth. It would be mid, or just OK.). You will need to add more ingredients (more survey items).

Further, another feature of a is that it tends to be sample specific: It is not a characteristic of the scale, but rather a characteristic of the sample in which the scale has been used. A biased, unrepresentative, or small sample could produce a very different a coefficient than a large, representative sample. The a can even vary from large sample to large sample. The picture of chicken noodle soup above shows mushrooms and green beans. Maybe you did not grow up eating chicken noodle soup with mushrooms and beans but instead had celery and onions. In Mexico, chicken noodle soup will almost certainly have cilantro instead of parsley. In China, chicken noodle soup typically has rice noodles or wonton noodles instead of egg noodles, while in Japan it will have soba or udon noodles. How you make the recipe depends on who and where you ask the questions. Your oma or grandma’s recipe for chicken noodle soup might not go over as well in China, Japan, or Mexico, where the cultural expectations for the recipe are a little different. Similarly, the questions we ask on our survey might not go over as well with a new group of people of a different age in a different place. That is why researchers would check the Cronbach’s a for their study even if they are using a measure that has worked well in other studies. Even if we believe it is a good recipe, we need to sip and taste it before serving it up.

Please watch: Cronbach’s Alpha (Simply Explained).

Summary

Despite some of its limitations, Cronbach’s a has been popular in psychology for estimating internal consistency reliability—for making sure that the items go together well. It is fairly easy to calculate, understand and interpret, so it can be a useful initial check on scale performance when you administer a survey with a different sample from a different setting, population, or age group, for example, than the original authors who created the measure.

Additional Resources

For tips and demonstrations on setting up and running statistical analyses in SPSS, please see Virginia Wickline: SPSS Statistics Helps (publishing dates vary).

Media Attributions

^[1] (CC BY-SA 2.0 by Casja Lillihook via Wikimedia Commons)

^[2] (CC BY-SA 2.0 by Boxley via Wikimedia Commons)

^[3] (CC BY-SA 2.0 by TaurusEmerald via Wikimedia Commons)

Text Attributions

Navarro, D. J., & Foxcroft, D. R. (2025). Learning statistics with jamovi: A tutorial for beginners in statistical analysis. Open Book Publishers. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified extensively by the current authors.

References

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555

Numiqo (2021, October 27). Cronbach’s alpha (simply explained). https://www.youtube.com/watch?v=W9uPvAmtTOk

Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s a. International Journal of Medical Education, 2, 53-55. https://doi.org/10.5116/ijme.4dfb.8dfd

Module 4: Analyzing Data – Calculating Inferential Statistics

Chapter 11: The Basics of Hypothesis Testing

After reviewing descriptive statistics, it is now time to turn our attention to a really big idea in statistics, which is hypothesis testing. In its most abstract form, hypothesis testing is really a very simple idea. The researcher has some idea about the world and wants to determine whether or not the data actually support that idea. However, the details are messy, and most people find the theory of hypothesis testing to be the most frustrating part of statistics. First, we will explore how hypothesis testing works in a fair amount of detail, using a simple running example to show you how a hypothesis test is “built”. Then, we will spend a bit of time talking about the various dogmas, rules, and heresies that surround the theory of hypothesis testing.

Logic and Purpose of Hypothesis Testing

A hypothesis is a prediction that is tested in a research study. Here we will present an example based on a James Bond movie where the superspy insisted that martinis should be shaken rather than stirred. Consider a hypothetical experiment to determine whether Mr. Bond can tell the difference between a shaken martini and a stirred martini. Suppose we gave Mr. Bond a series of 16 taste tests. In each test, we flipped a fair coin to determine whether to stir or shake the martini. Then, we presented the martini to Mr. Bond and asked him to decide whether it was shaken or stirred. If Mr. Bond was correct on 13 of the 16 taste tests, does this prove that Mr. Bond has at least some ability to tell whether the martini was shaken or stirred?

Figure 11.1 That is one classy-looking martini! We hope you approve, Mr. Bond^[1]

This result does not prove that he does; it could be he was just lucky and guessed right 13 out of 16 times. But, how plausible is the explanation that he was just lucky? To assess its plausibility, we determine the probability that someone who was just guessing would be correct 13/16 times or more. Using a probability table for assistance, this probability can be computed to be .011. This is a pretty low probability, and therefore someone would have to be very lucky to be correct 13 or more times out of 16 if they were just guessing. So, either Mr. Bond was very lucky and it was just a fluke, or he really can tell whether the drink was shaken or stirred. The hypothesis that he was guessing is not proven false, but considerable doubt is cast upon it. Therefore, there is strong evidence that Mr. Bond can tell whether a drink was shaken or stirred.

Our example aligns with what science philosopher Karl Popper (1959) called falsifiability: The goal of science is to rule out the bad or unreasonable answers, so that whatever we are left with could be plausible, which means apparently true or reasonable. Those who were fans of the TV show Mythbusters (see Figure 11.2) might remember that they had three outcomes for any of the fantabulous myths they tested: confirmed, busted, or plausible.

Figure 11.2 Mythbusters Grant Imahara, Jamie Hyneman, and Adam Savage at the 2009 San Diego ComicCon Convention^[2]

If your proposition is, “Everyone loves chocolate,” it only takes one chocolate-hater or one person with a chocolate allergy to prove you wrong. However, if everyone you happen to ask loves chocolate, that also does not prove without a doubt that everyone in the whole world loves chocolate. It just makes it plausible. You would need to encounter all the world’s inhabitants from all times and all places to prove your point without any doubt, which would be impossible.

The Probability Value

It is very important to understand precisely what probability values mean. In the James Bond example, the computed probability of .0106 is the likelihood that he would be correct on 13 or more taste tests (out of 16) if he were just guessing. It is easy to mistake this probability of .0106 as the probability he cannot tell the difference. This is not at all what it means.

The probability of .0106 is the probability of a certain outcome (13 or more out of 16) assuming a certain state of the world (James Bond was only guessing). It is not the probability that a state of the world is true. Although this might seem like a distinction without a difference, consider the following example. An animal trainer claims that a trained bird can determine whether or not numbers are evenly divisible by 7. In an experiment assessing this claim, the bird is given a series of 16 test trials. On each trial, a number is displayed on a screen and the bird pecks at one of two keys to indicate its choice. The numbers are chosen in such a way that the probability of any number being evenly divisible by 7 is .50. The bird is correct on 9/16 choices. We can compute that the probability of being correct nine or more times out of 16 if one is only guessing is .40. Since a bird who is only guessing would do this well 40% of the time, these data do not provide convincing evidence that the bird can tell the difference between the two types of numbers. As a scientist, you would be very skeptical that the bird had this ability. Would you conclude that there is a .40 probability that the bird can tell the difference? Certainly not! You would think the probability is much lower than .0001.

To reiterate, the probability value is the probability of an outcome (9/16 or better) and not the probability of a particular state of the world (the bird was only guessing). In statistics, it is conventional to refer to possible states of the world as hypotheses since they are hypothesized states of the world. Using this terminology, the probability value (p-value) is the likelihood of an outcome given the hypothesis. It is not the probability of the hypothesis given the outcome.

This is not to say that we ignore the probability of the hypothesis. If the probability of the outcome given the hypothesis is sufficiently low, we have evidence that the hypothesis is false. However, we do not compute the probability that the hypothesis is false. In the James Bond example, the hypothesis is that he cannot tell the difference between shaken and stirred martinis. The probability value is low (.0106), thus providing evidence that he can tell the difference. However, we have not computed the probability that he can tell the difference.

Figure 11.3 While p-values are really just labeled as significant or not depending on whether the p-value is smaller than the alpha level or not, this cartoon adds (hopefully) a little humor to your learning experience^[3]

A Menagerie of Hypotheses

Suppose we are interested in studying the phenomenon of extrasensory perception (ESP). Often displayed in sci-fi movies and TV shows, ESP is an ability to perceive information without any of the typical senses. Our first study is a simple one in which we seek to test whether clairvoyance exists: an awareness of objects that are hidden from view. Each participant sits down at a table and is shown a card by an experimenter. The card is black on one side and white on the other. The experimenter takes the card away and places it on a table in an adjacent room. The card is placed black side up or white side up completely at random, with the randomization occurring only after the experimenter has left the room with the participant. A second experimenter comes in and asks the participant which side of the card is now facing upwards. It is purely a one-shot experiment. Each person sees only one card and gives only one answer, and at no stage is the participant actually in contact with someone who knows the right answer. The data set, therefore, is very simple. I have asked the question of N people and some number (X) of these people have given the correct response. To make things concrete, suppose that we have tested N = 100 people and X = 62 of these got the answer right. A surprisingly large number, sure, but is it large enough for me to feel safe in claiming we have found evidence for ESP? This is the situation where hypothesis testing comes in useful. However, before we talk about how to test hypotheses, we need to be clear about what we mean by hypotheses.

Research Hypotheses Versus Statistical Hypotheses

The first distinction to keep clear is between research hypotheses and statistical hypotheses. In our ESP study, the overall scientific goal is to demonstrate that clairvoyance exists. In this situation there is a clear research goal: We are hoping to discover evidence for ESP. In other situations we might actually be a lot more neutral than that, so we might say that the research goal is to determine whether or not clairvoyance exists. Regardless of how we want to portray it, the basic point here is that a research hypothesis involves making a substantive, testable, scientific claim. If you are a psychologist, then your research hypotheses are fundamentally about psychological constructs. Any of the following would count as research hypotheses:

• Listening to music reduces your ability to pay attention to other things. This is a claim about the causal relation between two psychologically meaningful concepts (listening to music and paying attention to things), so it is a perfectly reasonable research hypothesis.
• Intelligence is related to personality. Like the last one, this is a relational claim about two psychological constructs (intelligence and personality), but the claim is weaker: correlational not causal.
• Intelligence is the speed of information processing. This hypothesis has a quite different character. It is not actually a relational claim at all. It is an ontological claim about the fundamental character of intelligence. It is usually easier to think about how to construct experiments to test research hypotheses of the form “does X affect Y?” than it is to address claims like “what is X?” And in practice what usually happens is that scientists find ways of testing relational claims that follow from the ontological ones. For instance, if I believe that intelligence is the speed of information processing in the brain, my experiments will often involve looking for relations between measures of intelligence and measures of speed. As a consequence, most everyday research questions do tend to be relational in nature, but they are almost always motivated by deeper ontological questions about the state of nature.

Notice that, in practice, research hypotheses could overlap a lot. The ultimate goal in the ESP experiment might be to test an ontological claim like “ESP exists”, but we might operationally restrict myself to a narrower hypothesis like “Some people can ‘see’ objects in a clairvoyant fashion”. That said, there are some things that really do not count as proper research hypotheses in any meaningful sense:

• Love is a battlefield. This is too vague to be testable. While it is okay for a research hypothesis to have a degree of vagueness to it, it has to be possible to operationalize the theoretical ideas (that is, to figure out how the ideas are going to be measured). It might be challenging to see how this interesting idea can be converted into any concrete research design. If so, then this is not a scientific research hypothesis – it is just a catchy pop song. A lot of deep questions that humans have fall into this category. Maybe one day science will be able to construct testable theories of love, or to test to see if God exists, and so on. Until then, there are some good questions that science just cannot address.
• The first rule of tautology club is the first rule of tautology club. This is not a substantive claim of any kind. It is true by definition. No conceivable state of nature could possibly be inconsistent with this claim. We say that this is an unfalsifiable hypothesis, and as such it is outside the domain of science. Whatever else one does in science, our claims must have the possibility of being wrong.
• More people in my experiment will say “yes” than “no”. This one fails as a research hypothesis because it is a claim about the data set, not about the psychology (unless of course your actual research question is whether people have some kind of “yes” bias!). Actually, this hypothesis is starting to sound more like a statistical hypothesis than a research hypothesis.

As you can see, research hypotheses can be somewhat messy at times, and ultimately, they are scientific claims. Statistical hypotheses are neither of these two things. Statistical hypotheses must be mathematically precise, and they must correspond to specific claims about the characteristics of the data generating mechanism (i.e., the “population”). Even so, the intent is that statistical hypotheses bear a clear relation to the substantive research hypotheses that we care about! For instance, in our ESP study, our research hypothesis is that some people are able to see through walls or whatever. What I want to do is to “map” this onto a statement about how the data were generated. So, what would that statement be? The quantity that we are interested in within the experiment is P(correct), the true-but-unknown probability (or likelihood) with which the participants in my experiment answer the question correctly. Let’s use the Greek letter q (theta) to refer to this probability. Here are four different statistical hypotheses:

• If ESP does not exist and if my experiment is well designed, then our participants are just guessing. So, we should expect them to get it right half of the time. Our statistical hypothesis is that the true probability of choosing correctly is q = .50.
• Alternatively, suppose ESP does exist and participants can see the card. If that is true, people will perform better than chance. The statistical hypothesis is that q > .50.
• A third possibility is that ESP does exist, but the colors are all reversed, and people do not realize it (okay, that is a little wacky, but you never know). If that is how it works, then we would expect people’s performance to be below chance. This would correspond to a statistical hypothesis that q < .50.
• Finally, suppose ESP exists but I have no idea whether people are seeing the right colour or the wrong one. In that case the only claim I could make about the data would be that the probability of making the correct answer is not equal to 0.5. This corresponds to the statistical hypothesis that q =/= .50.

All of these are legitimate examples of a statistical hypothesis because they are statements about a population parameter and are meaningfully related to the experiment.

What this discussion hopefully reveals is that when attempting to construct a statistical hypothesis test, the researcher actually has two quite distinct hypotheses to consider. First, there is a research hypothesis (a claim about psychology), and this then corresponds to a statistical hypothesis (a claim about the data generating population). In our ESP example these might be as shown in Table 11.1

Table 11.1 Research and Statistical Hypotheses^[4]

A key thing to recognize is this: A statistical hypothesis test is directly a test of the statistical hypothesis, NOT the research hypothesis. If a study is badly designed, then the inherent link between the research hypothesis and the statistical hypothesis is broken. To give a silly example, suppose that our ESP study was conducted in a situation where the participant can actually see the card reflected in a window. If that happens, we could find very strong statistical evidence that , but this would tell us nothing about whether “ESP exists.” Thus, a study needs to be designed well, with few flaws or errors (called confounds). To refer to a previous chapter, having a good design with no flaws or problems is otherwise known as internal validity.

Please watch 6 Steps to Formulate a STRONG Hypothesis

Null Hypotheses and Alternative Hypotheses

So far, so good. We have a research hypothesis that corresponds to what we want to believe about the world, and we can map it onto a statistical hypothesis that corresponds to what we want to believe about how the data were generated. It is at this point that things get somewhat counter-intuitive for a lot of people. Because what we do is invent a new statistical hypothesis (the null hypothesis, H₀) that corresponds to the exact opposite of what we want to believe or think will happen. The null hypothesis suggests that an apparent effect is due to chance. then focus exclusively on that almost to the neglect of the thing we actually interested in. Keep in mind that null means nothing, nada, zip, zero. It signifies an empty state, nothingness, or a thing negated. If we made a deal that was null, our deal is no good. If an Elvis chapel marriage is annulled after the parties sober up in Vegas, it is canceled as if it never happened.

If the null hypothesis is that in the population of physicians, the mean time expected to be spent with obese patients is equal to the mean time expected to be spent with average-weight patients where the Greek letter m (or “mu”) is used for the population means. This null hypothesis can be written as:

The null hypothesis in a correlational study of the relationship between high school grades and college grades would typically be that the population correlation is 0. This can be written as:

where r (Greek letter “rho”) is the population correlation value (we will cover correlation in a later chapter).

Although the null hypothesis is usually that the value of a parameter is 0, there are occasions in which the null hypothesis is a value other than 0. For example, if we are working with mothers in the U.S. whose children are at risk of low birth weight, we can use 7.47 pounds, the average birth weight in the U.S., as our null value and test for differences against that. In this case, using birth weight as an example, our null hypothesis takes the form:

The number on the right-hand side is our null hypothesis value that is informed by our research question. Notice that we are testing the value for m, the population parameter or mean, not the sample statistic’s mean (M). This is for two reasons: (1) once we collect data, we know what the value of M is—it is not a mystery or a question, it is observed and used for the second reason, which is (2) we are interested in understanding the population, not just our sample.

Again, keep in mind that the null hypothesis is typically the opposite or absence of the researcher’s hypothesis. It is what we do not expect to happen. In the example about doctors and patients, the researchers hypothesized that physicians would expect to spend less time with obese patients. The null hypothesis that the two types of patients are treated identically is put forward with the hope that it can be discredited and therefore rejected. If the null hypothesis were true, a large difference in the sample would be very unlikely to occur.

In general, the null hypothesis is the idea that nothing is going on: there is no effect of our treatment, no relationship between our variables, and no difference in our sample mean from what we expected about the population mean. This is always our baseline starting assumption, and it is what we seek to reject. If we are trying to treat depression, we want to find a difference in average symptoms between our treatment and control groups. If we are trying to predict job performance, we want to find a relationship between conscientiousness and evaluation scores. However, until we have evidence against it, we must use the null hypothesis as our starting point.

In our ESP example, the null hypothesis is that , since that is what we would expect if ESP does not exist. Our hope if we are studying it, of course, is that ESP is real and so the alternative to this null hypothesis is . In essence, what we are doing is dividing up the possible values of q into two groups: those values that we really hope are not true (the null), and those values that we would be happy with if they turn out to be right (the alternative). Having done so, the important thing to recognize is that the goal of a hypothesis test is not to show that the alternative hypothesis is (probably) true. The goal is to show that the null hypothesis is (probably) false. Most people find this pretty weird.

The best way to think about it might be to imagine that a hypothesis test is like an American criminal trial. The null hypothesis is the defendant, the researcher is the prosecutor, and the statistical test itself is the judge. Just like a criminal trial, there is a presumption of innocence. The null hypothesis is deemed to be true unless the researcher can prove beyond a reasonable doubt that it is false. You are free to design your experiment however you like (within reason, obviously!) and your goal when doing so is to maximize the chance that the data will yield a conviction for the crime of being false. The catch is that the statistical test sets the rules of the trial and those rules are designed to protect the null hypothesis, specifically to ensure that if the null hypothesis is actually true the chances of a false conviction are guaranteed to be low. This is pretty important. After all, the null hypothesis does not get a lawyer, and given that the researcher is trying desperately to prove it to be false, someone has to protect it.

Figure 11.4 Continuing to report on hypothesis testing is controversial to some statisticians, as humorized here^[5]

If the null hypothesis is rejected, then we will need some other explanation, which we call the alternative hypothesis, H_A or H₁. The alternative hypothesis is simply the reverse of the null hypothesis, and there are three options, depending on where we expect the difference to lie. Thus, our alternative hypothesis is the mathematical way of stating our research question.

In most cases, the alternative hypothesis is known as non-directional: We expect a difference or a relationship, but we are not sure which way the relationship or difference is going to go. In that case, our statistics will be what is called two-tailed. In other words a difference or relationship that was big could be either on the high end or on the low end of a distribution of possible scores.

If we expect our obtained sample mean to be above or below the null hypothesis value, which we call a directional hypothesis, then our alternative hypothesis takes the form:

or

based on the research question itself and is otherwise known as one-tailed. We should only use a directional hypothesis if we have good reason, based on prior observations or research, to suspect a particular direction.

We will set different criteria for rejecting the null hypothesis based on the directionality (greater than, less than, or not equal to) of the alternative. To understand why, we need to see where our criteria come from and how they relate to z scores and distributions.

Two Types of Errors

Before going into details about how a statistical test is constructed, it is important to understand the philosophy behind it. We hinted at it when pointing out the similarity between a null hypothesis test and a criminal trial, but we should now be explicit. Ideally, we would like to construct our test so that we never make any errors. Unfortunately, since the world is messy, this is never possible. Sometimes you are just really unlucky or, as Abelson (1995) said in his First Law of Statistics, chance is really lumpy. For instance, suppose you flip a coin 10 times in a row and it comes up heads all 10 times. That feels like very strong evidence for a conclusion that the coin is biased, but of course, there is a 1 in 1,024 chance that this would happen, even if the coin were totally fair. In other words, in real life we always have to accept that there is a chance that we made a mistake and just got a really unlikely outcome. As a consequence, the goal behind statistical hypothesis testing is not to eliminate errors, but to minimize them.

At this point, we need to be a bit more precise about what we mean by errors. First, we will state the obvious. It is either the case that the null hypothesis is true or that it is false, and our test will either retain the null hypothesis or reject it. So, as Table 11.2 illustrates, after we run the test and make our choice, one of four things can happen:

Table 11.2 Null Hypothesis Statistical Testing (NHST)^[6]

As a consequence, there are actually two different types of error here. If we reject a null hypothesis that is actually true, then we have made a Type I error (otherwise known as a false positive). In other words, we have said “something happens” when in real life “nothing happens. In real life, this would be like convicting an innocent person, or misreading a mammogram and saying someone has breast cancer when they do not. On the other hand, if we retain the null hypothesis when it is in fact false, then we have made a Type II error (false negative). In other words, we have said “nothing happens” when in real life, “something happens.” In real life, this would be like letting a guilty person go free or misreading a mammogram and saying someone does not have breast cancer when they do. Both Type I and and Type II errors can have grave consequences, and we do not want to make either of these kinds of errors. Please note that this is not saying the data were collected incorrectly, recorded incorrectly, or made up – those kinds of mistake by the researcher would be entirely different (and, some would say, worse) kinds of errors.

Remember how we said that statistical testing was kind of like an American criminal trial? A criminal trial requires that establishing “beyond a reasonable doubt” that the defendant did it. All of the evidential rules are (in theory, at least) designed to ensure that there is (almost) no chance of wrongfully convicting an innocent defendant. The trial is designed to protect the rights of a defendant. In other words, a criminal trial does not treat the two types of error in the same way. Punishing the innocent is deemed to be much worse than letting the guilty go free. A statistical test is pretty much the same. The single most important design principle of the test is to control the probability of a type I error, to keep it below some fixed probability. This probability, which is denoted a, is called the significance level of the test. A hypothesis test is said to have significance level a if the type I error rate is no larger than a.

So, what about the type II error rate? We would also like to keep those under control, and we denote this probability by b. However, it is much more common to refer to the power of the test, that is, the probability with which we reject a null hypothesis when it really is false, which is 1- b. To help keep this straight, here is the same table again but with the relevant numbers alphanumeric numbers added (see Table 11.3).

Table 11.3 Null Hypothesis Statistical Testing (NHST) – Additional Detail^[7]

A powerful hypothesis test is one that has a small value of b, while still keeping a fixed at some (small) desired level. By convention, scientists make use of three different a levels: .05, .01, and .001. Notice the asymmetry here; the tests are designed to ensure that the  level is kept small but there’s no corresponding guarantee regarding b.

Please watch Why 0.05? The Number That Decides What's True

Significance Level, Critical Values, and p Values

A low probability value indicates that the sample outcome (or an outcome more extreme) would be very unlikely if the null hypothesis were true. A low probability value casts doubt on the null hypothesis. How low must the probability value be in order to conclude that the null hypothesis is false? Although there is clearly no right or wrong answer to this question, it is conventional to conclude the null hypothesis is false if the probability value is less than .05. More conservative or picky researchers conclude the null hypothesis is false only if the probability value is less than .01. When a researcher concludes that the null hypothesis is false, the researcher is said to have rejected the null hypothesis. The probability value below which the null hypothesis is rejected is called the a-level or simply a (Greek letter “alpha”). It is also called the significance level. If a is not explicitly specified, assume that a = .05. This is the value we will use repeatedly for the course of the semester, since we are getting used to the language and use of statistical testing.

The significance level is a threshold we set before collecting data in order to determine whether or not we should reject the null hypothesis. We set this value beforehand to avoid biasing ourselves by viewing our results and then determining what criteria we should use. If our data produce values that meet or exceed this threshold, then we have sufficient evidence to reject the null hypothesis; if not, we fail to reject the null (we never “accept” the null).

There are two criteria we use to assess whether our data meet the thresholds established by our chosen significance level, and they both have to do with our discussions of probability. Recall that probability refers to the likelihood of an event, given some situation or set of conditions. In hypothesis testing, that situation is the assumption that the null hypothesis value is the correct value, or that there is no effect. The value laid out in H₀ is our condition under which we interpret our results. To reject this assumption, and thereby reject the null hypothesis, we need results that would be very unlikely if the null was true.

Let us say we were using a one-sample z-test, which compares a sample of a mean to a population to see if they are similar or different. In this case, we would use the following formula to calculate the z-test:

In this case M = mean of the sample, m = mean of the population, s = population standard deviation, and n = the sample size. If a z-score is zero, it indicates that the mean of the sample is the same as the mean of the population (average). If a z-score is positive, it indicates that the mean of the sample is greater than the mean of the population (above average). If a z-score is negative, it indicates that the mean of the sample is greater than the mean of the population (below average). After we calculate the z-score, we would compare the z to a standard normal distribution to see how unlikely this particular z-score is. Values of z which fall in the tails of the standard normal distribution represent unlikely values. That is, the proportion of the area under the curve as extreme as z—or more extreme than z—is very small as we get into the tails of the distribution. Our significance level corresponds to the area in the tail that is exactly equal to a. If we use our normal criterion of a = .05, then 5% of the area under the curve becomes what we call the critical region (also called the rejection region) of the distribution. This is illustrated in Figure 11.5. The shaded rejection region takes us 5% of the area under the curve. Any result that falls in that region is sufficient evidence to reject the null hypothesis.

Figure 11.5 The rejection region for a one-tailed z-test^[8]

The rejection region is bounded by a specific z value, as is any area under the curve. In hypothesis testing, the value corresponding to a specific rejection region is called the critical value (hence the other name “critical region”). If we go to a unit normal table for probabilities, we will find that the z score corresponding to 5% of the area under the curve is equal to 1.645 (z = 1.64 corresponds to .0505 and z = 1.65 corresponds to .0495, so .05 is exactly in between them) if we go to the right and −1.645 if we go to the left. The direction must be determined by your alternative hypothesis, and drawing and shading the distribution is helpful for keeping directionality straight.

Suppose, however, that we want to do a non-directional test. We need to put the critical region in both tails, but we do not want to increase the overall size of the rejection region (for reasons we will see later). To do this, we simply split it in half so that an equal proportion of the area under the curve falls in each tail’s rejection region. For a = .05, this means 2.5% of the area is in each tail, which, based on the z table, corresponds to critical values of z = ±1.96. This is shown in Figure 11.6.

Figure 11.6 Two-tailed rejection region for a z-test^[9]

Thus, any z score falling outside ±1.96 (greater than 1.96 in absolute value) falls in the rejection region. It is an extreme, unusual, or unlikely score. When we use z-scores in this way, the obtained value of z (sometimes called z obtained and abbreviated z_obt) is something known as a test statistic, which is simply an inferential statistic used to test a null hypothesis.

Figure 11.7 Relation between a, z_obt, and p^[10]

As illustrated in Figure 11.7, to formally test our hypothesis, we compare our obtained z statistic (z_obt) to our critical z value (z_crit). If z_obt > z_crit, that means it falls in the rejection region (to see why, draw a line for z = 2.5 on Figure 11.5 or 11.6. Thus, we reject H₀ – it looks like something interesting happened, rather than nothing. If z_obt < z_crit, we fail to reject the null hypothesis, or in other words, we retain (hang onto) the null hypothesis. Remember that as z gets larger, the corresponding area under the curve beyond z gets smaller. Thus, the proportion, or p value, will be smaller than the area for a, and if the area is smaller, the probability gets smaller. Specifically, the probability of obtaining that result, or a more extreme result, under the condition that the null hypothesis is true gets smaller.

The z statistic is very useful when we are doing our calculations by hand. However, when we use computer software, it will report to us a probability value (p-value), which is simply the proportion of the area under the curve in the tails beyond our obtained z statistic. We can directly compare this p value to a to test our null hypothesis: if p < a, we reject H₀, but if p > a, we fail to reject. Note also that the reverse is always true. If we use critical values to test our hypothesis, we will always know if p is greater than or less than a. If we reject, we know that p < a because the obtained z statistic falls farther out into the tail than the critical z value that corresponds to a, so the proportion (p value) for that z statistic will be smaller. Conversely, if we fail to reject, we know that the proportion will be larger than a because the z statistic will not be as far into the tail. This is illustrated for a one-tailed test in Figure 11.7.

When the null hypothesis is rejected, the effect is said to have statistical significance or to be statistically significant. It is important to keep in mind that statistical significance means only that the null hypothesis of exactly no effect is rejected; it does not mean that the effect is important, which is what “significant” usually means. When an effect is significant, you can have confidence the effect is not exactly zero. Finding that an effect is significant does not tell you about how large or important the effect is.

Do not confuse statistical significance with practical significance (importance). A small effect can be highly significant if the sample size is large enough.

Why does the word “significant” in the phrase “statistically significant” mean something so different from other uses of the word? Interestingly, this is because the meaning of “significant” in everyday language has changed. It turns out that when the procedures for hypothesis testing were developed, something was “significant” if it signified something. Thus, finding that an effect is statistically significant signifies that the effect is likely to be real and not likely due to chance. Over the years, the meaning of “significant” changed, leading to the potential misinterpretation. Also, if a finding is not statistically significant, it is called non-significant (not statistically likely) rather than insignificant (which would suggest it is unimportant).

The Hypothesis Testing Process

A Four-Step Procedure

The process of testing hypotheses follows a simple four-step procedure. This process will be what we use for the remainder of the textbook and course, and although the hypotheses and statistics we use will change, this process will not. Commit these steps of hypothesis testing to memory, as we will revisit them for every inferential statistical test that we run this semester.

Step 1: Look at the Data and State the Null and Alternative Hypotheses

Our hypotheses are the first thing we need to lay out. Otherwise, there is nothing to test! We have to state the null hypothesis (which is what we test) and the alternative hypothesis (which is what we expect). These should be stated mathematically as they were presented above and in words, explaining in everyday language what each one means in terms of the research question. Think of this as translating from math to English especially if you, like many of the authors, have ever thought that math was like a foreign language!

Step 2: Check Assumptions and Set the Critical Value(s)

Next, we consider any mathematical assumptions that exist for the particular test we want to use and formally lay out the criteria we will use to test our hypotheses. A critical value is the test statistic that exists at the particular probability value we decide is far enough away from the mean to be statistically significant, or unlikely to have happened by chance alone. There are three pieces of information that inform our critical value or values: 

• a, which determines how much of the area under the curve composes our rejection region
• the directionality of the test, which determines what the critical region or regions will be (one-tailed or two-tailed), and
• N, or sample size.

The sample size helps us calculate the degrees of freedom for the test, which will either be equal to N - 1 or N - 2, depending on the test we are running. Degrees of freedom are the number of scores that can vary in an analysis without violating any established limits or assumptions of the test. Once you have all of these pieces, you can look up the critical value(s) in a probability table for that particular test. If you are using a statistical program to run your analyses for a hypothesis test, the program calculates and utilizes the critical value(s), but it does not tend to show it in the output.

Step 3: Calculate & Report the Descriptive Statistics, Test Statistic, and Effect Size

Once we have our hypotheses and the standards we use to test them, we can collect data and calculate our test statistic—in this case, z. This step is where the vast majority of differences in future chapters will arise: Different tests used for different data are calculated in different ways, but the way we use and interpret them remains the same. As part of this step, we will also calculate effect size, or the magnitude of the difference between our groups or the relationship between variables. Does the statistic we found show no effect? Or is it small, medium, or large? Although effect size is not considered part of hypothesis testing, reporting it as part of the results is approved convention, especially in the American Psychological Association standards that guide and inform our work in psychology. A low probability value can occur with small effect sizes, particularly if the sample size is large.

Step 4: Interpret Your Findings in Relation to the Hypothesis (Translate Math to English)

Finally, once the test statistic is obtained, we can compare it to our critical value and decide whether we should reject or fail to reject (retain) the null hypothesis. When we do this, we must interpret the decision in relation to our research question, stating what we concluded, what we based our conclusion on, and the specific statistics we obtained.

Please watch: Hypothesis Testing

An Example: Movie Popcorn

Figure 11.8 A popcorn seller processing popcorn^[11]

Let us see how hypothesis testing works in action by working through an example. Say that a movie theater owner likes to keep a close eye on how much popcorn goes into each bag sold, so she knows that the average bag has 8 cups of popcorn and that this varies a little bit, about half a cup. That is, the known population mean is m = 8.00 (cups) and the known population standard deviation is s = 0.50 (cups). The owner wants to make sure that the newest employee is filling bags correctly, so over the course of a week she randomly assesses 25 bags filled by the employee to test for a difference (N = 25). She does not want bags over-filled (the theater would lose money) or under-filled (movie-goers would complain), so she looks for differences in both directions. This scenario has all of the information we need to begin our hypothesis testing procedure.

Step 1: State the Hypotheses

Our manager is looking for a difference in the mean weight of popcorn bags compared to the population mean of 8. We will need both a null and an alternative hypothesis written both mathematically and in words. We will always start with the null hypothesis:

Notice that we phrase the hypothesis in terms of the population parameter (m), which in this case would be the true average weight of bags filled by the new employee. Our assumption of no difference, the null hypothesis, is that this employee’s mean is exactly the same as the known population mean value that we want it to match: 8.00.

Next is the alternative hypothesis:

In this case, we do not know if the bags will be too full or not full enough, so we utilized a two-tailed alternative hypothesis that there is a difference, without specifying the direction.

Step 2: Find the Critical Values

Our critical values are based on three things: the directionality of the test, the level of significance, and (for some tests) the sample size. We decided in Step 1 that a two-tailed test is the appropriate directionality. We were given no information about the level of significance, so we assume that a = .05 is what we will use. As stated earlier in the chapter, the critical values for a two-tailed z test at a = .05 are z = ±1.96. These will be the criteria we use to test our hypothesis. We can now draw out our distribution, as shown in Figure 11.9, so we can visualize the rejection region and make sure it makes sense.

Figure 11.9 Rejection region for z* = ±1.96^[12]

Step 3: Calculate & Report the Descriptive Statistics, Test Statistic, and Effect Size

Now we come to our formal calculations. Let us say that the manager collects data and finds that the average weight of this employee’s popcorn bags is M = 7.75 cups. We can now plug this value, along with the values presented in the original problem, into our equation for z:

So, our calculated test statistic is z = −2.50, which we can draw onto our rejection region distribution as shown in Figure 11.10.

Figure 11.10 Test statistic (z-score) location in the z-distribution^[13]

Effect Size

When we reject the null hypothesis, we are stating that the difference we found was statistically significant, but we have mentioned several times that this tells us nothing about practical significance. To get an idea of the actual size of what we found, we can compute a new statistic called an effect size. Effect size gives us an idea of how large, important, or meaningful a statistically significant effect is. Each kind of test we run will have a different effect size. For mean differences like we calculated here, our effect size is Cohen’s d:

This is very similar to our formula for z, but we no longer take into account the sample size (since overly large samples can make it too easy to reject the null). Cohen’s d is interpreted in units of standard deviations, just like z. For our example:

Cohen’s d is interpreted as small, moderate, or large. Specifically, d = 0.20 is small, d = 0.50 is moderate (medium), and d = 0.80 is large (this is true whether Cohen’s d is positive or negative). Obviously, values can fall in between these guidelines, so we should use our best judgment and the context of the problem to make our final interpretation of size. Our effect size happens to be exactly equal to one of these, so we say that there is a moderate effect.

Effect sizes are incredibly useful and provide important information and clarification that overcomes some of the weakness of hypothesis testing. Any time we perform a hypothesis test, whether statistically significant or not, we should always calculate and report effect size as well.

Step 4: Interpret Your Findings in Relation to the Hypothesis (Translate Math to English)

Looking at Figure 11.10, we can see that our obtained z statistic falls in the rejection (critical) region. We can also directly compare it to our critical value: in terms of absolute value, −2.50 > −1.96, so we reject the null hypothesis. If we looked it up a unit normal table, we would see that a z-score of -2.50 has a probability of .005 (rounded to three decimals). We can now write our conclusion. When we write our conclusion, we write out the words to communicate what it actually means, but we also include the average sample size we calculated, the z statistic and calculated p-value, and the effect size. Our very last sentence should help translate math to English, or as one of the authors likes to say: “Now, tell my nana, my nephew, my neighbor—someone who does not ‘speak math’—what the numbers mean in everyday language.”

Based on the sample of 25 bags, the average popcorn bag from this employee is smaller (M = 7.75 cups) than the average size of popcorn bags at this movie theater (μ = 8.00 cups), z = −2.50, p = .005, d = 0.50. We would reject the null hypothesis; the alternative hypothesis is supported. The effect size was moderate. This employee seems to provide less popcorn than most employees.

Misconceptions in Hypothesis Testing

Misconceptions about significance testing are common. This section lists three important ones.

1. Misconception: The probability value (p-value) is the probability that the null hypothesis is false.
   Proper interpretation: The probability value (p-value) is the probability of a result as extreme or more extreme given that the null hypothesis is true. It is the probability of the data given the null hypothesis. It is not the probability that the null hypothesis is false.
2. Misconception: A low probability value indicates a large effect.
   Proper interpretation: A low probability value indicates that the sample outcome (or an outcome more extreme) would be very unlikely if the null hypothesis were true. A low probability value can occur with small effect sizes, particularly if the sample size is large.
3. Misconception: A non-significant outcome means that the null hypothesis is probably true.
   Proper interpretation: A non-significant outcome means that the data do not conclusively demonstrate that the null hypothesis is false.

Confidence Intervals

Previously, we discussed how scientists rely on statistics to make guesses about population parameters on the basis of a sample of data. Every data set leaves us with some uncertainty, so our estimates are never going to be perfectly accurate.

The thing that has been missing from this discussion is an attempt to quantify the amount of uncertainty that attaches to our estimate. It is not enough to be able guess that the mean intelligence quotient (IQ) of undergraduate psychology students is, say, 115 (yes, we arbitrarily made that number up). We also want to be able to say something that expresses the degree of certainty that we have in our guess. For example, it would be nice to be able to say that there is a 95% chance that the true mean lies between 109 and 121. The name for this estimate is a confidence interval for the mean.

Constructing a confidence interval for the mean is actually pretty easy. Here is how it works. Suppose the true population mean is m, and the standard deviation is s. Imagine we just finished running a study that has N participants, and the mean IQ among those participants is M (also sometimes written as X with a bar over top, as in the formula below). We know from a phenomenon known as the central limit theorem that the sampling distribution of the mean is approximately normal. We also know from our discussion of the normal distribution in Module 3 that there is a  chance that a normally-distributed quantity will fall within about two standard deviations of the true mean.

To be more precise, the more correct answer is that there is a 95% chance that a normally distributed quantity will fall within 1.96 standard deviations of the true mean. Of course, there is nothing special about the number 1.96. It just happens to be the multiplier we need to use if we want a 95% confidence interval. If we wanted a 70% confidence interval, we would have used 1.04 as the magic number rather than 1.96. As long as N is sufficiently large (large enough for us to believe that the sampling distribution of the mean is normal), then we can calculate our 95% confidence interval with this formula:

Interpreting a Confidence Interval

The hardest thing about confidence intervals is understanding what they mean. Whenever people first encounter confidence intervals, the first instinct is almost always to say that “there is a 95% probability that the true mean lies inside the confidence interval”. It’s simple and it seems to capture the common sense idea of what it means to say that I am “95% confident.” Unfortunately, this interpretation is not quite right. The intuitive definition relies very heavily on your own personal beliefs about the value of the population mean. I say that I am 95% confident because those are my beliefs. In everyday life, this it perfectly okay, but in statistics, we should be more precise. Our interpretation of a 95% confidence interval must have something to do with replication.

Specifically, if we replicated the experiment over and over again and computed a 95% confidence interval for each replication, then 95% of those intervals would contain the true mean. More generally, 95% of all confidence intervals constructed using this procedure should contain the true population mean. This idea is illustrated in Figure 11.10, which shows 50 confidence intervals constructed for a “measure 10 IQ scores” experiment (top panel) and another 50 confidence intervals for a “measure 25 IQ scores” experiment (bottom panel).

We would expect that around 95 of our confidence intervals would contain the true population mean, and that is what we found in Figure 11.11.

Figure 11.11 The 95% confidence intervals for 10 versus 25 samples of intelligence quotient (IQ) scores^[14]

In Figure 11.11, the top panel (a) shows 50 simulated replications of an experiment in which we measure the IQs of 10 people. The dot marks the location of the sample mean and the line shows the 95% confidence interval. Most of the 50 confidence intervals do contain the true mean (i.e., 100), but a few – in blue and marked with asterisks – do not. The lower graph (panel b) shows a similar simulation, but this time we simulate replications of an experiment that measures the IQs of 25 people.

When we are approaching hypothesis testing by looking at frequencies, the population mean is fixed, and no probabilistic claims can be made about it. Confidence intervals, however, are repeatable, so we can replicate experiments. Therefore, a frequentist is allowed to talk about the probability that the confidence interval (a random variable) contains the true mean but is not allowed to talk about the probability that the true population mean (not a repeatable event) falls within the confidence interval. I know that this seems a little pedantic, but it does matter. It matters because the difference in interpretation leads to a difference in the math.

Please watch Confidence Intervals: Crash Course Statistics #20

Summary

Hypothesis testing (including the null and alternative hypothesis) is one of the most pervasive elements to statistical theory and research – in other words, it is practically everywhere in all things statistics. The vast majority of scientific papers report the results of some hypothesis test or another. As a consequence, it is almost impossible to get by in science without having at least a cursory understanding of what a p-value means, even if you are a statistician who argues that science should more away from using p-values at all or exclusively in understanding statistics (e.g., Ahmed & Butt, 2025; American Statistical Association, 2016; Wang & Long, 2022).

Additional Video Helps

• How P-Values Help Us Test Hypotheses: Crash Course Statistics #21
• P-Value Problems: Crash Course Statistics #22
• Playing with Power: P-Values Pt 3: Crash Course Statistics #23
• What Confidence Intervals Actually Mean

Additional Resources

• Unit Normal Table for Z-scores: See Appendix A of Cote, L. R., Gordon, R. G. Randell, C. E., Schmitt, J., & Marvin, H. (2021). Introduction to statistics in the psychological sciences.

Media Attributions

^[1] (CC attribution 3.0 unported by Tobascoman77 via Wikimedia Commons)

^[2] (CC attribution 2.0 generic by Dave Fayram via Wikimedia Commons)

^[3] (“P-Values” by Randall Munroe/xkcd.com is licensed under CC BY-NC 2.5)

^[4] (CC BY-SA 4.0 by Navarro & Foxcroft, 2025)

^[5] (“Null Hypothesis” by Randall Munroe/xkcd.com is licensed under CC BY-NC 2.5)

^[6-7] (CC BY-SA 4.0 by Navarro & Foxcroft, 2025)

^[8] (Rejection Region for One-Tailed Test” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[9] (“Rejection Region for Two-Tailed Test” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[10] (“Relationship between alpha, z-obt, and p” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[11] (CC BY-NC-SA 4.0 by Rwebogora via Wikimedia Commons)

^[12] (“Rejection Region z+-1.96” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[13] (“Test Statistic Location z-2.50” by Judy Schmitt is licensed under CC BY-NC-SA 4.0.)

^[14] (CC BY-SA 4.0 by Navarro & Foxcroft, 2025)

Text Attributions

Cote, L. R., Gordon, R. G. Randell, C. E., Schmitt, J., & Marvin, H. (2021). Introduction to statistics in the psychological sciences. Pressbooks. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

Navarro, D. J., & Foxcroft, D. R. (2025). Learning statistics with jamovi: A tutorial for beginners in statistical analysis. Open Book Publishers. Licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, except where otherwise noted. Modified by current authors.

References

Abelson, R. P. (1995). Statistics as principled argument. Psychology Press (a Taylor & Francis Group).

Ahmed, E. S., & Butt, M. N. (2025). The misunderstood p-value: Why statistical significance is not enough in clinical practice. British Journal of Anaesthesia, 134(4), 909-913. https://doi.org/10.1016/j.bja.2025.01.008

American Statistical Association (2016, March 7). American Statistical Association releases statement on statistical significance and p-values. https://www.amstat.org/asa/files/pdfs/p-valuestatement.pdf

CrashCourse (2018, June 13). Confidence intervals: Crash course statistics #20. https://www.youtube.com/watch?v=yDEvXB6ApWc

Foundations of Math. (2025, September 10). Why 0.05? The number that decides what's true. https://www.youtube.com/watch?v=CS7nGURwdC4

Global Health with Greg Martin (2022, February 22). Hypothesis testing. https://www.youtube.com/watch?v=S2eKynREGM4

Popper, K. (1959). The logic of scientific discovery. Hutchinson.

Scribbr (2019, November 15). 6 steps to formulate a STRONG hypothesis. https://www.youtube.com/watch?v=PCgLjDDD4ek

Wang, M., & Long, Q. (2022). Addressing common misuses and pitfalls of p-values in biomedical research. Cancer Research, 82(15), 2674-2677. https://doi.org/10.1158/0008-5472.CAN-21-2978

Chapter 12: Correlation and Regression

Correlation and regression are both used to examine relations among variables, but they answer different kinds of questions. Correlation describes the strength and direction of the relation between two variables. Regression uses one or more predictor variables to predict or explain variation in a continuous outcome variable.

A key difference between correlation and regression is that correlation treats the two variables symmetrically. We ask whether two variables are related, but we do not label one variable as the predictor and the other as the outcome. Regression is inherently directional in its setup: one variable is the outcome, and one or more variables are used as predictors.

Table 12.1 Overview of Correlation and Regression Designs^[1]

We will continue using the same four-step hypothesis-testing process introduced in the previous chapter: look at the data and state the null and research hypotheses; check assumptions and set critical value(s); calculate and report the descriptive statistics, test statistic, and effect size; and interpret the results in relation to the hypotheses (translate math to English). The main difference is that we are now asking about relations among variables rather than comparing group differences.

One important caution applies throughout this chapter: relations do not automatically imply causation. A correlation or regression coefficient can show that variables are related, but causal conclusions depend on the research design, measurement quality, and alternative explanations.

Correlation

Correlation is used to describe the relation between two variables. In this section, we focus mostly on the Pearson correlation, which is used when we want to examine the linear relation between two continuous (interval or ratio) variables.

The correlation coefficient, r, developed by Karl Pearson in the early 1900s, is numerical and provides a measure of strength and direction of the linear association between the independent variable (x) and the dependent variable (y).

The correlation coefficient is calculated as

where n = the number of data points, x = the value of one variable, and y = the value of a second variable.

A correlation can tell us three things about a relation:

1. Direction: whether the relation is positive or negative.
2. Strength: how closely the variables are related.
3. Statistical significance: whether the observed relation is unlikely to have occurred by chance if there were no relation in the population.

Correlation coefficients range from -1 to +1. Anything beyond -1 or +1 is an impossible value. You cannot, for instance, have a correlation coefficient equal to -1.67 or +25.3.

• A positive correlation means that higher scores on one variable tend to go with higher scores on the other variable.
• A negative correlation means that higher scores on one variable tend to go with lower scores on the other variable.
• A correlation close to 0 means there is little or no linear relation between the variables.

Correlation does not show causation. A statistically significant correlation tells us that two variables are related, but it does not tell us whether one variable caused the other.

Pearson correlations are standardized, so they are also effect sizes. As a rough heuristic, values around (|.10|) are often described as small, values around (|.30|) as medium, and values around or beyond (|.50|) as large. These are only general guidelines. The importance of a correlation depends on the research context.

Scatterplots (or Scatter Plots)

The way we visualize a relation is by a type of graph called a scatterplot. A scatterplot looks at the relationship between two factors. Each dot in the graph is a respondent (participant) who has provided answers to both of the questions at hand. For the correlations we are doing, we want to be able to see or imagine a roughly straight line, rather than a wide cloud of dots (or what you might think of as a big chocolate chip cookie). The stronger the correlation, the more the dots will fall into or cluster around the regression line, otherwise known as the line of best fit, a visual representation of the relationship between the two factors. Remember: A correlation cannot be greater than +1 or lower than -1. Those are its limits.

Figure 12.1 Scatter Plots and Correlation Examples Ranging from -1 to 0 to 1^[2]

Please watch: What Are Correlations? (Daniel Storage)

Please watch: Pearson’s r correlation (Statistics Lectures)

Step 1: Look at the Data and State the Null and Research Hypotheses

Let’s turn to a topic close to every parent’s heart: sleep. The data set we’ll use is fictitious but based on real events. Suppose we’re curious to find out how much an infant’s sleeping habits affect a parent’s mood. Let’s say that the parent rates their grumpiness very precisely, on a scale from 0 (not at all grumpy) to 100 (grumpy as a very, very grumpy old man or woman). Let’s also assume that the parent has been measuring their grumpiness, their sleeping patterns, and their infant’s sleeping patterns for quite some time now. Let’s say, for 100 days. The main variables we will use are:

• dan.grump: the parent’s grumpiness, measured from 0 to 100
• dan.sleep: the parent’s sleep quality
• baby.sleep: the baby’s sleep quality

The dataset also includes day, which records the day of the study from 1 to 100.

Data Set-Up

To conduct a Pearson correlation in a statistical program, the dataset needs two continuous variables. Each row should represent one participant, case, or unit of analysis, and each case should have a score on both variables.

In this example, each row represents one day. The variables dan.grump, dan.sleep, and baby.sleep are continuous variables.