The Research Experience. Ann Sloan Devlin
the variables of interest. Here is an example. A college asks a researcher to test the impact of replacing mattresses in the residence halls on students’ self-reported quality of sleep. The students are randomly assigned to residence halls (a good start), but there is only enough money to replace half the mattresses, and the Facilities Department at the college has decided to replace them only in one contiguous area of campus (“south” campus) because it will make the installation process easier. The mattresses in north campus residence halls will remain the same for the year. As it turns out, north campus residence halls are near a major traffic artery, and noise from the highway is audible. Can you see the problem here? The location of the installation (south vs. north campus) systematically varies with the new vs. old condition of the mattresses. Therefore, location is a confounding variable. If the self-reported quality of sleep is better for students in south vs. north campus, you can’t rule out the possibility that it is the noise level rather than the mattress quality that makes the difference. In the case of the mattresses, random distribution of the new mattresses across both south and north campus would help to disentangle the issue of noise, and location would be included as a variable in the analysis (see Chapter 3 for a discussion of independent and quasi-independent variables).
Extraneous variable: A typically unwanted variable that exists in the context of the study but is not being measured.
Third variable: Variable that influences the relationship between the variables of interest; also called a confounding variable.
It would be impossible to measure every extraneous variable to avoid the issue of a confounding or third variable. Instead, the researcher needs to measure “the likely suspects,” and/or otherwise control for them in the research design. The likely suspects are typically those the literature suggests might be related to the variables being studied. In addition, the researcher’s observations of aspects of the situation (e.g., the location of the highway in our mattress example) are important.
As another example, would you predict a strong relationship between job performance and employee job satisfaction? Most of us would because we imagine that people who are satisfied with their jobs are productive. Yet the relationship between performance and job satisfaction is relatively small (Iaffaldano & Muchinsky, 1985). Thinking about this relationship more deeply, many factors are associated with why people are productive in their jobs, not just job satisfaction.
Try This Now 2.6
What reasons might people have for being productive at work and for keeping their jobs?
There are many reasons: benefits (including health care); status; no other job prospects; and commuting time. Thus, pressures, concerns, or other unmeasured variables may influence the relationship between job productivity and worker satisfaction, and if these variables are systematically related to the variables of interest (here job performance and employee job satisfaction), we have a problem with a confounding relationship. For example, one likely candidate that may systematically vary with job performance and job satisfaction is the amount of independence to make decisions on the job that a worker has. Another might be the satisfaction a worker has with life at home (which could spill over to work). Thus, several explanations exist for any relationship. These other variables are a problem in research. We want to rule out as many of these as we can by taking steps before we collect data. We do so by (a) refining the research question and closing the research gap and/or (b) accounting for potentially confounding variables in the design of our study.
When the variables under study in a relationship are somewhat broadly defined (e.g., satisfaction with participation in athletics and student academic achievement), many other variables may be potential candidates in explaining that relationship. You want to limit or control for the number of other variables that may explain the relationship in question, without limiting the research question to the point that it has no external validity (see Chapter 3) or application beyond the particular situation examined.
External validity: Ability to apply the results of research more broadly, that is, beyond the sample used in a particular study. Generalizability is a major emphasis of external validity.
In the example of the relationship between satisfaction with athletic participation and student academic achievement, there are a number of factors that may be connected to both of these variables. One may be the relationship with the coach; another may be whether participation involves an individual or team sport; a third may be the demands of a given sport (e.g., some sports are single season; other sports cut across seasons; others, such as sailing, have heavy travel schedules). Narrowing the research question to focus on a more specific variable or set of variables may enable the researcher to better rule out alternative explanations for the relationship. Here, the relationship with the coach may be a good candidate because a coach might serve as a mentor for both athletics and academics. Such research might help us better understand the role coaches play in the lives of student-athletes and their academic success, beyond what could be learned by asking about general satisfaction with athletic participation, because general satisfaction is a broad construct that itself includes many components.
To close the research gap even further, it might be prudent to tailor the research question to focus both variables on athletics, for example, examining the relationship with the coach and the satisfaction with participation in college athletics. Thus, one way to refine the research question is to target variables where a strong argument can be made about the logic of the relationship. As R. Kaplan (1996) has noted, the project needs to be “about something—not everything” (p. 172).
One of the most challenging aspects of doing a research methods project is the pressure of time and the problem of developing a sound research idea. While the scale of projects done in research methods courses tends to be small, there are aspects of the process that cannot be short-changed, whatever the scale. One of these is conceptualization, or the idea that is being tested. As R. Kaplan (1996) stated, “The single most damaging problem in attempts to do small experiments stems from devoting too little effort to thinking about what one hopes to learn. Conceptualization is probably the most ignored, most essential aspect of doing a successful small experiment” (p. 172). One good question Kaplan recommends that you keep asking yourself as your idea develops is “What [do] you want to be able to say when you are done?” (p. 172).
But in addition to closing the research gap, there may be other relevant variables you need to assess. By including background or demographic variables describing the participants (e.g., high school GPA, gender, or class year), you will reduce the number of relevant unmeasured variables. You cannot include an endless list of demographic questions; be mindful of the “likely suspects.” Select those variables from the literature most likely to be relevant (and that you thus need to measure; see Figure 2.12). In our example involving the relationship with the coach and student achievement (or satisfaction with athletics), it would seem prudent to ask about such relevant background variables as the student’s major and whether the student was a recruited athlete.
Demographic variables: Background variables (e.g., gender or age) about participants that are used to describe the sample and may be used in analyses in quasi-experimental research.
Revisit and Respond 2.6
Explain what a third variable is and why it makes sense to narrow the research gap.
What role do demographic variables play in narrowing the research gap?
Where should you look to determine which demographic variables to include?
Figure 2.12 Diagram Illustrating the Idea of Closing the Research Gap
Time