Research and program evaluation, without the jargon

A counselor predicts that clients who practice mindfulness daily will report LOWER anxiety than clients who do not. This statement is BEST described as a

1. null hypothesis
   The null hypothesis states that there is no difference or relationship, the opposite of this prediction.
2. operational definition
   An operational definition specifies how a variable is measured, such as an anxiety scale score; it does not make a prediction.
3. nondirectional research hypothesis
   A nondirectional hypothesis predicts only that groups will differ, without saying which way.
4. directional research hypothesis
   A directional hypothesis states which group will score higher or lower, as this one predicts lower anxiety for daily practitioners.

In a study of a new tutoring program, teachers of the control classes learn they are being compared with it and push their own students extra hard to outperform the program group. Which threat to internal validity is this?

1. Resentful demoralization
   Resentful demoralization is the opposite reaction: the comparison group gives up because it feels deprived of the treatment.
2. Diffusion of treatment
   Diffusion happens when the comparison group is exposed to the treatment itself, such as borrowing its materials.
3. Compensatory rivalry
   Compensatory rivalry (the John Henry effect) occurs when the comparison group works harder because it knows it is competing, shrinking the apparent effect.
4. Hawthorne effect
   The Hawthorne effect is a change caused by knowing one is being studied, not competition against another group.

Which correlation coefficient shows the STRONGEST relationship between two variables?

1. -.80
   Strength is the distance from zero regardless of sign, so −.80 is the strongest; the minus sign only shows that the variables move in opposite directions.
2. .00
   .00 means no linear relationship at all, the weakest possible value.
3. +.65
   +.65 is a moderate-to-strong positive relationship but is closer to zero than −.80; assuming positive means stronger is the classic error.
4. +.40
   +.40 is a weaker positive relationship, farther from ±1.00 than either −.80 or +.65.

A counselor delivers a coping-skills program in one existing classroom but not in another, then compares end-of-year anxiety scores. Students were not randomly assigned. What type of design is this?

1. Ex post facto design
   An ex post facto (causal-comparative) study compares groups that already differ, with no intervention by the researcher; this counselor runs the program.
2. True experimental design
   A true experiment requires random assignment to conditions, which the stem rules out.
3. Correlational design
   A correlational design measures existing variables without introducing any intervention; here the counselor delivers a program to one group.
4. Quasi-experimental design
   The counselor introduces the intervention but uses intact groups instead of random assignment, which defines a quasi-experiment.

A treatment shows strong results in a randomized trial with strict inclusion criteria at a university clinic. A later study tests it with typical clients in community agencies. The second study examines the treatment's

1. efficacy
   Efficacy is how well a treatment works under controlled, ideal conditions, which is what the original university trial showed.
2. effectiveness
   Effectiveness research asks whether a treatment works in real-world settings with typical clients and clinicians, as the community study does.
3. treatment fidelity
   Treatment fidelity is how closely clinicians deliver the treatment as designed; the second study's question is whether it still works with typical clients.
4. internal validity
   Internal validity is confidence that the treatment caused the change; moving to typical settings mainly tests generalizability.

A researcher lives in a rural community for a year, observing daily life and interviewing residents to learn how they understand and seek help for mental illness. Which qualitative tradition is this?

1. Phenomenology
   Phenomenology explores how individuals experience something, but through interviews about the experience, not a year of living in the community.
2. Case study
   A case study examines a bounded case in depth; describing a community's shared ways of understanding illness through immersion is ethnography.
3. Grounded theory
   Grounded theory aims to generate a theory from coded data; the researcher here seeks to understand a community's shared culture.
4. Ethnography
   Ethnography studies a cultural group's beliefs and practices through prolonged immersion, participant observation, and interviews in the field.

A study reports p = .03 with alpha set at .05. What is the correct interpretation?

1. The treatment effect is large and clinically meaningful
   Significance does not show size or practical importance; that requires an effect size and clinical judgment.
2. Reject the null hypothesis at the .05 significance level
   Because p (.03) is below the preset alpha (.05), the result is statistically significant and the null hypothesis is rejected.
3. The null hypothesis has a 3% chance of being true
   A p value is the probability of results this extreme if the null were true, not the probability that the null is true.

An evaluator tracks whether sessions were delivered on schedule, who attended, and how closely staff followed the curriculum. Which type of evaluation is this?

1. Impact evaluation
   Impact evaluation asks about broader or long-term effects attributable to the program, again a question of results rather than implementation.
2. Outcome evaluation
   Outcome evaluation measures changes in participants, such as reduced symptoms, not delivery and attendance.
3. Process evaluation
   Process evaluation looks at how a program runs: whether sessions are delivered as planned, who attends, and how closely staff follow the curriculum.

A cross-sectional study finds that 70-year-olds report less comfort with telehealth than 30-year-olds. Which rival explanation is the main limitation of this design?

1. Regression toward the mean
   Regression toward the mean needs selection on extreme scores and a retest; neither occurs in a single cross-sectional survey.
2. Cohort effects
   Cohort effects arise because a cross-sectional study compares different generations at one time, so the gap may reflect their different exposure to technology rather than aging.
3. Practice effects
   Practice effects arise when the same people are tested repeatedly; a cross-sectional study tests each person once.

A career center gathers participant feedback midway through a workshop series to improve the remaining sessions. Which type of evaluation is this?

1. Cost-benefit analysis
   Cost-benefit analysis compares a program's costs with the monetary value of its results, not participant feedback for improvement.
2. Summative evaluation
   Summative evaluation comes at the end to judge a program's overall worth or results.
3. Formative evaluation
   Formative evaluation is done while a program is running in order to improve it, such as mid-series feedback to adjust later sessions.

First graders in a 9-month program for reading anxiety show improvement, but children naturally gain confidence over a school year. Which threat to internal validity is this?

1. Maturation
   Maturation is natural change within participants over time, such as children gaining confidence across a school year, which could mimic a program effect.
2. Regression toward the mean
   Regression applies when people are chosen for extreme scores; the stem points instead to normal development over nine months.
3. Selection
   Selection concerns pre-existing differences between groups; the problem described is change that would happen in any first grader.

A career program surveys graduates six months later, and mostly employed graduates respond. Which threat MOST limits the conclusion that the program worked?

1. History
   History is an outside event during the study, such as a local hiring boom, that could affect outcomes; the stem points to who answered the survey.
2. Instrumentation
   Instrumentation is a change in the measuring tool, or in how it is scored, between measurements; a single survey given once has no such change.
3. Attrition
   Attrition, or nonresponse, biases results when those who drop out differ from those who stay; if mostly employed graduates answer, the success rate is inflated.

To reach consensus on telehealth supervision competencies, an evaluator sends several rounds of anonymous questionnaires to experts, sharing summarized feedback between rounds. Which method is this?

1. Nominal group technique
   The nominal group technique brings people together face to face to generate, discuss, and rank ideas, not anonymous mailed rounds.
2. Delphi technique
   The Delphi technique collects expert judgment through anonymous questionnaire rounds, feeding back summarized responses until consensus emerges.
3. Key informant interview
   Key informant interviews gather in-depth views from individuals one at a time, without rounds of shared feedback aimed at consensus.

An agency tests whether clients' preferred session times (morning, afternoon, or evening) are equally distributed. Which test is appropriate?

1. Chi-square goodness-of-fit test
   Goodness of fit compares observed frequencies across the categories of one nominal variable with expected frequencies, here an equal split.
2. Chi-square test of independence
   The test of independence asks whether two categorical variables are related; there is only one variable here, preferred time.
3. Kruskal-Wallis test
   Kruskal-Wallis compares ranked scores across three or more groups; the data here are counts in categories, not scores.

A distribution of test scores has a standard deviation of 6. What is the variance?

1. 36
   Variance is the standard deviation squared: 6 × 6 = 36.
2. 2.45
   2.45 is the square root of 6, reversing the relationship: the SD is the root of the variance, not the other way around.
3. 6
   6 is the standard deviation given in the stem; variance is a different statistic.

A voluntary after-school career program reports much higher college enrollment than the school average. Which threat to internal validity should the evaluator consider FIRST?

1. Instrumentation
   Instrumentation is a change in the measuring tool or observers over time; college enrollment is counted the same way for everyone.
2. Selection bias
   Selection bias is the first threat: students chose to join, so they may have differed from others in motivation or family support before the program began.
3. Statistical regression
   Regression applies when participants are chosen for extreme scores; these students volunteered.