Lab 9Last Lab

Statistical Analyses & Threats to Validity

Background

Recall from last lab that there are two major types of designs.
If

you are interested in the association between continuous variables, you should use a correlational design. If you are interested in comparing two (or more) groups, it is likely you will use an experimental design.

The design you use will determine:

statistical

analysis you will perform threats to validity you will most likely need to address

Correlational/Observational Design

In this design, you are not comparing groups. Rather, you are examining the association between two continuous variables. You can use either correlation or regression analysis to analyze your data.

Correlation

Used when interested in assessing the strength of association between two variables

Used with continuous variables

Example research hypothesis: are higher levels of self-esteem associated with lower levels of depression (what direction correlation would be expected?)

Regression
Simple Linear Regression is a statistical method concerned with the amount of variance that one variable accounts for in another variable (that is, the amount of shared variance between variables). You also want to use regression when you are predicting one variable from the other. Used when you have continuous variables (e.g., motivation measured on a Likert scale, body weight).

Regression
Shared variance

Variable X

Variable Y

http://www.math.csusb.edu/faculty/stanton/m262/regress/regress.html

Correlation vs. Regression

Both correlation and regression are concerned with the amount of variance that variables share The major difference is that with correlation we are only interested in an association between variables, but with regression we assign one variable to be the independent variable (x; the predictor) and the other to be the dependent variable (y; the criterion)

Note: While we typically are concerned with predictive relationships when using regression, the certainty with which we can say that two variables are causally related depends upon the design (true experiments are best!...why?) and quality of the study

Comparing Groups

If you have an experimental design (manipulation of IV), you should use the following analyses to compare groups (noncontinuous variables). In some cases, you will want to compare groups even when you dont manipulate anything.
Comparing

male versus female on performance. Examining differences in opinion of various political or religious groups.

Independent samples (betweengroups) t-test

Used to determine dependent variable mean differences between two different groups Used with categorical or dichotomous variables (e.g., gender, democrat or republican, yes/no responses) Only one independent variable with only two levels/groups Example research hypothesis: do women have higher levels of depression on average than do men?

Analysis of Variance (ANOVA)

Like t-test, used to determine dependent variable mean differences between groups/levels, but can use when there are more than two groups (which is the same as more than two levels of your IV) Remember, though, youre still only using one IV, just there are more than two groups/levels (e.g., you might have the IV of political party, with the groups Democrat, Republican, and Independent)

Analysis of Variance (ANOVA)

Can use for your final project if you have just one IV with more than two groups (i.e., one-way ANOVA), but can be used for more than one IV (i.e. two-way ANOVA, where say you have political party [3 groups] that may differ by gender [2 groups] for a 2x3 two-way ANOVA) Independent variable(s) have two or more levels; if you only have one IV with two levels, though, use a t-test Example research question: Do psychology graduate students have higher IQs than medical students, physics graduate students, and English

Repeated measures (withingroups) t-test or ANOVA

Used to test mean differences in the DV for the group measured at two or more times (e.g., the same group of clients is exposed to every type of counseling [the IV] and symptoms [the DV] are measured) Remember, the t-test involves an independent variable with two levels; ANOVA involves independent variable(s) with more than two levels. Population parameters and sample statistics the same as between-group t-test and ANOVA above.

Repeated measures (withingroups) t-test or ANOVA

Example research hypothesis for repeatedmeasures t-test: Do people like the taste of Pepsi more than Coke (same people would taste both types of pop)? Example research hypothesis for repeatedmeasures ANOVA: Does exposure to death metal music lead to better cognitive performance than exposure to jazz, classical, or country music (same people would listen to each type of music [IV] and perform a cognitive task [DV])?

THREATS TO VALIDITY

Threats to Internal & External Validity

Keep in mind that in science it is always important to balance multiple concerns.

Ethics:

Harm versus Benefit Sampling: Representativeness versus Practicality

When choosing a research design, it is important to address whether:

You

can address or eliminate alternative explanations for your results (Internal Validity). You can generalize your results (External Validity).

Research Designs & Validity

In general, experiments with random assignment are high on internal validity.

Why?

(What does random assignment do?)

Correlational studies are often higher than experiments on external validity but lower on internal validity. Ideally, you would want studies to both be high internal and external validities.
No

single method does this. Multiple methods important.

An example

You are interested in studying why individuals help each other. You believe that sadness leads to helping because it makes individuals feel better. Lets say you conduct your study on September 12, 2001. How do you think this would influence your results? To the extent that other explanations (outside of sadness) can account for the associations that you find, your internal validity is threatened.

Internal Validity Threats

Given that the IV is related to the DV, is it plausibly causal from one operational variable to another?

History is there something about the experience & history of one group that could lead to differences? Maturation would natural development influence the associations? Testing does actual participation in the study change behavior or responses? Instrumentation what happens if the instruments you use change during the course of a study or if different groups of participants interpret your questions differently? Regression towards the mean if you start with extreme groups, they are likely to be less extreme at another time point. Selection are participants in your study somehow different than those who are not? Mortality those who drop out of your study could be qualitatively different than those who remain.

Internal Validity Threats

Given

that the IV is related to the DV, is it plausibly causal from one operational variable to another?
Diffusion or imitation what would happen if participants in the different conditions communicated? Interaction with Selection what if any of the above happen differentially for one group? Direction of causality can the association happen in the opposite direction than what you predicted? Compensatory equalization of treatments what if experimenters know what conditions participants are in? Compensatory rivalry what would happen if participants in the different conditions started competing? Resentful demoralization is it possible that participants in one condition give up?

External Validity Threats

Given that there is probably a causal relationship from construct A to construct B, how generalizable is this relationship across persons, settings, and times?
Interaction

of selection and treatment (sample to

population) Interaction of setting and treatment Interaction of history and treatment (time)

For your paper

You will need to discuss the internal and external threats to validity in the discussion of your paper. Keep in mind that no study is perfect. However, when you bring up threats to validity, you MUST explain:
How

it is a threat to validity How your results would change if the threat were addressed What you could do to address the threat

In other words, DO NOT simply state the threat.

Homework 9

For each of the three research hypotheses described below, list what type of statistical test you would use to test it (independent (between-groups) t-test, between-groups ANOVA, repeated measures t-test, repeated measures ANOVA, correlation, regression), and whether external or internal validity is most threatened and why (6 points; 2 for each hypothesis). Research Hypotheses: Women who experience a great deal of stress during pregnancy are more likely to give birth to low birth weight or premature babies than pregnant women who do not experience a great deal of stress.

The degree to which someone believes in God relates to his/her psychological well-being. Further, the more someone believes

Homework 9

Research Hypotheses: A new manualized (standardized) treatment for depression will decrease depressive symptoms in a group of people seeking help at a mental health clinic. They will be assessed when they come into the clinic, immediately after the treatment is completed, and 6 months after treatment is complete. Lastly, describe your hypothesis, what design you will use for your project (from previous homework), what statistical analysis you plan to conduct, and identify 2 threats to validity (any combination of external or internal) and how they could influence the interpretation of your results. (4 points)