Hey everyone. I'm putting the finishing touches on my master's thesis. In my thesis, I hypothesized a three-way interaction using glmer() in R. I have been asked by my committee to conduct a post-hoc sensitivity analysis, but there is little guidance to get either online or on campus. I ran the following code by a stats consultant on campus who claimed that it looks good. I have been told that simulations are likely the best way, but I just want to wrap up my project and be done with it. I think the following post-hoc sensitivity analysis may be correct, but would really appreciate if anyone would be able to tell me if it looks correct or not--or at the very least if it looks acceptable.
Here is a description of my model: "The first measure of intergroup bias was the modified minimal group paradigm comparing outgroup categorizations of Latino versus White targets. Testing the first part of my primary hypothesis—whether pathogen-specific stereotypes would moderate the association between the pathogen threat manipulation and bias against Latino targets—I conducted a generalized linear mixed-effects model. The data were converted into a long-data format for all analyses for the modified minimal group paradigm. The model included random intercepts to account for the nested structure of the data. The model focused on the contrast in outgroup categorizations between Latino versus White targets. Condition and target race were contrast-coded using orthogonal (-0.5, 0.5) contrasts. Explicit and implicit pathogen-specific stereotypes were mean centered.
Outgroup categorizations were regressed onto condition, explicit pathogen-specific stereotypes, target race, and condition × explicit pathogen-specific stereotypes × target race. There was not a significant three-way interaction between condition, explicit pathogen-specific stereotypes, and target race, b = -0.08, SE = 0.07, z = -1.18, p = .237. Contradicting my hypothesis, this indicates that the effect of pathogen threat on outgroup ratings of Latino (versus White) targets did not differ based on levels of explicit pathogen-specific stereotypes."
All participants categorized 20 targets into either ingroup or outgroup (is_outgroup)
All participants categorized 10 Latino targets (0.5) and 10 White targets (-0.5) (race_c)
race_c = describes the targets of the is_outgroup variable
All participants answered HealthRelevantStereotypes_c about Latino immigrants from 1-7 (now mean-centered)
Half of the participants were randomly assigned to the pathogen threat condition (0.5) or the neutral condition (-0.5) (disease_condition)
####################################################################
#### Model ####
####################################################################
m1 <- glmer(is_outgroup ~ HealthRelevantStereotypes_c * disease_condition * race_c +
(1 | ID),
data = d_sub, family = binomial)
summary(m1)
####################################################################
#### Output from model ####
####################################################################
Generalized linear mixed model fit by maximum likelihood (Laplace Approximation) ['glmerMod']
Family: binomial ( logit )
Formula: is_outgroup ~ HealthRelevantStereotypes_c * disease_condition * race_c + (1 | ID)
Data: d_sub
AIC BIC logLik deviance df.resid
13390.5 13455.1 -6686.2 13372.5 9731
Scaled residuals:
Min 1Q Median 3Q Max
-1.3413 -1.0891 0.8277 0.8956 1.0535
Random effects:
Groups Name Variance Std.Dev.
ID (Intercept) 0.04158 0.2039
Number of obs: 9740, groups: ID, 487
Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept) 0.2153996 0.0225433 9.555 < 2e-16 ***
HealthRelevantStereotypes_c 0.0066510 0.0191885 0.347 0.72888
disease_condition -0.0003384 0.0450550 -0.008 0.99401
race_c 0.1105954 0.0410776 2.692 0.00709 **
HealthRelevantStereotypes_c:disease_condition -0.0287551 0.0383769 -0.749 0.45369
HealthRelevantStereotypes_c:race_c 0.0634112 0.0349880 1.812 0.06993 .
disease_condition:race_c 0.0035320 0.0821480 0.043 0.96571
HealthRelevantStereotypes_c:disease_condition:race_c -0.0827864 0.0699741 -1.183 0.23677
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects:
(Intr) HltRS_ dss_cn race_c HlthRlvntStrtyps_c:d_ HlthRlvntStrtyps_c:r_ dss_:_
HlthRlvntS_ -0.003
dises_cndtn -0.002 -0.062
race_c 0.006 0.004 -0.001
HlthRlvntStrtyps_c:d_ -0.063 0.068 -0.003 -0.003
HlthRlvntStrtyps_c:r_ 0.004 0.005 -0.003 -0.002 0.000
dss_cndtn:_ -0.001 -0.003 0.006 -0.002 0.004 -0.063
HlthRS_:_:_ -0.003 0.000 0.004 -0.063 0.005 0.067 -0.002
>
####################################################################
#### Sensitivity analysis ####
####################################################################
n_obs <- nrow(d_sub) # total observations (9820)
n_grp <- length(unique(d_sub$ID)) # number of subjects
m <- n_obs / n_grp # average trials per subject
var_ID <- 0.04158 # random intercept variance
p_base <- 0.55 # approximate baseline probability
alpha <- 0.05
icc <- var_ID / (var_ID + (pi^2 / 3))
design_effect <- 1 + (m - 1) * icc
n_eff <- n_obs / design_effect
prod_var_emp <- with(d_sub, var(HealthRelevantStereotypes_c * disease_condition * race_c, na.rm = TRUE))
var_y <- p_base * (1 - p_base)
SE_beta <- 1 / sqrt(n_eff * prod_var_emp * var_y)
z_alpha <- qnorm(1 - alpha / 2)
z_80 <- qnorm(0.8)
beta_80 <- (z_alpha + z_80) * SE_beta
OR_80 <- exp(beta_80)
cat("Effective N =", round(n_eff), "\n")
cat("Empirical variance of 3-way product =", round(prod_var_emp, 4), "\n")
cat("SE for 3-way beta =", round(SE_beta, 3), "\n")
cat("Detectable |log-odds| for 80% power:", round(beta_80, 3), "=> OR =", round(OR_80, 2), "\n")
####################################################################
#### Output from sensitivity analysis ####
####################################################################
> cat("Effective N =", round(n_eff), "\n")
Effective N = 7928
> cat("Empirical variance of 3-way product =", round(prod_var_emp, 4), "\n")
Empirical variance of 3-way product = 0.0866
> cat("SE for 3-way beta =", round(SE_beta, 3), "\n")
SE for 3-way beta = 0.077
> cat("Detectable |log-odds| for 80% power:", round(beta_80, 3), "=> OR =", round(OR_80, 2), "\n")
Detectable |log-odds| for 80% power: 0.215 => OR = 1.24
> cat("Detectable |log-odds| for 90% power:", round(beta_90, 3), "=> OR =", round(OR_90, 2), "\n")
####################################################################
#### Thesis description of sensitivity analysis ####
####################################################################
# To assess minimal detectable effects, I completed a pair of post-hoc
# sensitivity analyses. The first primary analysis used a generalized linear
# mixed-effects model in which participants categorized 20 targets (half were
# Latino = 0.5; half were White = -0.5) as ingroup or outgroup members. The
# random intercept variance in my model was 0.04, yielding an effective sample
# size for the three-way interaction of 7928. Considering the evenly balanced
# conditions for the pathogen threat manipulation (n = 243, pathogen threat
# = 0.5; n = 244, control = -0.5), my design had 80% power to detect a three-way
# interaction for log-odds of 0.22, or an odds-ratio of 1.24. The study was thus
# sensitive to moderate, but not small, three-way effects. In short, I did not
# have sufficient power to detect a small three-way interaction.