\title{dataset on situation awareness and task performance associations}
\description{results from 77 papers with 678 effects evaluating associations among measures of situation awareness and task performance.}
\bold{author}            \tab \code{character} \tab paper author(s) \cr
\bold{year}              \tab \code{integer}   \tab year of paper publication \cr
\bold{title}             \tab \code{character} \tab title of paper \cr
\bold{doi}               \tab \code{character} \tab digital object identifier (doi) \cr
\bold{dtic.link}         \tab \code{character} \tab permanent link for defense technical information collection (ditc) reports; see: \verb{https://www.dtic.mil} \cr
\bold{sa.measure.type}   \tab \code{character} \tab type of sa measure \cr
\bold{sample.size}       \tab \code{integer}   \tab reported sample size \cr
\bold{sample.size.stats} \tab \code{integer}   \tab reported sample size based on reported statistics (this reflects excluded participants) \cr
\bold{es.z}              \tab \code{numeric}   \tab z-transformed correlation coefficient; includes ghost results (disclosed and undisclosed non-significant effects not reported in detail) imputed using the draw method described in bakdash et al. (2021a) \cr
\bold{vi.z}              \tab \code{numeric}   \tab variance for z-transformed correlation (calculated using \code{sample.size.stats}, \emph{not} \code{sample.size}) \cr
\bold{sampleid}          \tab \code{character} \tab unique identifier for each experiment/study \cr
\bold{outcome}           \tab \code{integer}   \tab unique value for each effect size
the dataset contains behavioral experiments from 77 papers/79 studies with a total of 678 effects, evaluating associations among measures of situation awareness (\dquote{knowing what is going on}) and task performance. examples of situation awareness include knowledge of current vehicle speed in a simulated driving task and location and heading of aircraft in a simulated air traffic control task. corresponding examples of task performance include \dquote{the number of collisions in a simulated driving task} and \dquote{subject matter expert rating of conflict management in a simulated air control task} (bakdash et al. 2021a, p. 2). this dataset and the \squote{examples} are a highly simplified version of the data and code in bakdash et al. (2021b; 2021c). the journal article by bakdash et al. (2021a) describes the systematic review and meta-analysis in detail.
this dataset is used to illustrate multilevel multivariate meta-analytic models for the overall pooled effect and pooled effects by situation awareness measure. we also adjust meta-analytic models using cluster-robust variance estimation / cluster-robust inference with the \code{\link[metafor]{robust}} function in \emph{metafor}. results are shown graphically in a customized forest plot with a prediction interval (estimated plausible range of individual effects). last, we create a table summarizing the estimated meta-analytic heterogeneity parameters.
the meta-analytic results show most pooled effect sizes in the positive medium range or less. there was also substantial meta-analytic heterogeneity (estimated systematic variance in true effects), nearing the magnitude of the overall pooled effect. we interpret the meta-analytic results as situation awareness typically having limited validity for task performance (i.e., good situation awareness does not tend to have strong probabilistic links with good performance and vice-versa). more formally, measures of situation awareness do not generally and meaningfully capture cognitive processes and other relevant factors underlying task performance.
the code run-time can be greatly sped-up using a linear algebra library with \emph{r} that makes use of multiple cpu cores. see: \url{https://www.metafor-project.org/doku.php/tips:speeding_up_model_fitting}. to measure the run-time, uncomment these three lines: \code{start.time <- sys.time()}, \code{end.time <- sys.time()}, and \code{end.time - start.time}. run-times on windows 10 x64 with the intel math kernel library are:
\tab \emph{cpu} \tab \emph{run-time (minutes)} \cr
bakdash, j. z., marusich, l. r., cox, k. r., geuss, m. n., zaroukian, e. g., & morris, k. m. (2021b). the validity of situation awareness for performance: a meta-analysis (code ocean capsule). \verb{https://doi.org/10.24433/co.1682542.v4}
bakdash, j. z., marusich, l. r., cox, k. r., geuss, m. n., zaroukian, e. g., & morris, k. m. (2021c). the validity of situation awareness for performance: a meta-analysis (systematic review, data, and code). \verb{https://doi.org/10.17605/osf.io/4k7zv}
bakdash, j. z., marusich, l. r., cox, k. r., geuss, m. n., zaroukian, e. g., & morris, k. m. (2021a). the validity of situation awareness for performance: a meta-analysis. \emph{theoretical issues in ergonomics science}, 1--24. \verb{https://doi.org/10.1080/1463922x.2021.1921310}
jonathan bakdash, \email{jonathan.z.bakdash.civ@army.mil}, \email{jbakdash@gmail.com} \cr
### multilevel meta-analytic model to get the overall pooled effect
res.overall <- rma.mv(es.z, vi.z, mods = ~ 1,
### cluster-robust variance estimation (crve) / cluster-robust inference
res.overall.crve <- robust(res.overall, cluster = sampleid)
### multilevel meta-analytic model for sa measures
res.sa <-  rma.mv(es.z, vi.z, mods = ~ 0 + sa.measure.type,
res.sa.crve <- robust(res.sa, cluster = sampleid)
### format and combine output of meta-analytic models for the forest plot
all.z        <- c(res.sa.crve$beta,            # sa measures
res.overall.crve$beta,       # pooled effect for confidence interval (ci)
res.overall.crve$beta)       # pooled effect for prediction interval (pi)
all.ci.lower <- c(res.sa.crve$ci.lb,           # sa measures
res.overall.crve.pred$ci.lb, # pooled effect, lower ci
res.overall.crve.pred$pi.lb) # pooled effect, lower pi
all.ci.upper <- c(res.sa.crve$ci.ub,           # sa measures
res.overall.crve.pred$ci.ub, # pooled effect, upper ci
res.overall.crve.pred$pi.ub) # pooled effect, upper pi
### note: there is no p-value for the pi
all.labels <- c(sort(unique(dat$sa.measure.type)), "overall", "95\% prediction interval")
### function to round p-values for the forest plot
plot.vals <- data.frame(all.labels, all.z, all.ci.lower, all.ci.upper)
transf = transf.ztor, # transform z to r
ilab = c(all.pvals.rounded, ""), ilab.xpos = 2.45, ilab.pos = 2.5,
digits = 2, refline = 0, annotate = false, header = false)
output <- cbind(sprintf("\%.2f", round(transf.ztor(plot.vals$all.z), 2)),
text( 2.30,  14.00, substitute(paste(italic('p-value'))), cex = 1.05)
addpoly(all.z[11], ci.lb = all.ci.lower[11], ci.ub = all.ci.upper[11],
rows = 2, annotate = false, efac = 1.5, transf = transf.ztor)
addpoly(all.z[12], ci.lb = all.ci.lower[12], ci.ub = all.ci.upper[12],
rows = 1, col = "white", border = "black",
annotate = false, efac = 1.5, transf = transf.ztor)
par(mfrow=c(1,1), cex = 1) # reset graph parameters to default
### confidence intervals for the variance components
### fit model using alternative multivariate parameterization
res.overall.alt <- rma.mv(es.z, vi.z, mods = ~ 1,
random = ~ factor(outcome) | factor(sampleid),
### confidence intervals for the total amount of heterogeneity variance component
### i^2 (variance due to heterogeneity): 61\%
### 95\% ci for i^2 using uncertainty around tau^2
i2.ci.lb <- 100 * res.overall.alt.tau[1,2] /
i2.ci.ub <- 100 * res.overall.alt.tau[1,3] /
table.heterogeneity <- data.frame(matrix(ncol = 3, nrow = 4))
table.heterogeneity[4,] <- c(i2, i2.ci.lb, i2.ci.ub)
psychology, human factors, engineering, correlation coefficients, multilevel models, multivariate models, cluster-robust inference