ORCID

https://orcid.org/0000-0002-6127-4836

Date of Award

Summer 2025

Language

English

Embargo Period

8-5-2025

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

College/School/Department

Department of Educational and Counseling Psychology

Program

Educational Psychology

First Advisor

Mariola Moeyaert

Committee Members

Benjamin Solomon, Eunkyeng Baek

Keywords

single-case experimental design; single-subject design; interrupted time series; meta-analysis; individual participant data; multilevel modeling

Subject Categories

Educational Psychology | Quantitative Psychology

Abstract

Two-stage individual participant data (IPD) meta-analysis can be used to synthesize intervention effects from single-case experimental design (SCED) studies with multilevel modeling. However, various challenges exist at both stages, including standardizing the intervention effect size, accounting for autocorrelation, and estimating the between-study and between-case variances. While both restricted maximum likelihood estimation (REML) and Bayesian estimation can be used to estimate model parameters, REML has been more commonly applied despite the potential advantages offered by Bayesian methods. Moreover, no methodological studies have focused on Bayesian estimation in the context of meta-analysis of SCED studies, let alone when IPD meta-analysis (often recommended for SCED studies) is applied. To fill this gap, this dissertation examined the performance of Bayesian estimation for IPD meta-analysis of SCED data using the two-stage approach, in comparison with REML estimation. In particular, the dissertation includes two large-scale Current Monte Carlo simulation studies aimed at examining and comparing various statistical properties, including relative bias, mean squared error, relative standard error, coverage proportion of the 95% confidence/credible interval (CP95), statistical power, and Type Ⅰ error rate, for estimating key parameters at Stage 1 and/or Stage 2. Specifically, in Study 1, the statistical properties were tested for estimating (1) the standardization factors (i.e., white noise standard deviation or residual standard deviation), (2) the autocorrelation parameter, and (3) the case-specific standardized intervention effect size. Those estimates are completed at Stage 1 of the two-stage IPD meta-analysis, which often serves as a preprocessing step before synthesizing case-specific effect sizes at Stage 2 for an overall intervention effect. The major aim of Study 1 is to compare the performance of Bayesian and REML estimation at Stage 1 and to identify promising conditions where unbiased estimates of the case-specific standardized intervention effect and its corresponding standard errors can be obtained. In Study 2, the same statistical properties examined in Study 1 were tested for estimating (1) the overall intervention effect, (2) the between-study variance of the intervention effect, and (3) the between-case variance of the intervention effect in the context of the full two-stage analysis. The results of Study 1 indicate that using white noise standard deviation is preferable compared to residual standard deviation when autocorrelation is present, regardless of the estimation procedure. In terms of estimating autocorrelation and the standardized intervention effect, neither Bayesian nor REML estimation consistently outperformed across all examined statistical properties. Additionally, neither estimation method consistently demonstrated acceptable performance across all statistical properties. However, an important finding for estimation at Stage 1 is that when at least 20 measurement occasions are included for each case and when the autocorrelations are expected to be nonnegative and moderate (between 0 and .5), Bayesian estimation with noninformative priors for autocorrelation is recommended, as it can yield unbiased estimates of both the standardized intervention effect and its standard error. According to the results of Study 2, both Bayesian estimation, regardless of the priors used, and REML estimation tended to result in unbiased overall intervention effect estimates and satisfactory power for estimating the intervention effect across all conditions. Among the other statistical properties examined, the Type Ⅰ error for estimating the intervention effect was always too small compared to .05, regardless of the conditions and the estimation procedure. As for the estimation of between-study and between-case variances, greater challenges were found, particularly in terms of statistical power for estimating between-study variance, as well as CP95 for estimating between-case variance. Overall, Bayesian estimation, especially when a weakly informative prior with more information was applied, produced unbiased or less biased estimates of the variances and yielded acceptable or nearly acceptable CP95 values under a broader range of conditions compared to REML estimation. Among the other statistical properties examined, contradictory patterns were observed in the estimates of standard errors: Bayesian estimates of the standard errors of between-study variance were consistently positively biased, whereas REML estimates of the standard errors of between-case variance were mostly negatively biased. Based on the results, specific recommendations are provided for applied researchers interested in assessing standardized case-specific effect sizes and/or synthesizing those effect sizes using a two-stage IPD meta-analysis with either Bayesian or REML estimation. Limitations and ideas for future research are discussed.

License

Creative Commons Attribution 4.0 International License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Download

Share

COinS