Evaluations of Literacy-Based Programs: Empirical Values for Designing Studies Probing Mediation

Abstract

Multilevel mediation analyses are vital for testing and refining theories of action in program evaluations. Prior literature and established guidelines have repeatedly underscored the need to prospectively incorporate mediation considerations when designing a study even when mediation is considered a secondary or exploratory aim. Despite the critical role mediation analyses play across disciplines, the available empirical resources to facilitate effective and efficient design of such studies are limited. This study contributes to this gap by developing empirical estimates of design parameter values for literacy programs implemented within classrooms by teachers. Using data from two large studies with thousands of students and hundreds of teachers and schools, we develop empirical guidelines for studies using a three-level structure (students, teachers, and schools) for a diverse set of school contexts (e.g., urban and geographic regions) and mediators (e.g., teacher knowledge, instruction, affect, attitudes, and classroom climate). The results suggested considerable variation in the clustering of teacher mediators and student outcomes within schools and that such clustering is dependent on the variable type (e.g., instruction, knowledge, climate, and literacy achievement) and study context. The findings underscore the importance of context-, mediator-, and outcome-specific design parameter values in planning studies.