Assessing different implementation modalities of an EdTech intervention for lower‐primary learners: A difference‐in‐difference study in Sierra Leone

Abstract

This paper presents findings from a quasi‐experimental difference‐in‐difference study on three different implementation modalities of the same EdTech intervention, implemented in Pujehun district in Sierra Leone. It offers a critical discussion of the impact of two different personalised modalities and one non‐personalised modality on literacy and numeracy of early primary learners. The results show a positive impact on both numeracy and literacy in different modalities, while demanding a more nuanced evaluation of what aspects of the personalised features of the intervention contribute to such an impact and calling for more consideration of the contextual factors during implementation. Based on evidence gathered while working closely with the implementation partner, the research provides insights into the advantages and challenges of implementation research. Practitioner notes What is already known about this topic Low‐ and Middle‐Income Countries (LMICs), such as Sierra Leone, face challenges in improving Foundational Literacy and Numeracy (FLN) that are exacerbated by other challenges related to educational access, poor infrastructure, teacher availability and other socio‐economic barriers. LMICs also face acute challenges when implementing Educational Technology (EdTech), often stemming from cost and access issues. Further, the implementation of EdTech is often poorly contextualised or done from a top‐down orientation, emphasising outcomes over process and excluding local communities in decision‐making and implementation. Statistically significant moderate positive effects have been observed on learning outcomes of Digital Personalised Learning (DPL) interventions in low‐ and middle‐income contexts, though questions remain in regard to impacts on equity, implementation and the role of teachers, among others. What this paper adds When working closely with local implementing partners, both DPL interventions and non‐DPL interventions have positive impacts on learning outcomes in numeracy and literacy, suggesting equalising effects on marginalised learners observed from personalised interventions, specifically related to girls and lower‐performing learners. That collaborative, participatory‐driven, non‐extractive means of implementation research can provide a reflexive way of understanding EdTech implementation challenges and better contextualise interpretations of the observed impacts of EdTech interventions. DPL interventions may not be as cost‐effective when compared to other personalised interventions, though this needs to be viewed in mediation with policy and scaling objectives. Implications for practice and/or policy Implementation research should integrate close and non‐exploitative partnerships and collaborations with local community members over the entire implementation process in LMICs to better address ongoing challenges and form holistic understandings of education interventions. When implemented in ways that address unique local challenges and implementation needs, both DPL and non‐DPL interventions have the potential to improve FLN outcomes and DPL may have potential for bridging learning gaps for different groups of learners, though cost‐effectiveness needs to be carefully assessed in terms of goals and their time frames.