Building “Policy as Social Practice” Into Evaluation

What the paper says

The inspiration for this Special Issue and the editors' notes that follow come from observations the guest editors have made regarding the recent state of evaluation, particularly in the Australian context. The guest editors met in 2020 when the first (Alderman) employed the second (Harris) as junior evaluator. The first (Alderman) has been heavily involved in the Australian evaluation landscape for several decades, holding the positions of Chief Evaluator of the Department of Social Services and President of the Australasian Evaluation Society. The second (Harris) came to evaluation in 2020, following the completion of a PhD in international politics. Many of our conversations have highlighted the gap in recent evaluation literature regarding how and why evaluators use relatively well-known and straightforward methodologies. While this is one issue, another is the recent developments in the Australian evaluation context, which simultaneously are indicating a governmental preference for randomized controlled trials (RCTs) but providing less funding for evaluation services overall. As such, this Special Issue draws attention back to well-established but under-discussed methodologies that provide meaningful contributions in funding environments that cannot support experiment designs. The second, third, and fourth articles address this gap by providing renewed focus on benchmarking, environmental scanning, and rapid reconnaissance. An additional theme comes from the experience the first guest editor (Alderman) had during her tenure in the Department of Social Services. As the Chief Evaluator, the remit to establish external evaluation services to all Commonwealth entities was co-dependent on influencing sound policy positions to address social disadvantage for Australian citizens on social benefits. Therefore, the role of Chief Evaluator was required to be across both program evaluation (tactical domain) and the policy level (strategic domain). As such, requests for the Chief Evaluator to move into the policy design and redesign space continued to increase over time. We address how the evaluation methodologies discussed in this issue can be stretched into the strategic domain, as the requests to operate in this environment appear likely to increase. In the initial concept of this edition, we soon realized the dichotomy between cost-effective methodologies and solving complex policy issues. This prompted the inclusion of the fifth article on strategic environmental assessment (SEA), a strategic methodology used for policy development (and ongoing monitoring and evaluation), borrowed from the field of environmental science. Ironically, this may be a methodology many Australian governments are calling for, but do not know how to ask for, nor have the funding to support. Australian evaluation has a shorter history than its United States and European counterparts. Hence, these debates and discussions may feel like history repeating itself to some. For example, the Australian Evaluation Society was established in 1982, decades after its global counterparts. More recently, the landscape of evaluation in Australia has been changing, and there are both weak signals for change in the environment to suggest that this change is inevitable and warranted, while at the same time, there are also strong signals for stability in the evaluation landscape. This section outlines three strong indicators for change in Australia. The first established a legislative position to require evaluation practices at the Commonwealth level. The second promoted evaluation on an international plane that reinforced the Commonwealth position. Finally, third, is the support of innovative pilot studies as good practice to establish new programs, service providers, and industry partners. In Australia, a significant driver for change was when the Australian Government's Department of Finance launched the Public Governance, Performance and Accountability Act 2013 (PGPA; Australian Government 2013). This act addresses the governance, performance, and accountability of Commonwealth entities. It further addresses (a) the use and management of public resources by the Commonwealth and Commonwealth entities and (b) the accountability of Commonwealth companies (see Federal Register of Legislation—Public Governance, Performance and Accountability Act 2013). This Act brought about significant change for the Australian Commonwealth entities, whereby a financial portfolio detailing the expenditure of government funds was to be accompanied by a non-financial portfolio, which was to document the outcomes against the objectives for each department or entity. Essentially, this required some kind of qualitative assessment of government expenditure to accompany the already existing financial analysis. The Act clearly acknowledged that blunt financial accounting practices were incapable of capturing nuance in government spending. In 2015, this Act was further developed with the Enhanced Commonwealth Evaluation Framework (See Morton and Cook 2018 for a detailed description of the design and development of the PGPA Act.) At the same time, the United Nations designated 2015 as the Year of Evaluation. This created a wealth of opportunities and events to promote evaluation, evaluative thinking, and legitimized evaluation as business as usual within Australian government services. This international event allowed for a plethora of different ways in which to build evaluation capacity in commissioners, emerging evaluators, and key stakeholders such as funders. This international event led to all evaluation societies working closely together to promote evaluation, support emerging evaluators, and build capacity in all levels of government to support and fund evaluative thinking and practice. An Australian outcome of this event was the Try Test and Learn Fund, with a budget of AUD$96.1 million, with 52 projects over a 4-year period (2017–2021; Australian Government Department of Social Security 2024). One of the 52 pilots (Kezar 2000) was the Train and Care had a budget of AUD$1.2 million with successful outcomes for 100 participants (Australian Department of Social Services 2024). With an evaluation costing of AUD$22,500, this equates to 1.9% of the project budget. In 2022, the evaluation found that this pilot achieved a 100% return on investment within six months of the participants being off unemployment benefits, with a positive savings to the commonwealth within 12 months. This fund is an example of the Australian Department of Social Services' commitment to adopting pilots as a sound research method to test to see whether a program has value and delivers on the objectives to address areas of social need. Individually and together, the PGPA Act, United Nations Year of Evaluation, and the Try, Test, and Learn Fund continue to reinforce evaluation as an inherent good practice at a Commonwealth level and through to program delivery to Australian citizens. Despite the above strong drivers for change in the Australian Evaluation landscape, there are also weak signals on the horizon that may indicate a future shift in evaluative thinking and practice. While not clearly documented anywhere in legislation or policy documents, the guest editors have noted a clear shift in how funding for evaluation is distributed. As articulated below, most programs are the implementation of broader policy initiatives, and they are typically implemented by non-government bodies that compete for finite government funding. Similarly, the evaluation of these individual programs has historically gone through a public tendering process, whereby the government calls for quotations. However, there has been a noticeable decrease in the number of evaluation services going through the government tendering process and an increase in the number of evaluations being commissioned by service providers directly. This has undoubtedly saved the government money, but has seen a drastic reduction in money set aside for evaluation services (Australian Evaluation Society 2023). Moreover, it has introduced a new player into the evaluation landscape, who will appear throughout this Special Issue, the uninformed commissioner. This is not meant to denigrate individuals who run these programs but rather indicate their expertise in program delivery and not evaluation. With the delegation of evaluation to the service providers, the guest editors have seen a reduction in funding (albeit noting the government was arguably being overcharged at times) and commissioners who rarely understand what they require and how to ask for it. This environment requires rigorous, but simple and cost-effective methodologies. It is this phenomenon that has driven much of the guest editors' practice over the past 5 years and is why many of the methodologies making up this edition have been selected. A more recent and curious development was the announcement of the Australian Centre for Evaluation in 2023 (see the Australian Federal Government Minister's Treasury Portfolio 2000 website for media announcement titled “Australian Centre for Evaluation to measure what works”). This was initially viewed with excitement and interest by the guest editors and other evaluators within Australia. However, as more media releases and information came to light, excitement turned to skepticism. The Center has a small footprint, limited funding has been directed to it, and the staffing profile is low (see the Australian Center for Evaluation web page titled “Our team” at evaluation.treasury.gov.au). It appears the Center will only conduct evaluations of select high-priority or impact activities. Primarily, it will act as a capacity-building unit for other governmental and private evaluators. Controversially, the Center has indicated a strong preference for RCTs and other experimental methodologies (see the Australian Government, Treasury Department web page titled “About the Australian Centre for Evaluation” at treasury.gov.au). This has been jarring for many Australian evaluators, particularly in the social services context, given the historical preference for qualitative or mixed methods approaches. Situated in the Australian Treasury department, the Assistant Minister for the Treasurer Department, Andrew Leigh, addressed the Australian Evaluation Society at its international conferences in 2023 and 2024, regarding the Center. At the 2023 conference, Leigh presented an example from medicine, specifically radical mastectomies in the context of breast cancer care. Leigh explained RCTs in this field over the past several decades demonstrated how more conservative surgical or non-surgical approaches led to better outcomes for patients. While this development is positive, the example from medicine failed to resonate with an audience overwhelmingly comprised of social program evaluators, leading to confusion regarding whether and how the new Center would expect RCTs in all contexts. This recent development in Australian evaluation has some striking similarities with the experience in the United States of America around 20 years ago. In 2001, the Bush Administration passed the No Child Left Behind Act 2001; with its objective to increase accountability and improve outcomes for students, particularly those with disadvantages. Grover Whitehurst was appointed the head of the Institute of Education Sciences, the research and evaluation arm that would oversee this. The Institute had a strong preference for RCTs, with Whitehurst publicly stating that “it privileges, or gives preference to, randomized trials because randomized trials are the gold standard for determining effectiveness” (THE Journal Technological Horizons in Education 2004). He would then go on to cite an example from pharmaceuticals, again another medical analogy. An earlier edition of this journal was critical of this approach, suggesting that a methodology-first approach to evaluation is fundamentally flawed (Berry and Eddy 2008; Mabry 2008). This is a common thought in evaluation practice, yet it tends to get lost in the allure of RCTs (Julnes and Rog 2007; Rog 2012; Harris et al. 2025). In 2024, Leigh returned and joined a panel of Australasian evaluations experts, where RCTs were again discussed as being best practice in evaluation. This led to a spirited debate among the panel about various methodologies, with various individuals arguing that some were better than others. What struck the guest editors was that while the debate about the best methodologies ensued, there was little discussion of context and absolutely no discussion regarding reasoning. The danger in methodological debates is that we risk losing sight of more important considerations like context and reasoning. The circumstances in which an evaluation takes place, what it hopes to discover, and the most overlooked, the logic for discovery, are significantly more important discussions. The following article in this special issue by Harris and Alderman focuses on the key approaches to reasoning in the evaluation context and why evaluators should be significantly more concerned with this debate, rather than the age-old methodological debate of RCTs versus the rest. In June 2024, the Paul Ramsay Foundation launched a grant round for experimental evaluation funding of overall AUD$2.1 million, and the expectation of granting seven evaluations at AUD$300,000 each. It will be interesting to observe how this grant round unfolds in terms of assessing how many experimental programs are actually funded to test hypotheses within a social policy context. The scope and scale of the experiential programs and the experimental evaluation design will provide some insights as to the current state of experimental program designs in Australia today. This is considered a weak signal as the outcomes are yet to be realized and released in the public domain, therefore, it has the potential to influence change. In contrast to the conversation about experimental program designs above, the Victorian Federal Government in August 2024 announced an AUD$6.3 million grant for the Local Government Learn and Earn Pilot Program led by RMIT University, together with a consortium of several universities and vocational education centers. The guest editors were approached by RMIT University to design an evaluation for this project, where the commissioner had set an evaluation costing of AUD$150,000, which equates to 2.4% of the program budget (Victorian Government 2024). This is an example of where the commissioner, the Victorian Federal Government, has delegated authority for an independent evaluation of the program to the service provider. Although it is an example of evaluation being funded for a program of works funded by the state government, it clearly indicates that evaluation is an element (albeit small) of the overall budget. This constitutes a weak signal for change when you consider that the Paul Ramsay Foundation considers AUD$30,000 a modest amount for an experimental evaluation this government budget indicates a significantly amount is are these or can they be there be a between these different approaches that evaluative thinking and evaluators to move between and reasoning on the context of given a an evaluation research evaluators use the “it when with commissioners about evaluation about the evaluation an will on Evaluation is and on a set of the the program the the evaluation the conduct evaluation research to research For is the we are to the implemented as the the objectives of the there of the program have a of methodologies that may be given the at what when evaluators are into the policy they the usual evaluation methodologies, or is there a to our evaluative or thinking into strategic This to the another of about between the of and we about it from a social services policy design to address social is the of disadvantage to be is the legislative are required to support a positive outcome for this or a policy with programs to in social disadvantage be to have a positive A strategic requires evaluators to their evaluative thinking from the to the strategic The first guest editor (Alderman) the position of Chief Evaluator for the Australian Department of Social Services within the and this was the As the Chief Evaluator, the remit to establish external evaluation services to all Commonwealth entities was co-dependent on influencing sound policy positions to address social disadvantage for Australian citizens on social benefits. Therefore, the role of Chief Evaluator was required to be across both the and the strategic and the requests for the Chief Evaluator to move into the policy design space continued to increase over time. This evaluators and evaluative thinking to move across the policy to policy design and implementation through to evaluation as policy implementation The issue in the Australian Department of Social Services was the of across program to the where some were programs where the and objectives of the program were through the of However, between the and strategic up the issue about which methodology is most in each As we all evaluators are methodological who move across operate in and in the evaluation of programs and projects to and In the strategic domain, governments use and as to social change. to of the strategic and found within government the Chief Evaluator found an of as social that clearly the and between and et al. 2007; and in policy and evaluation commissioners to their positions and an of how which in In below, this was from an from a project which to a to the and noting that is in in 5 in this special issue and Alderman 2025). This the strategic as policy and influencing for social change. In the then influence programs, and The had indicating influence was one from to (see as presented in one by Alderman and Harris of this special However, the guest editors that the influence of and strategic are clearly articulated at the level and and at the level and This evaluators with a of a position and then to change and through a strategic The following example how the Australian Government's from a which was by and development was driven by support at the together with state and and from individual The as a from a into the of (Australian Government This was who while in an The and can to In the Australian announced that would to the that a be appointed to into to As a of the from this in 2018 were within legislation to establish the The Australian Department of Social Services was delegated for the development of a to the which each state and to up to the and by the were then required to up to the As many of the were the Australian had the delegated to an should they to up to the The Australian Department of Social Services established a project to the were with a and straightforward process for The legislation was that evaluation was no were to be as their was already documented within the and such Therefore, process evaluation was as an evaluation The has set the for this Special Issue of for Evaluation. the Australian evaluation landscape there are strong drivers for change in the PGPA Act, United Nations Year of Evaluation and the Australian Department of Social Try, Test, and Learn Fund that evaluation as practice. In more recent there are also weak signals for change such as the of the Australian Centre for Evaluation, funding opportunities for experimental evaluation design and continued low for evaluation that provide the evaluation potential opportunities for change. However, the at the Australian Evaluation Society conferences brought to the the guest about what constitutes evaluation best there a debate between RCTs and other evaluation should the debate be is the best evaluation methodology for the context in which a program is The guest editors the context should earlier at the approach to reasoning. The context and current of the program to be will whether or reasoning is the most This in will to an evaluation Therefore, the guest editors have the Special Issue in the following to support the of experimental and program and evaluation designs. In of this special issue Alderman and Harris ongoing debates within research and approaches and approaches to reasoning. debates are to evaluation practice with Alderman and Harris arguing for the for evaluators and commissioners of evaluations to have a clear of approaches and reasoning. than a on these the the of evaluators how their approach or and approach to reasoning or impact their evaluation and methodological Alderman and Harris that evaluators be of these and have an of their evaluation context to adopting a and reasoning The a evaluators through these to methodological than one approach as the evaluators to regarding their evaluation In Alderman and best practice as a evaluation methodology that has potential to support in of and to support The position within a reasoning as a methodology evaluators to such as and within their context. The discussions of into the policy as a key influence on strategic The on their evaluation to the of at both and strategic levels the of and In Harris and environmental as an evaluation methodology to and external influencing the and future of an The provide an of environmental as a methodology for information This methodology with a of the environment their programs and operate a the article how of and strategic by an evaluation commissioners, and Harris and further the of adopting environmental as a rigorous, yet methodology that can be into an practice to program and policy outcomes for their In and Alderman rapid as a evaluation methodology in and research in the is as an initial three key expertise to and rapid assessment In this and Alderman the broader of rapid its evaluation context, its use in and education on their experience within the education the its in its value as a and evaluation In and Alderman strategic environmental assessment as a and methodology from environmental and to of programs, and over the to 100 the as a methodology providing evaluators with a for and at both strategic and levels of an The that by and into evaluation, can their to across both strategic and The article an of methodologies and how evaluators can to their practice and In the of this special issue, Harris and together key from across the and methodological The on the special of established evaluation methodologies and their in complex and While the of in evaluation practice, Harris and for environmental scanning, rapid and strategic environmental assessment strategic and evaluation these methodological approaches evaluators to their practice and to the limited attention these methodologies have in the evaluation the that by and methodological evaluators and their commissioners can evaluation outcomes that to by University of as of the University of the of Australian University