The idea of basing policies on the best available evidence has spread from medicine to several policy fields in the twentieth century (Baron,
2018; Head,
2015). Today, evidence is an important resource for governments in most modern democracies to identify effective solutions to policy problems (Adam et al.,
2018,
2019; Davies et al.,
1999; Parkhurst,
2017; Parsons,
2004; Sanderson,
2002). Policy scholars have examined the mobilization of evidence by research agencies (Powell et al.,
2018), the capacities of governments to deliver policy analysis and advice (Howlett,
2009; Migone & Howlett,
2022), and the use of evidence in various decision-making bodies, including government agencies (Landry et al.,
2003), state agencies (Jennings & Hall,
2012), and legislatures (Geddes,
2021). Yet, many studies find that research often fails to influence policy decisions and that research utilization varies across policy areas and contexts (Boswell,
2009; Landry et al.,
2003; Lester,
1993; Lindblom & Cohen,
1979; Oliver et al.,
2014; Weiss et al.,
2008). Low levels of research utilization have been attributed to the shifting preferences of policy-makers (Majone,
1989), value conflicts over policy aims (Boswell,
2009), and technical barriers to the use of evidence, including poor timing between research and policy, lack of robust impact evaluations, and a poor fit between evidence-based findings and policy priorities (Cairney,
2016; Capano & Malandrino,
2022; Nutley et al.,
2007; Weiss,
1995). The continuous importance of political and practical concerns has led some scholars to abandon the notion of “evidence-based policy” in favor of the more modest phrase “evidence-informed policy” (Bundi & Pattyn,
2022; Head,
2015; Nutley et al.,
2007). Nevertheless, the extent to which evidence is used in public administrations remains an empirical question that is affected both by the level of policy analytical capacity and conflict in different policy subsystems (Howlett,
2009; Jennings & Hall,
2012).
The article adds to theory and research by comparing how and why evidence standards affect research utilization in two ministries with available evidence, similar policy analytical capacities and broad political agreement on key policy aims. The article relies on a most-similar case study design and a combination of content analysis of documents (
N = 1,159) and interviews with civil servants and relevant external stakeholders (
N = 13) to compare and explain research utilization in the Ministry of Employment and the Ministry of Children and Education from 2016 to 2021. By addressing the effect of evidence standards on research utilization in policy decisions, the article advances the study of research utilization and evidence-based policy as a limited number of contributions have measured levels of research utilization over time or across different public administrations (cf. however, Jennings & Hall,
2012; Landry et al.,
2003; Newman et al.,
2016).
The analyses capture both smaller and larger policy decisions, such as the 2014 Public School Reform, which has had a significant impact on the Danish education system, and a series of labor market reforms adopted in the period under investigation. While both ministries have invested in building policy analytical capacity to accumulate and consider evidence in the policy process, the content analysis finds higher average levels of research utilization and higher utilization of studies from the top of the evidence hierarchy in the Ministry of Employment than in the Ministry of Children and Education. Interview respondents attribute this variation in research utilization to internal efforts in the Ministry of Employment to adopt specific evidence standards and its investment in creating a knowledge bank for rating policy effects based on evidence. In addition, the Ministry of Employment has had a continuous dialogue and coordination with the Ministry of Finance to agree on principles for utilizing evidence in budget forecasting to estimate future economic gains with general support from stakeholders in the policy subsystem. By contrast, the Ministry of Children and Education has had a more conflicting relationship with stakeholders about letting evidence shape policy reforms. Even though the central ministry department has recently invested resources in a knowledge bank, there is not yet sufficient evidentiary basis to persuade the Ministry of Finance to include derived economic effects of public school policies in economic modeling, thus contributing to lower utilization levels.
Evidence-based policy and the politicization of evidence
The so-called “evidence movement” and its emphasis on rational decision-making has been an important driver of evidence-based policy, promoting that research is used to find policy solutions that have been documented to “work” across policy domains (Baron,
2018; Bundi & Pattyn,
2022; Head,
2015; Howlett,
2009; Parsons,
2004; Sanderson,
2002). Such “what works” evidence is typically produced following an evidence hierarchy, which values knowledge based on the ability of research designs to systematically eliminate bias when determining the causal effects of policies (Evans,
2003; Nutley et al.,
2013; Petticrew & Roberts,
2003). The idea of using evidence to foster utilization based on policy effects as well as costs is also central to the rational model of decision-making because it can underpin policy goals of enhancing the cost-effectiveness of policies (Davies et al.,
1999; Greany & Brown,
2017; Oliver,
2022). An additional appeal of evidence-based policy decisions taken at central levels of government, is that even small policy changes can have substantial aggregate effects if applied at scale, e.g., to whole populations (Baron,
2018). Evidence can therefore help identify effects that are not immediately visible to policy-makers but have effects on desired policy aims. On this basis, evidence-based policy has moved beyond the idea that politics and science are separate communities with different knowledge ideals and incentives and has spread to more policy domains over time including employment and education (Baron,
2018; Caplan et al.,
1975; Caplan,
1979; Head,
2015).
Existing evidence-based policy studies, however, have identified barriers to research utilization and voiced criticisms of using a rational model of decision-making in public policy (Capano & Malandrino,
2022; Fischer,
2021; Lester,
1993; Newman et al.,
2017). The politicization of evidence and the challenge that policymakers may cherry-pick evidence to suit their preferences have been emphasized as adverse uses of evidence (Weiss,
1979; Oliver,
2022; Parkhurst,
2016,
2017). In this view, evidence works as a reservoir that can be drawn upon for symbolic or strategic reasons when politicians desire to justify decisions (Daviter,
2015; Majone,
1989; Mosley & Gibson,
2017) or to pre-empt public criticism of government policies (Boswell,
2009). Several studies have thus found that research production and utilization are shaped by actor preferences, especially those of political decision-makers (Boswell,
2009; Fobé & Brans,
2013; Sanderson,
2002). “Policy-based evidence” has been suggested as a more appropriate term for evidence that political actors use to underpin predetermined policy positions (Sanderson,
2011). If policy aims are changing or are disputed, the evidence might become a “moving target”, as previously collected evidence can become irrelevant, e.g. with a change of government. Moreover, continued external politicization of evidence can interact with policy contestation and become a barrier to research utilization, as ministries may refrain from focusing their capacity on evidence collection and utilization in light of policy conflict (Jennings & Hall,
2012). Even in cases where there are low levels of conflict, there is a gap in understanding mechanisms that link capacity to research utilization. This article focuses on the effect of policy analytical capacity and evidence standards on research utilization to address this gap.
Explaining the effect of evidence standards on research utilization
Notwithstanding politicization and policy conflict as barriers to evidence-based policy, there are drivers, pressures, and opportunities for public administrations to increase their policy analytical capacity and utilize evidence systematically and over time (Carpenter & Krause,
2015; Christensen,
2022; Howlett,
2009). Building capacity for evidence acquisition and utilization is not necessarily motivated by political ambitions to influence policy decisions directly (Blom-Hansen et al.,
2021; Head,
2015; Parsons,
2004; Simon,
1955); it might also reflect a broad and continuous political agreement on policy objectives and political demand for effective policy solutions to reach such goals. Public administrations might therefore enhance their capacity to base policy decisions on evidence despite perceived barriers to research utilization (Kroll & Moynihan,
2018; Parsons,
2004). Inspired by the promises of using evidence, ministries may invest in their policy analytical capacity, understood as their capacity for: “…knowledge acquisition and utilization in policy processes” (Howlett,
2009, p. 162–63). Policy analytical capacity captures the capacity of public administrations to acquire, manage, communicate, and integrate knowledge into the decision-making stage of the policy process – something that has been emphasized as a key factor for research utilization, while lacking capacity has been associated with failure to design effective long-term policy measures (Howlett,
2009; Jennings & Hall,
2012).
In studying ministries’ policy analytical capacity, we specifically focus on evidence standards, defined as the criteria used to guide decisions for accepting or rejecting available evidence as a basis for making policy decisions (Parkhurst,
2017, 161). We argue that ministries rely on evidence hierarchies to set evidence standards. We propose that ministries set evidence standards based on a consideration of the trade-off between the legitimacy derived from inclusive standards and the expected efficiency gains associated with setting exclusive evidence standards (Adam et al.,
2018; Petticrew & Roberts,
2003). Ministries may set inclusive standards to gain legitimacy by signaling a willingness to include many types of knowledge in decision-making processes. Under inclusive evidence standards, many types of knowledge and organizations are likely to be included for consultation (e.g. in commissions, councils, or other types of meetings) in a broad search for the best available knowledge. As methods at the top of the evidence hierarchy, notably randomized controlled trials (RCTs), can identify average effects of policies with higher levels of certainty than studies without rigorous causal controls, ministries adhering to a rational ideal might choose to adopt exclusive evidence standards to increase research utilization and make policies more efficient, even if this excludes some types of evidence and actors not possessing such evidence from the policy process. By explicitly setting exclusive evidence standards a ministry can signal to stakeholders that it is interested in causal evidence from the top of the evidence hierarchy. In addition, exclusive evidence standards and the accumulation of causal knowledge may signal that the ministry is systematically accumulating causal evidence on policy aims that are prioritized by political decision-makers and that the ministry can credibly underpin decisions with evidence. Thus, ministries can adopt exclusive evidence standards to strengthen a bridging function between evidence supply by stakeholders, in some cases by commissioning research directly, and the demand for evidence by political decision-makers and thus enhance research utilization (Howlett,
2015). We do not suggest, however, that exclusive evidence standards will always increase research utilization. Under conditions of scarce evidence and disagreement on key policy aims, the effect of exclusive evidence standards is unlikely to affect research utilization positively or at all. This is in line with other studies that emphasize conflict and a lack of evidence as barriers to research utilization (cf. Jennings & Hall,
2012). We therefore make the following proposition:
Under conditions of available evidence and agreement on key policy aims, more exclusive evidence standards in ministries will lead to higher levels of research utilization, as they link more studies from the top of the evidence hierarchy to policy decisions.
As elaborated below, we address the proposition by studying how and why evidence standards affect research utilization in the two ministries with available evidence, policy analytical capacity, and broad political agreement on key policy aims. To address the proposition analytically, evidence standards are operationalized by studying whether ministries formally set more exclusive or inclusive criteria for using evidence in organizational documents as well as informally as captured by interviews with civil servants. As ministry practices might vary internally (different ministry divisions may have different practices and ideals for evidence use), we distinguish evidence standards set by central ministerial departments and ministerial agencies to consider organizational variation in evidence standards (Hammond,
1986). We do not consider changes in portfolio design as ministerial portfolios have been constant in the period under study for both policy domains (Sieberer et al.,
2021; Fleischer et al.,
2023).
In the following, the focus is specifically on evidence standards as reflected in systematic, organization-level procedures undertaken by the ministries to facilitate research utilization rather than other aspects of policy analytical capacity, such as professional analytical employment or the individual-level analytical skills of policy workers (Howlett,
2015; Migone & Howlett,
2022). Furthermore, research utilization is defined and examined as the process whereby knowledge influences political decision-making and focuses mainly on the reception and influence stages of this process, as elaborated below (Landry et al.,
2003). We study differences in research utilization in the “reception stage” of research utilization (Landry et al.,
2003, p. 194) both in terms of the research that ministries order and commission and the research that ministries disseminate to the public and political decision-makers. Even if crude, this distinction is important, as government ministries may apply evidence standards differently in terms of their demand and supply of evidence. Our study further aims to capture the “influence stage” of research utilization (Landry et al.,
2003, p. 194) both in absolute terms by comparing the extent to which particular sources of evidence are reflected in subsequent policy decisions and in relative terms by addressing the utilization of evidence at different levels of the evidence hierarchy. Our document analysis does not capture informal discussions or uses of evidence in other phases of decision-making even if they may be relevant factors for research utilization (Knott & Wildavsky,
1980). Below, the article advances the study of evidence-based policy in two main ways. First, it develops and applies a matching method to compare levels of research utilization in policy decisions. This addresses an analytical challenge in the evidence-based policy literature, which has focused on individual events of research utilization, often discovering barriers rather than patterns of utilization over time. Moreover, as utilization is challenging to observe directly, empirical studies have predominantly been based on perceptual data in the form of interviews or surveys with policy-makers (Boaz et al.,
2009; Christensen,
2023; Head et al.,
2014; Jennings & Hall,
2012; Landry et al.,
2003; Newman et al.,
2016; Newson et al.,
2018; Toner et al.,
2014). Second, the combination of content analysis and interviews with civil servants and stakeholders allows the article to examine how ministries use their policy analytical capacity and thus capture mechanisms that link evidence standards and research utilization.
Case selection
In line with international trends, the evidence movement has gained ground in Denmark, and evidence-based policy has become an ambition in many policy subsystems, including health, employment, education, and social policy (Andersen,
2020; Hansen & Rieper,
2009,
2010). Today, most Danish ministries have relatively strong capacities for knowledge acquisition and utilization, and reports have documented increasing tendencies in the ministries to consider research when preparing, enacting, and implementing policies (Arnold et al.,
2015; Rambøll,
2015). Another important factor in the Danish case is that public spending levels are comparatively high. Denmark had the highest expenditure on active labor market policies as a percent of GDP among all OECD countries in 2020 (OECD Employment Database,
2023), while public expenditure on public school policy as a percentage of GDP was substantially above average in the EU27 in 2019 (Eurostat,
2023). The combination of a high capacity and high public spending levels, which incentivizes cost reduction, makes Denmark a likely context for evidence-based policy, as ministries have cost-reduction incentives and capacities to utilize evidence.
We compare evidence standards and research utilization in the Ministry of Children and Education and the Ministry of Employment from 2016 to 2021 focusing on public school policies and active labor market policies (ALMPs). The two policy areas share several background conditions for research utilization: The employment and education sectors have both been subjected to effects evaluations (Coryn et al.,
2011), and evidence accumulation through systematic literature reviews have been undertaken in both sectors (e.g., through the Campbell Cooperation). Echoing the recommendation that each Danish ministry develops a knowledge strategy (DFiR,
2016), both government ministries have capacities for knowledge acquisition and utilization, affiliated research organizations, and internal agencies that work with gathering, interpreting, and applying research and data (the Danish Agency for Labour Market and Recruitment and the National Agency for Education and Quality). Furthermore, there has been a broad agreement on key policy aims in both sectors in recent decades: Due to the spread of international test regimes, student learning and well-being have been consolidated as main goals in public school policy across different governments. Concerning active labor market policy, the same goes for increasing employment through increased labor supply (Andersen,
2020,
2021; Hansen & Rieper,
2010; Ministry of Employment,
2021; STAR,
2017,
2023). On this basis, and with reference to Jennings and Hall’s typology of the expected use of evidence (2012, p. 261), we regard the ministries as most-similar cases, which are expected to function as “evidence-based agencies” with comparable and high levels of research utilization, but possible variation in outcomes due to the evidence standards adopted.