C2021 Theme

Evaluation utilization: achieving our potential?

Since the original Michael Quinn Patton’s Utilization-Focused Evaluation in 1978, evaluation utilization has become the ultimate value for many evaluators world-wide: for them, evaluation information has no purpose if it does not get used. But over the last 40 years, the very notion of evaluation utilization has become more complicated and its dynamics, more complex: there is talk of the utilization of individual evaluations, of evaluation as a process, of evaluative thinking, of evaluation insights, etc. We hear evaluation managers craving for their evaluations to be useful and used, where historically this was taken for granted. There is still confusion as to what constitutes a quality evaluation and whether it is defined by its process, its product, its use, or some other criteria. Moreover, evaluation is only one source of information potential users have access to – and not necessarily the most credible or actionable one. Also at issue is who gets to define what utilization evaluations are expected to have and how the needs, the expertise, and the interests of beneficiaries and disenfranchised groups are used to judge an intervention.

The‌ ‌2020‌ ‌CES‌ ‌Conference‌ ‌will‌ ‌explore‌ ‌the‌ ‌world‌ ‌of‌ ‌evaluation‌ ‌utilization‌ ‌through‌ ‌five‌ ‌strands‌ ‌by‌ ‌first‌ ‌positioning‌ ‌it‌ ‌(what‌ ‌is‌ ‌it?‌ ‌is‌ ‌it‌ ‌a‌ ‌priority?),‌ ‌then‌ ‌by‌ ‌achieving‌ ‌it‌ ‌(how‌ ‌is‌ ‌it‌ ‌achieved?‌ ‌how‌ ‌do‌ ‌we‌ ‌know‌ ‌it‌ ‌has‌ ‌been‌ ‌achieved?),‌ ‌and‌ ‌by‌ ‌questioning‌ ‌it‌ ‌(whose‌ ‌utilization?).‌ ‌We‌ ‌will‌ ‌also‌ ‌investigate‌ ‌the‌ ‌impacts‌ ‌of‌ ‌the‌ ‌COVID-19‌ ‌pandemic‌ ‌on‌ ‌utilization‌ ‌and‌ ‌the‌ ‌links‌ ‌between‌ ‌utilization‌ ‌and‌ ‌marginalization.‌ 

Positioning Utilization

This strand investigates the nature of evaluation and evaluation utilization, its various forms and intensities, as well as its measurement. What is evaluation utilization? What different types of uses are there (Kusters identifies instrumental use, conceptual use, process use, symbolic use, relational use, value use, and external use)? Does pursuing some types of utilization prevent from achieving other types? Are some forms of uses more effective than others at supporting the common good? Can evaluation and evaluations be misused; if so, how and why does this occur (Alkin and King are inspirations)? What can be done to reduce the risk of misuse? How far have we come in assessing utilization: how do we observe, describe, and measure use? Does the value placed on utilization negatively affect how evaluations are designed, executed, and reported? Is it possible to conduct a quality evaluation without consideration for use (otherwise stated, is utilization a necessary condition of evaluation quality)?

Impact of the pandemic on utilization

2020 has been scarred by a global pandemic of unusual proportions. All aspects of human life have been affected although these effects have varied according to geography, sociology, power, economics, etc. Non-human systems have been affected as well, sometimes positively, like air quality and water cleanliness. Evaluation practice was not spared. Many thinkers and practitioners have already started to assess the impact of the global pandemic on evaluation practice. This sub-theme will focus on the impacts of the pandemic on the utilization of evaluation. Has the pandemic sparked an interest in evaluative evidence and how? How did the timeliness requirement of pandemic response affect utilization capacity? Do evaluators need to adopt a different approach to the utilization of evaluation findings in a period of crisis? Are the levers of utilization the same?

 

Achieving Utilization

This strand examines the mechanisms at play in achieving evaluation utilization. What are the factors that determine whether or not an evaluation or evaluation information or an evaluation function, is put to maximum use? Are these factors context-dependent: are they the same across organizations, cultures, geography, languages, etc.? Which combinations of factors are more likely to maximize use and in which context? What is the role of evaluation capacity in evaluation use (give a read to Cousins and others)? What lessons can be drawn from the literature and practice on the use of information in decision-making and behavioural change? Does the flexibility given to federal departments and agencies in the 2016 Policy on Results help or hinder subsequent utilization? Has the increased emphasis on examining diversity within evaluation (e.g., SGBA+ analysis) enhanced use? Has an increased focus on performance measurement by programs increased the value of evaluations and their utilization or obscured the real aim of self-criticism?

Questioning Utilization

This strand addresses the subversive aspects of evaluation utilization. Who defines what utilization is sought for a given evaluation or for evaluation practice as a whole? Who determines what gets evaluated and how? Is evaluation use systematically reflective of the interests of those with resources to commission evaluations? Is the valuing of utilization a representation of the power structure in organizations and in society where the powerful may emphasize self-serving use over inquisitive knowledge? What is the responsibility of evaluators themselves in promoting use, if any: are we seditious agents of change or servile information gatherers? How can disenfranchised groups become part of the evaluation entreprise, and contribute to and benefit from evaluation utilization? Whose perspective defines what is legitimate use and questionable misuse are?

Marginalization and utilization

Another important global shift in 2020 was the increased awareness and galvanized actions to address the significant inequities that become more apparent during the pandemic. Whether it was the vulnerability of our elderly, our Indigenous peoples, our impoverished, racialized minorities, or others, we saw a collective call to action to address long standing social and environmental inequities. Along with these are calls for new ways to address these inequities, such as : defunding policing, establishing a basic guaranteed income, re-imagining long-term care, transitioning to renewable energy and circular economics, and investment in Indigenous Protected Areas . What are the implications on the utilization of evaluation? Has the renewed attention to inequities affected the use of evaluation information? How can evaluation find its way into decision making in the context of intense socio-political debate? Can evaluation contribute to reducing inequities in our society today and, if so, how? What changes are needed in evaluators’ frame of mind and / or practices to ensure that the results of our evaluations contribute to social and environmental justice?