Evaluation and value for money: development of an approach using explicit evaluative reasoning

Abstract

There is increasing scrutiny on social investments to determine whether they deliver value for money (VFM), but current approaches to assessing VFM are incomplete. The disciplines of economics and evaluation share an interest in valuing resource use, but tend to operate as complementary or rival disciplines rather than being integrated within an overarching logic. Cost-benefit analysis (CBA) is often regarded as the gold standard for evaluating VFM, but has recognised limitations. For example, collective values, distributive justice, power dynamics, public dialogue, and qualitative evidence are peripheral to the method. Conversely, program evaluation offers more capacious approaches to determining value but rarely includes costs, let alone reconciling value added with value consumed. This disciplinary divide may diminish capacity for good resource allocation decisions.

The aim of this theory-building research was to develop a model to guide the evaluation of VFM in social policies and programs. A conceptual model was developed through critical analysis of literature, proposing requirements for good evaluation of VFM. Gap analysis was conducted to determine the extent to which CBA can meet the requirements of the conceptual model. Cumulative findings from the first two studies were dissected into a series of theoretical propositions. A process model was developed, identifying a series of steps that should be followed to operationalise the conceptual model. Case studies of real-world VFM evaluations in two international development programs were analysed to assess the conceptual quality of the theoretical propositions.

This research makes seven significant and novel contributions to the field of evaluation. First, VFM is an evaluative question, demanding a judgement based on logical argument and evidence. Second, VFM is a shared domain of two disciplines, because it is concerned with merit, worth and significance (the domain of evaluation) and resource allocation (the domain of economics). Third, CBA is not a rival to evaluation; it is evaluation. It evaluates an important dimension of VFM (aggregate wellbeing) and can strengthen the validity of an evaluation. Fourth, CBA is not the whole evaluation; it is usually insufficient on its own because of limitations in its scope and warrants. Fifth, a stronger approach involves explicit evaluative reasoning, with methods tailored to context including judicious use of economic methods where feasible and appropriate. Sixth, program evaluation standards should guide economic evaluation, and this has implications for the way CBA is used including the nature and extent of stakeholder involvement, the use of CBA in conjunction with other methods, and decisions about when not to use CBA. Seventh, the case studies are themselves a contribution, modelling the use of probative inference to corroborate the propositions of the conceptual model. Ultimately, this thesis provides proof of concept for a practical theory to guide evaluation of VFM in social policies and programs.