Acknowledgements
Warranted by Andrew, this work is built on the wisdom of many at ARTD including Chris Milne (who taught me the Outcomes Hierarchy approach to program logic) and Michael Brooks (who coined the phrase 'Outcomes Assurance'). It is built on deep conversations with numerous colleagues over the years and other people whose work I have been fortunate enough to read. Above all, thank you to the countless clients that have given me the opportunity to reflect deeply on what our profession is teaching about the promises of evaluation for better public policy.
Andrew Hawkins
Partner, ARTD Consultants
Andrew started work as a full-time evaluator at ARTD Consultants in 2007 in Sydney Australia. He has a background in Psychology, Law and Public Policy.
Having conducted and led hundreds of evaluations across the public policy spectrum he found the academic discourse and injunction about how to do evaluation was leading to too many inconclusive results and reports that ended with phrases such as 'more time and data is needed to measure long term outcomes'. Given we don't have more time, and the world will always be an uncertain place he started to look for ways of generating better answers.
The jumping-off point was an inability to find the 'logic' in program logic and the 'theory' in program theory. Seeing that logic and theory were often used interchangeably in evaluation seemed an afront to these grand old dames of western thought. The problem eventually became clearer; it was not a lack of theory or data, but clear thinking about why a proposed course of action was a good idea that was holding back credible and useful evaluation of public policy.Â
Experimental design, Realist Evaluation and systems thinking had great impacts on his ideas as did the ideas of Nancy Cartwright on evidence-based policy. But he found that most evaluation for causal inference was still operating in the positivist shadow of the 19th-century ideas of Auguste Comte and the pursuit of a stable and enduring 'Truth' about policies and programs.
'Evaluation' did not begin with the 19th-century origins of the welfare state or the 20th-century reforms of the 'Great Society'. It spans back as far as people have been discussing the merits of a proposed course of action. Reaching back further into the history of ideas in the western intellectual tradition he found a divergence between Plato and his focus on abstract ideal forms and Aristotle's focus on what is reasonable in the apparent world. The debates between the philosophers, sophists, and rhetoricians about the differences between the pursuit of truth compared to what is reasonable to conclude - as well as those between idealism and empiricism, underpin much of the history of western thought. With the triumph of science in the age of enlightenment, anything other than truth was not considered worthy of debate. Armed with new ideas about complex adaptive systems and the limits to certainty, and with great respect for the continuing mysteries of causality, he found inspiration in Stephen Toulmin's work on argumentation and reason as a counterpoint to pre-complexity science and its pursuit of certainty. A program wasn't a theory at all or even 'theory incarnate'.
The end result of this journey was incredibly simple - a program or any other intervention into a complex system can be understood simply as a plan. And as a plan, it would be logical if it could be rendered into a sound proposition. On this account, a program would be sound if it could be represented as a valid argument with well-grounded or empirically verifiable premises with a conclusion that would follow if the premises held. And if something went wrong it could be determined if there was a failure of logic or a failure of implementation. Propositional (Program) Design Logic was born, and in the process so too was Propositional Evaluation.
Propositional Design Logic was built on the outcomes hierarchy that was taught by Chris Milne, took inspiration from Aristotle's enthymeme, and Stephen's Toulmin's work on informal logic, John Mackie's work on causality to propose a form of deductive program logic with a theory of causality founded on necessary and sufficient conditions for a specific course of action to be efficient and effective.
Outcomes Assurance using Propositional Evaluation provides the process to interrogate Proposital Program Logic and guide the collection and use of information to make better decisions, manage the risk of failure and improve the human condition with humility about what can be known with any degree of certainty.