Complexability

Pty Ltd
acn 142 164 078 viv.read@complexability.com.au www.complexability.com.au PO Box 91, Lorn NSW 2320

Measuring and Monitoring ALL that matters

A discussion paper.. If a measure becomes a target, then it ceases to be a measure Charles Goodhart. Goodharts Law I believe a practice has developed where the drive to meet numerical targets has overshadowed the interest of children despite evidence that they may be at risk. John Taylor, Acting Victorian Ombudsman 2011 Performance, measurement and accountability are high priorities on any organisations agenda. The unintended consequences of using targets and KPIs as measures in complex systems are becoming well recognised. For example, the 2010 Productivity Commission report into the Not For Profit Sector demonstrates that focus on the use of statistical information and consistent outputs as a way of measuring the impacts and activity of various community services organisations as inadequate. Likewise the 2011 report of the Australian National Audit Office (ANOA), identifies significant flaws in the delivery and evaluation of government services over the last two decades. However an approach that will address the flaws identified has not yet emerged.

What would a new approach look like?

Quantitative and qualitative measures have been regarded as separate processes using very different approaches. Increasingly however, organisations providing services need more than the numbers to demonstrate the value they provide. Numbers obtained this way do not provide answers to the question Why? Based on our own work, and conversations with clients, the desirable characteristics of an effective evaluation framework that we identified were: It would be able need to address the whole system, and identify relationships between the various key elements It would assess the current state and provide insights that gave options for improvement

C o m p l e x a b i l i t y P t y L t d i s t h e r e s e a r c h a r m o f E m e r g i n g O p t i o n s P t y L t d , a n d h a s

It could be readily customised to meet the specific contexts of different organisations, while having sufficiently commonality to enable comparisons It must be cost effective, dynamic and provide feedback quickly It should provide both quantitative and qualitative measures the insights would be the basis for action , improvement and feedback for both service delivery and policy it is scalable appropriate for small and large systems

What do we use?
Most human systems are complex; and simple linear relationships between various elements do not exist. The One size fits all approach to various policies, systems and approaches has been found to be wanting. Likewise, imitating what is seen to work in another country, another organisation of even another part of the same organisation rarely works context is critical. The many management tools and processes that are available to address organisational issues are based on the assumption that there is a definable relationship of cause and effect between identified elements that can inform strategy and interventions. In current complex and turbulent environments, it is increasingly obvious that these approaches do not serve us well when dealing with complex interactions. Traditional evaluation methods, that rely on hypotheses about cause and effect relationships remain appropriate for some parts of a system but not all.

At the heart of the Complexability approach is the Cynefin framework; an open systems model that enables the identification of the different elements of a system that need to be monitored. Understanding the distinction between ordered and unordered elements of a system leads to the application for different methods for diagnosis and intervention depending on the nature of the issue.

The Ordered domains..

Ordered systems are based on the belief that it is just a matter of time and resources before the relationships between cause and effect could be discovered in any situation . The relationship may be obvious (Simple) or require research and analysis (Complicated), but it is able to be identified and mapped. These include Force

Field Analysis, SWOT for strategy. Or, like LEAN, 6 Sigma and Business Process Reengineering, they map a systems inputs, throughputs and outputs as a way of identifying system improvement. Measuring, monitoring and evaluating issues that are in the ordered domains relate to outputs,

standards and targets.

There are a wide range of existing tools and methods designed to determine what numerical information to capture, and how to report it. The quantitative data is usually subject to analysis and interpretation by experts. These approaches are founded on assumptions of order, and cause and effect.

The Unordered Domains

The primary feature of unorder is that we do not know the nature of relationships other than in retrospect. There are far too many elements and possibilities to develop and test hypotheses. Therefore the approach is to make sufficient sense of the current state where multiple interventions ( or probes) can be implemented, monitored and quite quickly either stopped or enhanced depending on what happens. Unordered domains required tools and methods that measure and monitor impact and

outcomes.
Qualitative research has primarily relied on expert led discussions and inter views, such as focus groups. It is significantly more expensive that collecting quantitative data, and the results can be susceptible to researcher bias. Narrative research is now used in a variety of settings, given that narrative allows complex information to be shared. However, most of the processes for analysis still depend on the researchers hypotheses, conceptual frameworks and beliefs.

Complex'unorder'

The Cynen FrameworkPerformance Monitoring

Complicated'Order'
Decision(Making:( Sense/Analyse/Respond( Performance'Monitoring:' Good'Prac7ce'' Indicators'

un,ordered%systems%

Decision(Making:( Probe/Sense/Respond( Performance'Monitoring:' Emergent'prac7ce' (innova7on)' impacts'

ordered%systems%

Chao7c'Unorder'
Decision(Making( Act/Sense/Respond( Performance'Monitoring:' Novel'Prac7ce'

Simple'Order'
Decision(Making: (( Sense(/Categorise/Respond( Performance'Monitoring:' Best'Prac7ce'' Outputs' Quan7ta7ve'Measures'

Sensemaker
Is software that has been developed by Cognitive Edge specific ally for issues that are in the complex domain. It can capture large amounts of narratives quickly and efficiently, and provide participants with the opportunity to attribute their own meaning to their experiences, rather than experts attributing that meaning. The software analytics provide quantitative measures of qualitative date, ( impacts and outcomes) and insights that enable action and intervention should this be required.

The Complexability approach?

The Complexability approach compliments any current monitoring and evaluation methods being used by an organisation, or program. It is specifically concerned with the impacts and outcomes of activities, rather than the outputs and targets. Questions are customised, based on the context of the organisation, and the issues of concern Each project has the common elements of: (a) Understanding the current situation the Discovery Phase This phase is concerned with WHAT needs to be explored, and HOW the data will be captured, analysed and utilised. The WHAT includes the SCOPE and BOUNDARIES of the project, ensuring that there is appropriate focus, and the purpose for undertaking the project is clear and shared. The HOW relates to the options for the way data will be collected, and analysed. This can include both software and non software methodologies. It is concerned with gaining an understanding of the CURRENT STATE, and the relationships (or not) between various elements that seem to contribute to the current situation being the way it is.

This phase will usually encompass questions related to the impact on clients and other stakeholders, the audit and/or evaluation of a program or project to determine the extent to which desired outcomes are being achieved , and the extent to which a program is delivering value (social and economic) for the money invested. Such assessments will involve perspectives from Customers Service delivery staff Policy makers Funders Other stakeholders All will have different, and legitimate, perspectives that need to be taken into account. Data collection and analysis uses Sensemaker software, developed by Cognitive Edge specifically to be a decision support tool for complex issues. By capturing the experiences of many, it provides quantitative analysis on the qualitative data, (more details below).

(b) Sensemaking- analysis and reports This stage is about gaining insight, as support for decisions. Depending on the nature of the project, a range of activities may be undertaken. At a minimum, a report is prepared documenting the results and insights from the analysis undertaken by the software. This report is a combination of visual representations of patterns and trends, in addition to the traditional graphs capturing demographic data, and responses to other general research questions. It may also contain recommendations. Going beyond an audit of a project or initiative, or an assessment of what has been achieved requires people from the system under consideration to interact with data. Where Sensemaker is being used to provide the basis for intervention and improvement, the analysis differs from most other software packages. As a decision support tool, insights and possibilities for further exploration are identified. Significant insights are gained through discussion, and consideration of the narrative information that has led to the patterns and trends identified by the software. There are a number of non software methods designed to aid in this process and provide structure for the deliberations and conclusions, including agreed actions. (c) From insights to action.. and further monitoring This stage focuses on designing intervention and initiatives that take into account insights gained, and the nature of the various issues that need to be addressed. The Cynefin framework is used to ensure that most appropriate actions are taken for each of the issues. For some applications, the performance monitoring requires and assessment of impact over time. Sensemaker can be used to compare data collections at different time frames, for different stakeholder groups.

Applications

Narrative research, using Sensemaker is being used in a wide variety of settings globally. Complexability consultants have been involved in performance monitoring projects related to The value of investment in Executive education Service delivery outcomes in the community care sector Workplace Cultural Assessment Outcomes of a government sponsored foreign aid project Other projects have captured perspectives on issues such as climate change and volunteering, to inform policy initiatives; identifying contributing factors to excellence in vocational education; and community perspectives on regional development plans. Vivienne Read June 2013 viv.read@complexability.com.au