Assessment and learning are the processes of ongoing reflection about visions, strategies and actions that enable continual readjustment. Strategic adjustment and operational improvements are ideally not driven by crisis but by deliberate, information and experience-based reflections. The terms ‘monitoring’ and ‘evaluation’ are more commonly used to refer to such reflection. However they are often associated with specific, donor-defined obligatory systems to prove and be accountable for funding. To emphasise the focus on internal usefulness for improvement, the ASC discussions used the alternative terms ‘assessment’ and ‘learning’. They have the added advantage of being less automatically associated with specific methodological processes. However, caution is needed with the term assessment, which some define as a survey process (see Box 7).
Assessment serves multiple complementary functions2 that require explicit attention to ensure that learning occurs. The most commonly accepted function that shapes many assessment procedures is that of accountability, i.e. demonstrating to diverse audiences that expenditure, actions and results are as agreed or can reasonably be expected. Assessment also supports operational management by providing basic information for coordinating the human, financial and physical resources needed for achieving objectives. A third function is to support strategic management, to facilitate the processes needed to set, question and adjust goals and strategies. A fourth function is that of generating new insights to development, in this case to the understanding of how social change occurs and why. Finally, and often forgotten, is its potential to build the capacity, self reliance and confidence of those involved to undertake development initiatives. Learning is assumed to result from M&E processes that are designed with accountability as the underlying purpose. And it often fails to do so, with a disconnection between learning and assessment in many organisations. Hence learning processes need to be explicitly designed for in assessment processes.
Box 7. When assessment does – and does not – trigger organisational reflection and learning
There is the potential for confusion about how assessment enhances social change processes. For some, assessment is not a reflective exercise but is a monitoring process that is subsequently used to lobby for change. Take the case of Amnesty International, which collects data on human rights abuses and uses at the local level and then uses this data for pushing for social change at a higher level in government or internationally. This approach to ‘assessing social change’ is commonly found in the human rights tradition. The other perspective is the one that has so far been more central in the ASC discussions – which is viewing assessment of social change as integral to the actual process of inducing change. This means collecting information and different perspectives on the quality of the change process and its impact, critically looking at this and then refocusing and restrategising.
No one would dispute the need to create assessment and learning processes that can help see if change has happened. This can fulfil many purposes as described above. But why not just use the mainstream M&E approaches based on logic models that have dominated for over two decades? Doug Reeler (2007) summarises the M&E mainstream as follows:
“Created to help control the flow of resources, these frameworks have, by default, come to help control almost every aspect of development practice across the globe, subordinating all social processes to the logistics of resource control, infusing a default paradigm of practice closely aligned with conventional business thinking. As such, Project approaches to change bring their own inbuilt or implicit theory of social change to the development sector, premised on an orientation of simple cause and effect thinking. It goes something like this: In a situation that needs changing we can gather enough data about a community and its problems, analyse it and discover an underlying set of related problems and their cause, decide which problems are the most important, redefine these as needs, devise a set of solutions and purposes or outcomes, plan a series of logically connected activities for addressing the needs and achieving the desired future results, as defined up front, cost the activities into a convincing budget, raise the funding and then implement the activities, monitor progress as we work to keep them on track, hopefully achieve the planned results and at the end evaluate the Project for accountability, impact and sometimes even for learning.”
By and large, the reality is that mainstream monitoring and evaluation (M&E) does not serve the types of change processes discussed in this paper. Standard M&E systems and processes have evolved from an image of development as infrastructural. As social change occupies an increasing proportion of development agencies’ budgets and priorities, the tensions with the expectations of standard M&E are growing. Batliwala (ibid) argues that the core motivation is fraught, leading to problems in practice. Many M&E efforts occur, she says, because donors require them, enabling organisations to sustain or obtain funding that is used to expand and consolidate organisational structures rather than innovate or invest directly. Result assessment data is rarely shared with primary stakeholders, target groups are rarely involved in setting goals or shaping evaluation frameworks or in assessment processes themselves. Furthermore, such processes are rarely accompanied by or lead to critical reflection on or re-casting of the theories of change that guide the work.
There is growing recognition of the limitations of mainstream M&E approaches to do justice to developmental social change. Dlamini (2006) refers to the dominance of an instrumentalist managerialist approach to M&E that interferes with organisational intentions “to stand back from their ‘doing’ and genuinely try and see how things are going” and inhibits the creation of the relationships on which change is based. The Institute for Development Research Canada (IDRC) has developed an alternative approach, outcome mapping (Earl et al 2001), that recognises the diffuse nature of development while Guijt (2007) advocates a dialogic and sense-making focus to make learning possible. Organisations like Oxfam International and ActionAid International have revised their M&E processes to build reflection in at all levels, based on the recognition that learning can then occur through the conversations that are made possible.
Moving to assessment and learning that strengthens social change means recognising the specific features of such developmental processes and then accommodating these methodologically. Although fundamentally, it remains about gathering evidence and analysing it in the context of intended effects, by prioritising the local relevance of assessment and learning, question marks emerge about the merits of information needs and modalities that have been developed to serve donors. Differences occur in what evidence is considered important and credible, how evidence is gathered and particularly processed, the rhythm (frequency and speed) with which this takes place, and so forth.
The methodological challenges will depend on the nature of the change process. If it is an exogenously-driven pro-poor change process, then the challenges will lie in the interface between the exogenous vision/procedures and the dynamics and information needs of the local change process. Alternatively, if it is an internally driven change process, then the challenges lie with the people, relationships and capacities within the system. In many cases, the challenges lie on both fronts.
Assessment and learning about social change offers different opportunities depending on whose perspective is taken. For insiders to the social change process, it is about creating the capacity for reflection so that innovative breakthroughs can be sustained. For donors, it is about using assessment and learning to question their policies and procedures, while for academics, it is about creating better insights about social change work. Thus far, the last two groups have had a poor track record of using assessment and learning.
Features of Social Change that Affect Assessment and Learning
Five interlinked features of social change have particularly significant implications for how assessment and learning takes place. These are:
non-linear and unpredictable;
multiple efforts on multiple fronts;
the fuzzy boundaries of social change;
the difficulty of recognising ‘valid’ results; and
the long term nature of social change.
Progress towards social justice and transforming relations of power does not follow a linear or predictable trajectory, with uncertainty beforehand about the impact and the most effective route. They are complex change processes, multi-dimensional and resulting from multiple actions and circumstances, involving a mix of intentional and opportunistic actions. Furthermore, the shifting nature of challenges faced, with some obstacles fading while others surface, make a rigid plan of action or accountability on specific results a potential hindrance to strategic efforts. There must be space for seizing the moment and unanticipated innovations. Sheela Patel gives a poignant example of SPARC which reserves 50% of its funding for such precedent setting initiatives, yet struggles to make donors understand the importance of this strategy. Objectives change during the process as a result of contextual changes but also through compromises resulting from working in alliances, thus making the use of pre-set indicators and strict adherence to predetermined objectives a problem.
The system-wide change that is being strived for requires efforts by and depends on multiple groups on diverse fronts; hence the merit of attributing impact is highly questionable. The process and multidimensional nature of pro-poor social change means that efforts intertwine in changing contexts, goalposts inevitably shift, and impact is perhaps best described in terms of ‘emergent’ phenomena3 of change. This makes it irrelevant to talk in terms of attribution to specific individuals, efforts or organisations and trying to disentangle which efforts have made what difference. Recognising the broad system interactions needed for pro-poor social change means letting go of an attribution obsession. Standard M&E approaches based on fixed, time-bound achievements and segmented realities fail to do justice to intertwined efforts over a long time period. A focus on attribution diverts attention from the efforts themselves to who can claim which part.
Drawing the lines in a process of social change with fuzzy and moving boundaries means valuing incremental shifts. A key problem occurs if social change is viewed not as a process with progress markers, but rather as an end point and product. This leads to a focus on concrete outcomes rather than progress markers and ignoring the value of small, incremental changes. Sheela Patel states the challenge as follows: “How can early initiatives and breakthroughs be articulated and learnings that are institutionalised for sustained impact and scalability be seen as potential outcomes of the process?” Being accountable to a process rather than a product to which groups are committed means that “the down stream long term results become the lighthouse that guide the action and not the rod with which impact is measured”, as ASC group member Natalia Ortiz describes. There is a need to capture the little moments of truth, the value of the accumulated small steps, rather than just the big bang at the end (see Box 8). The interdependence of efforts makes discerning progress difficult, with effects only evident if various causes are simultaneously (or subsequently) addressed. The mindsets of many in the development sector stand in the way: Ashish Shah, ASC group membe, evokes a powerful image: “I think the difficulty is in the fact that we’ve all become so result-oriented and target-driven to the extent that the product has become more important than the process. Imagine if a donor was trying to assess Gandhi’s work – how many years would we have waited if were waiting to assess his end dream, yet there are so many lessons we have to learn and so many changes that happened during the Gandhi driven change process.”
Box 8. Seeing success only in terms of the big bang (excerpt from Patel 2007)
This [relocation] project was clearly successful. There were tangible, quantifiable outcomes, partnerships involved, good governance, gender equity, and civil society participation. And yet this kind of assessment is unsatisfactory and even misleading without the full examination of the depth of the relationships of trust that evolved over years, the risk taking and creativity that produced workable innovations, the ‘toolkit’ processes that were refined and systematized over time, the story is a thin one. If the years of working and waiting, of two steps forward and one back, are not valued and not given their due, then the final resulting success is not properly understood. This not only fails to recognise the difficulties, tensions, triumphs and very essence of development, it then fails to help us change our understanding of development – perpetuating strategies and policies that have stood in the way of change that has benefited the poor. We need to see the full complexity and non-linear nature of such social change processes if we are to learn how to ‘do development’ differently.
Recognising a valid result requires valuing efforts along the way. Defining success and failure in a complex process is fraught with interpretation difficulties. Part of the problem is the difficulty of striving towards results that may not be measurable, as it is not always about an improvement or a tangible change. The impact of social change work can take the form of something not occurring or sustaining a past gain. A seeming success can suddenly shift from an upward change trend to stagnation or deterioration – or the reverse. Years of struggle can unexpectedly yield results. Such struggles often entail activities such as organising dialogues, lobbying governments and advocacy work, of which the intermediate results are not always evident. Although focused campaigns have led to quick results, focusing entirely on a tangible change as evidence of impact ignores what is often slow shifts in norms, institutions, political reform over the longer term.
Acknowledging the timeframe of change and clarifying expectations of change is essential. In exogenously driven initiatives, a timeframe mismatch often occurs between the long term impacts and expectations of short-term externally funded initiatives. Many intermediary organisations, such as NGOs, contribute to this by romanticizing and ‘commoditizing’ their social change work, in the process creating unrealistic expectations of the timeframe for goal achievement. Whereas mainstream M&E processes are based on defining change within the given time period, the time needed to effect that change is often much longer and requires negotiating which aspect of change is being expected and will be valued (see Box 9).
Box 9. Social change is like a supermarket (Shah, personal communication)
It’s like trying to define ‘social change’ as a supermarket. For the supermarket to be a supermarket, you need to have several different products, of different shapes and types – from vegetables, to soaps, to juices, etc. In the same way, all of us who intend to be part of a process need to recognize that some of us are juices, some of us are apples, some of us are soaps, but all of us are part of the same process, except with different strengths and weaknesses, ideas, concepts, resources, etc. From the start, all those involved need to be clear first and foremost about what change we are fighting for and at what levels. Some may be comfortable to see change to a certain point, others to a different point. If most of the vegetables in the supermarket actually want to be sold and cooked and eaten than that’s the right reason to be in the supermarket. If one doesn’t want to be eaten then it shouldn’t be in the supermarket. For us, the key lesson with SUCAM4 was to be very sure from the start why all of us were engaged and what we were fighting for – though interests may have been different (interest of farmers were different from those of some NGOs). Once you are sure of what you intend to fight for, and you are sure of all the people involved and what each player bring, then you can go into an honest discussion on what social change is in terms of the process and what it means for everyone and recognize that everything we do matters – the small changes probably matter more than the big visible ones.