In practice, creating an appropriate assessment and learning process requires mixing and adapting a combination of frameworks, concepts and methods to ensure they address the information and reflection needs and match existing capacities. Inspiration for an overall approach can be drawn from a range of existing perspectives on evaluation. These can be complemented by specific concepts, such as ‘power’ or ‘gender’ to focus the assessment process. These then need to be grounded by considering certain methodological considerations and need to be implemented by applying specific methods.
Several perspectives, or ‘schools of thought’, about evaluation, assessment and/or learning have particular relevance for social change processes (see Box 10). They all seek explicitly to address power inequities and tackle structural causes of injustice, as well as stressing the need for the assessment process to have local value and strengthen ongoing work. This means they are committed to engaging participants in the process, making them fit within the broad domain of participatory development. Each school of thought, or perspective, articulates a set of principles and practices that require adaptation for each context.
Box 10. Seven schools of thought on assessing and learning for social change
Action research and appreciative inquiry pursue action (or change) and research (or understanding) simultaneously to change practices and social structures. The cycle process of action and critical reflection is participatory, value-oriented, and democratic in its intentions.
Organisational learning relates to a set of perspectives and procedures that enable learning to be embedded in a programme or organisation. The focus is pragmatic and seeks to reconcile the need for individual learning with the dynamics of organisational contexts.
Popular education is a school of thought and an educational approach seeking to expand people’s consciousness about how individual experiences relate to larger societal problems, thus making people better able to change the problems that affect them.
Feminist evaluation has explicit emancipatory intentions, views evaluation as a political activity, and stresses that knowledge should be of and for the people who create it.
Participatory and empowerment evaluation stresses the need for people to assess the merits of their own or externally-driven initiatives that affect them, thus enabling improvement and strengthening people’s agency.
Democratic evaluation and dialogue aims at equity and inclusion in programme evaluation, and to promote public accountability and transparency. It seeks to resolve societal problems by creating opportunities that enable the development of mutual understanding and concessions.
Utilization-focused evaluation has as its central tenet the need for any assessment process to be useful in-situ. Its concern for concern for ensuring that learning ensues from an assessment process among those living with the programme or process being evaluated makes it relevant for social change processes.
A second key building block for developing methodological clarity involves clarifying which concepts will guide the question-setting. Each school of thought (see above) can be ‘filled’ in a range of ways. Notwithstanding their underlying principles, they can all be more or less gender-focused, more or less explicit about power relations, more or less centred around the dynamics of conflict, and so forth. Such choices must be made explicitly about the concepts that will guide the questions and reflective practice. Box 11 lists several concepts or ‘lenses’ that help focus on a specific aspect of the change process. Each concept can be used within the context of one (or a mix) of the frameworks discussed above.
Box 11. Key concepts or ‘lenses’ of particular relevance to social change
Rights-based approaches are central to much of the discourse in development that seeks to redress injustices although there is considerable diversity of understanding about the concept.
Power analysis is central to strategising for social justice and pro-poor change – and is central in assessing if it has occurred. Many different terms and understandings of ‘power’ exist, which need clarification prior to use in the context of evaluation.
Gender empowerment offers a powerful lens through with to better understand inequality of relations between women and men, and its redress.
Accountability definitions and issues are increasingly central to development with a surge in deliberate efforts to hold governments to account to citizens, organisational leadership to account to its members, and corporations to society at large.
Peace and conflict resolution contexts offer specific challenges for assessment and learning processes, such as the extreme dynamics and non-linearity of change plus the added urgency.
Change is increasingly accepted as ‘complex’8 for which systems thinking can provide important insights, as it recognises the non-linear, intertwined nature of change and organisations.
An interest in innovation is inevitable in social change. Many such change processes require innovations of some kind including new types of relationships, unknown partners, precedent setting practical work and experimentation.
Capacity-building as a domain of intervention is central to much social change work. Its complexity and diversity offers unique challenges for assessment processes.
Dialogue that fosters relationships of trust is the basis of coming to a shared understanding enough to move forward together.
The third building block is more practical, requiring choices about how to deal with some of the challenging features of social change identified in section 2.2 above, and other key concerns. Practitioners are increasingly critical and vocal about the limitations of mainstream M&E practice. They are questioning some hitherto unquestioned non-negotiables and are adding new issues to the agenda. In so doing, they are opening up the way for the emergence of practical alternatives and greater acceptance of other standards of practice. How to deal with attribution, what to do with the restrictive effect of indicators – without losing the potential (see Box 12), where to locate a concern for ethics and standards? These are some critical considerations that need practical attention (see Box 13).
Box 12. How not to work with indicators (excerpt from Batliwala 2006)
“Target groups or service users or communities are rarely involved in setting goals or choosing indicators. Indeed, their involvement is actively discouraged by many donors as compromising the ‘objectivity’ of the assessment. Yet communities often offer the most sensitive indicators of their own change, and can be far more critical and objective about the distance they have travelled than outside evaluators, who can sometimes completely fail to see the significance of the shift that has occurred. I was present when members of a collective of very poor and oppressed rural women in South India told a group of ‘objective’ outside evaluators that one of their indicators of success was the failure of the upper castes in the village to break their solidarity as a group, despite repeated attempts to do so through bribes and threats. The evaluators had no way of quantifying this evidence, and were clearly uncomfortable with it. So they ignored it and kept asking the women how many cases of wife beating or dowry harassment they had taken up as a group. Since the answer was none, the group was considered to have failed.”
Box 13. Key concerns for alternative M&E practice that require practical choices
The importance of understanding social change and working with assumptions is hard to overemphasise. It means articulating the theories of change that shape strategies and policies and surfacing underlying assumptions.
Dealing with attribution is a recurring headache for those engaged in multi-actor, multi-location, multi-level and multi-strategy change work. How to ‘prove’ causality?
Making the most of indicators (and seeing the limits) means deciding whether or not to use indicators – or opt for questions – and if so, how to construct and use them to tell the story of change.
Ensuring the capacity to assess social change processes means looking at capacity to facilitate critical reflection on power, justice, policy processes, and social change, and at the access and ability of people to design and implement assessment and learning processes.
Caring about relationships, ethics and standards requires a hard and honest look at the often unequal power relations between North and South, donors and grantees, external experts and local people, etc.
Building in critical reflection is the motor that drives high quality assessment and learning, and means stepping out of one’s comfort zone and encouraging critical thinking of participants.
Generalizing insights and systematizing lessons is a growing area of work as assessment processes are called upon to help fuel the new generation of knowledge.
The fourth building block involves the actual selection of methods and tools that can be used to pursue the type of assessment and learning process that has been selected. Many potential methods exist that can be of use in assessment and learning9. Specifically for social change trajectories, this may involve methods that focus on assessing advocacy and policy influencing efforts (Ringsing and Leeuwis forthcoming; Coates and David 2003). It may require using methods to assess partnerships and networks as part of the social change strategy (Wilson-Grau and Nunez 2007; Church et al 2002). Assessing conflict resolution efforts may be relevant, and may prove inspiring even for those not directly involved in peace efforts due to the similarity of challenges with social change processes (Schmelzle 2005).
Two specific methods are increasingly referred to as useful alternatives to logic model-based approaches: Outcome Mapping and narratives.Outcome mapping tackles some of the dilemmas of mainstream M&E that are most tricky for social change initiatives (Earl et al 200110). Demand for outcome mapping is growing rapidly as it provides practical options for tough M&E questions such as: how to understand an individual’s contribution to social change within complex and dynamic partnerships; how to bring analytical rigour to monitoring and analysis based on qualitative information; and how to consciously detect and understand surprises for strategic reorientation. Interest in written or video stories is also growing, as the use of narrative allows the richness of the often complex stories of change to be told. In this context, the ‘Most Significant Change’ method (Davies and Dart 2005) offers a specific approach to using stories that consciously seeks to reveal the extremes, rather than the average.
Rethinking the Idea of ‘Method’
Methods need to be neither comprehensive nor complex. No single method will ever be able to cover all the bases; different methods, tools and techniques provide valuable insights into social change. In three of the cases studies written by ASC participants (see Box 3), simple meetings in which stories were shared and voices listened to fulfilled a central function. Such reflection meetings or workshops can incorporate specific methods from the PLA toolbox, SWOT analysis11, or Social Analysis-Integrated Triangle (based on Training for Transformation, Hope and Timmel 1996). Other methods that might be overlooked include: supportive supervision that integrates monitoring, coaching, mentoring, and joint supervisor-supervisee problem-identification and solving; and 360 degree review, which involves significant preparation and time to probe and understand the questions and issues in a particular context, followed by reflection with the individuals involved in the intervention. This have, for example, been introduced within Oxfam Great Britain. One method that is less well known is that of ‘Socratic Dialogue’12, which allows for in-depth understanding of various issues concerning everyday life. Through group-based rigorous inquiry based on a participant’s lived experience, consensus is sought about the underlying issues and participants’ perspectives.
Case studies can provide a valuable process that forces reflection, articulation and clarification. Four participants from the ASC process wrote a case study of their experiences in exploring assessment and learning for social change (see Box 3). Box 14 offers some views of the case study authors about what the writing and receiving of feedback meant to them. Those providing feedback to the case study authors valued the learning opportunity because:
“when you’re doing a certain kind of work in a certain context, you tend to believe that this is the ‘natural’ way to work. This reminded me that strategies, methodologies etc are so contextual.”
“In some contexts, it’s not unusual to have people trivialize what you do. It’s valuable to be in a place where people don’t do that, and to be able to move away from the myth that change is simple.”
“About the pressure to talk about the good things – you have to do both. The representation to a different party makes it understood in a different way. It’s not appreciated when you explain that there’s good and bad.”
“The North-South dialogue was interesting – and how much it needs to be encouraged. Listening to [them] talk about their experience of the North was really valuable.”
Box 14. Learning from case study writing and reviewing
M. Reilly: This process helped me think about what we need to do next. And how the context in the North influences the opportunities and the time frame very negatively. There are challenges that don’t exist to the same degree in the South.
E. Samba: It was a chance to ask the questions we have never asked before. There are some things I’ll never take it for granted again. For me, they may be normal but how come? Those kinds of questions, I rarely ask.
M. Mwasuru: I tremendously appreciated the group’s input. Realizing that there could be two or three stories in this case study, and the need to find a focus. This highlights the challenge of writing – managing to keep clear of too many details and to focus on what’s most relevant. The details but also the broader issues.
S. Patel: In our case, we could draw many parallels [with the Washington DC case] although the context was so different. Tracking milestones and seeing what decisions have to be made at each point. What choices are available at a particular time? Which choices do you take? The navigation of choices came out very strongly.
Popular education offers another example of how methods can be reconceived without requiring a new or elaborate methodological invention. To be relevant for assessing social change, it should be constructed as a process of problem posing and mutual inquiry, where people explore deeply felt problems, raise questions and challenge assumptions, seeking deeper understanding of the structural and systemic factors that shape the quality of their lives. This includes looking at the political dynamics at play in the evaluation process itself and not reducing it to a technical set of steps. A critical contribution that Freireian and popular education can make is recognising the politics of learning and knowledge production and the complexities and levels of consciousness that shape behaviour. The Freireian concept of ‘conscientization’ (awareness-raising) can serve as an important bridge between education and assessment as it helps to generate critical questions and reflections on the nature and causes of problems that people face.
Mainstream methods may also make an interesting contribution, including (aspects of) the much-critiqued logframe or results-based management, or indicators. This approach seems to stand symbol for the tensions in assessment paradigms. Much has been written on the problems with the logframe approach (Gasper 1997, Reeler 2007) and other similar methods that are derivates of the same rationale and paradigm. The logframe contains elements that some have found useful. For example, it offers a systematic set of questions that can help define a clear strategy, which can, in turn facilitate the task of developing a meaningful assessment process. If undertaken with clarity about the type of critical reflection that is required and whose capacity is strengthened, then logic models can be of value.
However, much caution is required with the use of any logic model. Such a model assumes, as Natalia Ortiz cautions, “that it is possible to standardize the description of all kinds of programs in terms of linear relationships of cause and effect, while the institutional, policy and cultural particularities of each context are overlooked”. Reeler (2007, 13) argues that the use of logic models to guide assessment may be most useful “where problems, needs and possibilities are more visible, under relatively stable conditions and relationships, which are not fraught with crisis or stuckness”. However, he adds, ‘emergent change’ and ‘transformative change’ are more common forms of social change that defy logic models.
Making the Most of a ‘Theory of Change’
Considerable confusion abounds about what a theory of change is. Many think, for example, that the logframe approach involves articulating a theory of change. However, this constitutes a theory of action, which, in turn, needs to be distinguished from an understanding of change and a vision of change (see Box 15). In facilitating or otherwise engaging in assessing social change, it is important to be clear about these distinctions and their role in the process.
The theory of change that guides personal choices is philosophical, historical, political, psychological and experiential, i.e. ideological. It includes personal standpoints or worldviews based on class, ethnicity, belief systems, personal values, commitment, etc. It also includes the short and long term agenda and interest of those involved (individually and collectively) in the process of social change. Mwambi Mwasuru, ASC group member, stresses that this requires those involved in assessing change to also assess themselves in terms of their worldview, interests and agenda at local, national and global level. This can provide clarity about the methodology and methods used by evaluators “and the potential and actual manipulation of tools (intentionally and un-intentionally) by the users of those tools influenced by their short term and/or long term agenda, as well as their worldview or standpoint [and their] inherent biases”.
Box 15. What a ‘theory of change’ is and isn’t (VeneKlasen, cited in ASC 2006)
Theory of change concerns the overarching assumptions and philosophies that influence individual visions and understandings. They shape how each person think change occurs in society.
Theory of action is an organisation’s specific role with respect to achieving a theory of change, based on an assessment of how it can add the most value to the change process.
Vision of change is an individual’s ideal or in some cases, the feasible dream of where she/he wants to go with particular initiatives – the change being aimed for.
Understanding of change has to do with specific methodologies, approaches – like empowerment, popular education, organising, lobbying.
But why is a ‘theory of change’ so important? Doug Reeler (2007:2) sums it up clearly:
“We need good theories of social change for building the thinking of all involved in processes of development, as individuals, as communities, organisations, social movements and donors. The conventional division in the world today between policy-makers (and their theorising) and practitioners is deeply dysfunctional, leaving the former ungrounded and the latter unthinking. …Good concepts help us to grasp what is really happening beneath the surface. In the confusing detail of enormously complex social processes, we need to turn down the volume of the overwhelming and diverse foreground and background “noise” of social life, to enable us to distinguish the different instruments, to hear the melodies and rhythms, the deeper pulse, to discover that “simplicity on the other side of complexity.” We need help to see what really matters. … As social development practitioners we need theory to help us to ask good questions, more systematically and rigorously, to guide us to understanding, to discovering the real work we need to be doing, primarily assisting communities and their organisations to understand and shape their own realities.”
Being clear about personal theories of change helps to strategise and to give a focus to learning and assessment. Take the example of conflict, which is central in addressing inequality. As Lisa VeneKlasen, ASC group member, explains: “If our ‘theory of change’ is shaped around notions of surfacing, understanding and shaping conflict, then the way we approach to social change is different than if we simply rely on a planning framework (that in many cases may embody a theory of change but practitioners may be unaware of it). This changes our sense of how much control we have over the outcomes and how we manage inevitable risks.” Another example is how the role of and balance between individual and collective change is viewed. Some see change as built on bedrock of informed and critical individuals and may choose to invest efforts there. Others may value the united front and critical mass that collective action might offer, and choose to strategise around creating unity and cohesive action.
Assessing a pro-poor social change effort effectively requires building a shared, context-specific understanding of how power inequities may be challenged and in which diverse actors and strategies are located. This, in turn, requires articulating underlying assumptions. It is not uncommon for activist strategies to be based on faulty assumptions of how change occurs and, therefore. where efforts should focus. Or for organisations to stick to their familiar strategies, despite limited effectiveness. This is no easy task as Ritu Shroff, ASC group member reminds us: “The actual process of defining social change and coming to a common understanding of ‘developmental’ social change is perhaps the greatest challenge in assessment and learning”.
For many, the idea of ‘theory’ and articulating ‘one’s theory’ is a scary thought – and balancing it with practice is essential. Some feel that theories are of no value to practitioners but that is a dismissive patronising perspective. The value of theory depends profoundly on the conversation in which it is used. Sheela Patel both cautions and encourages: “When theories are thrown around without justification or explanation, it’s just off putting. But sometimes my work has been reflected back to me through the explanation of a theory in a way that has broadened my understanding of what I do. There are cases where I have been able to justify my practice through references to theory.” Within social change work, and indeed as it should be in all of development, it is about balancing theory and practice. Every time someone acts, there is a theoretical basis for action. But as Mwambi Mwasuru stresses: “If you don’t take time to interrogate theory, to allow it to be informed by practice, then it just becomes a dry thing. When critical thinking is part and parcel of acting, it means a better understanding of ‘theory’. But how can we have the unity of theory and practice that results in useful knowledge?”. Box 16 offers some ideas that emerged during the ASC discussions on how ‘social change’ as theory and experience can be discussed.
Box 16. Ways to Discuss ‘Social Change’ (ASC 2005)
Broach the topic via personal stories based on grounded, concrete experiences.
With this personal, experiential basis, use simple frameworks on identifying assumptions about how change occurs and analysing power dynamics in this. You can also develop a storyline yourself to do this. Or ask questions like: “What changes are you striving for in your respective struggles? Which of these changes would really improve your life and the lives of your children and grand children?’. Asking the right questions is harder than often assumed!
Do this at the onset, as it creates a reference point. Dive into the power/change discussions after a positive, affirmative visioning process.
Be conscious of the power dynamics present in the methods chosen and who is really calling the shots. If discussions of social change occur within an NGO workshop setting, it is already enacting power dynamics based on a certain development paradigm. Other, more socially more embedded media may be useful, such as music and art, which can help to symbolise what otherwise would be limited to linearly logical, rational explanations. For example, do not insist on written reports if these further marginalise participants. Use metaphors or fables to articulate visions of social change and locating people
Take time to un-pack terms like ‘corporate social responsibility’ (which is fundamentally a public relations exercise within existing patron-client relations of power), as the language of rights can evolve into another form of disguised domination by the powerful within the new era and framework of globalization.
Be continually aware of the question ‘change for whom’ that rests behind the social change work and behind assessment and learning about it.
Staying Mindful of Core Principles
Methods will never be the full answer to the challenges of assessment and learning. A mix is needed of frameworks, methods, support mechanisms, and spaces for reflection – the choices for which remain an issue of perspective and deliberation. The danger always lurks to reduce what are by necessity dynamic approaches into ‘technologies’ and over-simplified how-to’s. Therefore, a focus on and processes for critical reflection remain paramount.
Due to its system-wide nature and, therefore, the need to engage a range of actors, assessment and learning for social change will always require negotiating about information needs and about learning modalities. It involves a diffuse information and deliberation system. This requires addressing the well-known questions of who needs to assess and for what. Clarity about and commitment to questions of purpose, use of results, and useful for whom is essential for making decisions about how and when assessment will occur.
In these negotiations, awareness of the power dynamics inherent in the use of methods, as well as in the overall development process, is central. A method and its use cannot be disconnected from the users’ worldview – ‘benevolent’ tools can be used towards undesirable ends. Hence the importance of developing an awareness of the elitisms that enter into assessment and learning, including being clear about who each person is in such a process.
One is always dealing with elitisms of various kinds. Elitisms can include grassroots elites within civil society organisations and communities, when local power elites begin to appropriate people’s struggles and use them for their own advancement. But it also means dealing with intellectual elitism between professionals or donors and community-based organisations. This requires bridging realities to come to a shared understanding of what can be achieved, while being critical about the dynamics to avoid creating new patron-client relationships. Alliances with academia and other professionals can be assets to the grassroots. However, in assessment and learning, the politics of knowledge need particularly stringent ethical consideration, as the knowledge generated in communities is far too easily appropriated and traded on by researchers and development professionals. Explicit negotiation is needed to filter external ideas so that context-specific essentials are respected and power and knowledge strengthen the positions of the marginalised.
View assessment and learning as a ‘way of being’ in relationships that matter. Everything can and should be scrutinised. For those inside the process, a vital aspect is institutionalising all learning, with spaces, capacities and rituals for the necessary reflection. For relationships between those inside the process and those on the outside, Sheela Patel stresses that “the most valuable process of learning occurs within a process of trust between those who are participants and those who are from the outside asking questions seeking to test various theories and hypothesis that deepen and sharpen nascent articulations”.
Therefore, assessment and learning that serves social change will hinge on the quality of relationships and establishing a trusting (internal) learning environment. Seeing relationships as extending well before and beyond a specific assessment or learning endeavour can help, says Sammy Musyoki, to “build trust, confidence and quality interaction that may create opportunities for critical reflections on how we work with each other, what difference our partnership is making in terms of the aspired change”. Informality is key and basing it on relationships rather than procedures or protocols: “the more informal you make the environment, the more truth you get in terms of assessment and learning”, says Ashish Shah. While the most effective learning comes from peers and social equals, this does not mean that we should shirk seeking out hard criticism from our critics and adversaries.
Balance unity and diversity. Interests are continually (re)aligned during a conscious developmental social change process. Reflection on social change requires the capacity to create strategic unity while allowing diversity and complexity of viewpoints and avoiding their dogmatic entrenching. There may also be varying views on the basis of the unity of a social change alliance. This is not dissimilar to a coalition government that must develop a common vision, clarifies Sundar Burra from the NGO SPARC. Everyone agrees to something when signing on, while deviation from this is a practical expression of inevitable diversity. Defining the non-negotiables and core values is essential. The fundamentals must be questioned as a trajectory develops but requires caution, as they should not be portable or suddenly dispensable, making them little more than fads. An important dilemma is how values of unity and diversity can operate within institutional frameworks, as institutional hierarchies usually cannot accommodate them due to their inflexible procedures and lack of space for making changes.
Multi-stakeholder and multi-perspective learning can occur under certain conditions13. This happens when those involved are able to suspend judgements and not allow prejudices to immediately and continually interfere in efforts to think and act ‘out of the box’. It also requires that asymmetric power relationships are managed more horizontally so as to allow other bodies of knowledge to enter the space.
Clarity about where accountability lies in assessment and learning is critical. Ways are needed to recognize multiple accountabilities in different directions and to devise approaches for each that foster learning. As part of ‘downward’ accountability to beneficiaries, information needs to be used to improve the work, not just to share with people, lest it risk becoming a new politically correct protocol.