Between May 2005 and November 2006, a small group of development professionals discussed the opportunities and challenges about assessing and learning about social change in ways that strengthen the change process. This became known as the ‘assessing social change’ or ASC group. Central to the group’s discussions was a common concern with the chasm between the need for reflective social change practice and the existing understanding and repertoire of approaches. What processes for assessing and learning about social change can help improve the strategies and results of social change organisations?
This document draws extensively on the experiences and reflections that were shared through e-discussions, documenting case studies and two workshops. However, the final synthesis remains my own interpretation of the diverse perspectives and additional readings. They are offered with the intention to encourage further dialogue and debate with key stakeholders (donors, citizens, activists, and facilitators), inspire methodological innovation and, above all, shift the dominance of the current paradigm of thinking on assessment and learning to one that helps rather than hinders social change.
The interest in the topic has emerged from three converging developments. A first development occurred in the late 1990s around clarifying the concepts and promising practices of participatory monitoring and evaluation but which appears to have stagnated. A second development has been the strong debates on social movements, democracy building, governance and the role of civil society organisations, including non-government organisations– and the importance of ensuring they remain active learners. A third, and direct, trigger for the subsequent ASC dialogue was the Gray Rocks conference in September 2003 on ‘Strengthening Social Change through Assessment and Organisational Learning’ (Mott 2003).
The conference concluded that despite the emergence of some alternatives to mainstream monitoring and evaluation (M&E) approaches, many of the learning and assessment challenges faced by social change-oriented groups are still uncharted in many ways and remain largely unresolved. Many such organisations resort to mainstream M&E approaches that originated under pressure to show measurable and direct changes. These approaches have proven seriously inadequate when applied to efforts aiming to build capacities and social movements, shifting social norms, and strengthening citizenship and democracy. Furthermore, the almost exclusive focus on accountability to donors has often been to the detriment of self-reflection and internal learning that enhances social change processes and to the detriment of accountability to the grassroots.
Two challenges needed addressing. The first challenge is to make progress with advancing approaches that can better meet the assessment and learning needs of organisers and activists, donors, and the evaluators and learning partners who assist them – while remaining true to their visions and strategies for social change. The second is to create the basis for a vibrant dialogue that can further the reform that is urgently needed in assessment and learning about and for social change. Hence, the strong desire for opportunities to delve more deeply into key issues, share and create useful assessment and learning methods that saw the start of the ASC discussions.
I am grateful to many individuals for their help in thinking through the notion of ‘assessing social change’ over the course of this dialogue. The biggest thanks go to those in the ASC discussion group for identifying critical issues – Marta Foresti, Valerie Miller, Sammy Musyoki, Mwambi Mwasuru, Natalia Ortiz, Sheela Patel, Molly Reilly, Roger Ricafort, Evelyn Samba, Ashish Shah, Ritu Shroff, and Lisa VeneKlasen. Ritu Shroff, Iñigo Retolaza Eguren and John Gaventa provided detailed feedback on the paper. Many thanks to John Gaventa for overall guidance and patience and to Sammy Musyoki for being a committed colleague and sounding board throughout the process. Finally, only with the financial support of the Ford Foundation were the ASC outputs, including this publication, possible – for which my gratitude.
(I)NGO international non-governmental organisation
SPARC Society for the Protection of Area Resource Centres
SUCAM The Sugar Campaign for Change
What processes for assessing and learning about social change can help improve the strategies and results of organisations working to transform inequalities in favour of the poor? Between May 2005 and November 2006, a small group of development professionals discussed the opportunities and challenges related to this question. The ‘assessing social change’ or ASC group shared a concern for the chasm between the need for reflective social change practice and the existing understanding and repertoire of approaches. This paper draws extensively on the experiences and reflections that were shared through e-discussions, documenting case studies and two workshops.
This paper is aimed at development professionals interested in social change processes and in issues dealing with assessment and learning. Four central themes guided discussions and provide the structure of the paper.
Understanding ‘social change’ and how this affects learning and assessment;
Frameworks, concepts and methods to ensure critical reflection;
Understanding and dealing with different actors in assessment and learning; and
Issues of scale and interconnectedness.
The paper ends with an agenda for action for donors, social change activists and their organisations, and facilitators of assessment and learning.
Understanding Social Change and the Implications for Assessment and Learning
The term ‘social change’ is generic and neutral, hence easily co-opted and misunderstood. It needs to be qualified. Poor people, power and process are central to the understanding of social change in this paper, which essentially, and immutably, concerns transformational processes related to the (re)distribution of power.
Change, in general, is a given but pro-poor social change efforts require conscious actions. Social change is a collective process of conscious efforts to reduce poverty and oppression by changing the underlying unequal power relationships. Social change efforts are characterised by multiple actions on multiple fronts that seek a systemic, structural impact.
Strategic adjustment and operational improvements of such efforts are ideally not driven by crisis but by deliberate, information and experience-based reflections. Assessment and learning are the processes of ongoing reflection about visions, strategies and actions that enable continual readjustment. However, by and large, the reality is that mainstream monitoring and evaluation approaches (M&E) do not serve the types of change processes discussed here.
Moving to assessment and learning that strengthens social change means recognising the specific features of such developmental processes and then accommodating these methodologically. Five generic features of social change hinder the use of mainstream approaches. First, progress towards social justice and transforming relations of power does not follow a linear or predictable trajectory, with uncertainty beforehand about the impact and the most effective route. Second, the system-wide change that is being strived for requires efforts by and depends on multiple groups on diverse fronts; hence the utility of a focus on proving achievement in order to attribute impact to specific players is questionable. Third, drawing the lines in a process of social change with fuzzy and moving boundaries means valuing incremental shifts. Fourth, recognising a valid result requires valuing efforts. And finally, it is essential to acknowledge the timeframe of change (and divergence between realities and project lifespans) and, therefore, clarifying expectations of change.
Frameworks, Concepts and Methods: Towards a Purpose-built Assessment Process
The spread and evolution of participatory approaches in development has contributed to the interest and desire to move beyond applications for appraisal and planning to use for monitoring and evaluation. Increased attention to and experiences by some large and influential development organisations with more participatory forms of assessment and learning have helped to draw attention from a purely upward accountability orientation to more interactive accountabilities. As social change groups, particularly those involved in rights-based initiatives, begin to grapple with issues of power more directly as part of a repoliticisation of development, they are also looking beyond conventional tools and techniques to the experiences of social movements over the years. But despite the growing demand for alternatives and increasing attempts to develop new methods or approaches processes, few innovations exist that meet the needs and recognise the challenges as discussed in this paper. This will need to be constructed per context.
When confronted with the limitations of mainstream M&E approaches, many in the development sector seek solace in methodological alternatives. They hope that somewhere there is an approach that will overcome the paradigmatic tensions, enable clarity of analysis, prove effectiveness, and strengthen people’s organisations. While there are some relatively innovative approaches emerging, assessment and learning requires more than a method. To be effective, frameworks, values and skills need to merge with a method to construct an appropriate assessment process.
In practice, creating an appropriate assessment and learning process requires mixing and matching and adapting a combination of frameworks, concepts and methods – to ensure they address the information and reflection needs and match existing capacities. Methods do not need to be either comprehensive or complex. A simple case study can provide a valuable process that forces reflection, articulation and clarification. Popular education offers another example of how methods can be reconceived without requiring a complex ‘new’ methodological invention. Conventional methods may also make an interesting contribution, including (aspects of) the much-critiqued logframe or results-based management, or indicators. However, much caution is required with the use of any logic model due to the inherent assumptions about change that fit uncomfortably with the features of social change discussed here.
A critical methodological aspect is articulating the theories of social change that guide social change. This is an essential starting point for assessing it and learning what to do better. Considerable confusion abounds about what a theory of change actually is. The theory of change that guides personal choices is philosophical, historical, political, psychological and experiential, i.e. ideological.
Being clear about the theories of change that individuals and groups have helps to strategise and provide a focus to learning and assessment. Assessing a pro-poor social change effort effectively requires building a shared, context-specific understanding of how power inequities may be challenged and in which diverse actors and strategies are located. For many, the idea of ‘theory’ and ‘articulating one’s theory’ is a scary thought – and balancing it with practice is essential. Furthermore, even in the context of pro-poor social change, there will be competing versions of and trajectories for social change that require negotiation and acceptance of diversity.
Methods will never be the full answer to the challenges of assessment and learning. Due to its system-wide nature and, therefore, the need to engage a range of actors, assessment and learning for social change will always require negotiating about information needs and about learning modalities. In these negotiations, awareness of the power dynamics inherent in any process that involves using methods, as well as in the overall development process, is central. Elitisms of various kinds will inevitably emerge, which need due attention to ensure that inequitable power relationships are not perpetuated or exacerbated. Assessment and learning go beyond methods to a way of being in relationships that matter. Therefore, critical for ensuring that assessment and learning serve the social change process is the quality of relationships and establishing a trusting (internal) learning environment. This includes being clear about where accountability lies in assessment and learning.
Four short examples offer a flavour of how methodologies can be of use in the context of organisations that support and value social change as discussed here. Mama Cash and ActionAid International use frameworks that allow for great flexibility and yet provide a general direction based on the values they consider important for development. SPARC’s story from India and that of CTA-ZM in Brazil provide insights into how local organisations deal with the need for flexibility and rigour in learning and assessment.
The relationships between actors in assessing a social change process are enormously diverse. The extensive range of relationships that influence and need to be considered in assessing social change determines what is possible and the quality of the assessment and learning. Any single relationship is subject to multiple variables – history of relationship, contextual issues, interpersonal connections, competence of those involved, perceived importance of the social change process being funded, etc. Anyone’s position in this web of relationships will differ, depending on whether being contracted to assess social change (in an evaluator’s role) or assessing social change as part of a funded programme of activities in which they are actively engaged. Central among these are relationship with donors, those active in social change work on the ground, and professional evaluators and facilitators.
Relationships with donors can be particularly problematic when it comes to agreeing on what constitutes social change and how to assess it. A direct relationship between a civil society organisation and a donor is nestled within a more extensive hierarchy of accountability, with different emphases placed on accountability or learning at each level in the hierarchy. Stereotyping and simplistic assumptions that power only resides with donors and that the donors have certain views on social change opposed to those on the frontline inhibit open conversation and self-critical reflection.
Nevertheless, the core issue in the donor-recipient relationship does seem to be the different theories of change that guide decisions and actions. The differences in theories of change has consequences, such as differing expectations of what ‘success’ should be able to occur and how that can be seen, rigid application of standardised frameworks, and the difficulty of bureaucracies to deal with cross-cutting work, integrated change or intertwined phenomena of progress. Clashes in vision and resulting procedural emphasis are exacerbated by contextual factors. Large international NGOs, such as Oxfam and ActionAid International that espouse social change values, face specific and tough challenges.
Managing relationships with donors will probably never be comfortable and the approach taken with donors will depend largely on the donor itself. Different perspectives are inevitable - accepting them and working with that diversity can be healthy.
In many locally driven social change processes, sooner or later an intermediary is inserted or invited to support the process. The position of the intermediary will vary, depending on whether it is in a parachuting assessment/learning role or embedded within the social change process. Working on social change and assessing it in ways that maintain the values requires attitudes and principles, knowledge and skills. Credibility and trust are essential to effective assessment processes and can be seen as a by-product of the core competencies and qualities. These start at the personal level but are ideally reflected in convergence within the organisation around core, non-negotiable values and practices for both social change and assessing social change.
Issues of Scale in Assessment and Learning
An often automatic association with the term ‘issues of scale’ is that of scaling up of impact. However, complications of scale quickly arise when seeking to implement social change at a larger geographic or population scale that can lead to dilution of original principles or strategies, exclusion of significant groups, high costs or long time frames due to the desire to ensure participation. Another set of issues arises in trying to scale up lessons from a specific context. How can lessons be shared meaningfully in other contexts, what are the pitfalls in conveying and taking up lessons from elsewhere?
The central challenge in scaling up is how to stay true to the original vision and processes despite the greater numbers involved and larger diversity of experiences. Can integrity of principles and focus be maintained across levels, with social change staying locally relevant? Can assessment and learning stay grounded in local endeavours, despite larger scales of analysis, thus informing and inspiring them?
Interconnectedness also encompasses the challenges of scaling down. Recently debates about development among bilateral aid agencies and the international financial institutions has fuelled an almost obsessive focus on the ‘national level’. But what about local governance and local accountability? Who is paying attention to this and to ensure that efforts to assess change at national level have local relevance?
A critical factor in scaling down of social change efforts concerns ensuring citizen engagement in development processes that originated from higher levels of generality and abstraction A national or international derived process or policies contains risks for assessment and learning in terms of who participates, who facilitates, and the focus of the learning. International donors are tending to support the strengthening of state bureaucracy but do not always sufficiently value and invest in what is needed for citizens to help build effective states. Whether one’s challenge of interconnectedness lies in scaling down or scaling up, there are risks for both pathways of jumping between levels or scales. For many international development organisations, it is hard to find a good balance between investing in internal processes and global objectives, and keeping an ear to the ground and investing there. In the process of ‘jumping across scales’, the so-called intermediary organisations are often in a particularly tricky position.
A focus on ‘assessing social change’ as advocated in this paper can be helpful to bridge the disconnection between levels that lead to confusion and mismatches across scales. An ‘ASC’ perspective can also help in the debates about accountability, an ongoing critical challenge for international and national NGOs. Emerging practices such as ‘downward accountability’ can be the basis for a dialogue between national or global level strategising and local level needs that can reduce the current disconnect between scales. Accountability and transparency in the assessment and learning process itself is also crucial. Assessment and learning will only ever show part of the picture, and thus should not be burdened with unrealistic expectations. Organisations working at larger scales have an opportunity to merge diagnosis and critical reflection to create a change process in which assessment and learning is embedded. One unresolved issue is how to argue for the intangibles as one progresses along the development chain, up and away from the local level. In such cases, intermediaries working on social change must understand and invest in their role as mediator between scales, which requires clarity about the discourses that dominate at different levels.
An Agenda for Action
A far more politicised understanding of development as social change is gaining strength in contrast to development as projects delivered by external agencies. Critical reflection in strategic alliances with unlikely partners, articulating theories of change, and the role of stories to clarify and convey the complexity of transformation are part of a new emerging discourse and practice. Social movements and evaluators are considered agents of change. The received wisdoms of monitoring and evaluation are being fundamentally challenged based on a different understanding of development itself as complex, emergent, and transformative. However, much is needed to come to assessment and learning processes that strengthen social change rather than hinder it.
Donors are critical partners and can make all the difference in development processes that recognise the value of local social change. However, notwithstanding the use of a discourse that refers to ‘critical reflection’ and ‘learning, donors, by and large, favour a mode of M&E that is rooted in fears of non-compliance of agreements based on a development model that is considered predictable. In practical terms, donors need to rethink the principles on which they base their models of evaluation and learning. Amidst what might seem like a daunting agenda, one action point merits special attention, that of consistency - donors must be more rigorous in aligning their espoused values with the protocols and systems they use.
Intermediaries can play critical roles as innovators, challengers and bridgers – for many a continuation of roles they already take seriously. One critical task lies in dialogues with donors to rethink the basis of assessment and learning processes. As innovators, intermediaries have to scrutinise how they contribute to perpetuating problems. Many of the considerations for donors also apply to intermediaries, who often fulfil a funding role in the development chain and are part of hierarchies of power. A particular area of attention that can strengthen the bridging role concerns better understanding the issues of scale. Methodologically, intermediaries can make important contributions.
The challenges posed here offer many opportunities for facilitators and evaluators. Much has been written in this paper that can inspire those facilitating assessment and learning processes and those responsible for formal evaluations. The core shift that must be recognised is that infusing assessment processes with political consciousness will require new skills and capacities.
Cutting across these key players are action points in which all have roles to play to carry forward the challenge of assessing and learning that strengthens social change. For all those involved – activists, intermediaries, evaluators, donors – generating practical ideas and sharing inspiring examples is essential. This means investing in: efforts to systematise and review the respective benefits and limitations of experiences; training efforts for social change organisations; peer support opportunities for those in social change organisations; and seeding experimentation.
Development is described by Sheela Patel of SPARC as ‘the golden goose’. Assessing and learning about development as a process of social change means charting the ‘golden eggs’, in the form of processes that multiply and serve increasing numbers, building capacities and provoking shifts of thinking in government as well as among the poor. However, by valuing only the eggs, the goose is in danger of serious neglect and can die. External assessment processes are often too rigid to understand the dynamics and processes that lead to mature and sustained social change. New visions of assessment and learning that builds on the reflections in this document would be more effective at strengthening social change that tackles the persisting injustices about which development should care.