The extensive range of relationships that influence and need to be considered in assessing social change determines what is possible and the quality of the assessment and learning. Central among these are relationship with donors, those active in social change work on the ground, and professional evaluators and facilitators. This section considers the web of relationships around social change and its assessment and bones of contention and tension in these relationships. It discusses how those in an external support role can maintain the integrity of their input into social change during assessment and learning processes. It closes with observations on the attitudes and principles, knowledge and skills needed when working on social change and assessing it in ways that maintain the core values.
Seeing the Web of Relationships
The relationships between actors in assessing a social change process are enormously diverse (see Box 18). Two relationships were stressed in the ASC discussions as of particular importance for those who are guiding or facilitating assessment and learning – with donors and with local people and their organisations. The relationships with donors is particularly problematic due to the strong views and obligations around assessment and learning, hence this section focuses largely on that relationship. A third crucial relationship is with state institutions – judiciary, legislative, executive structures at different levels. This was not discussed in depth within the ASC group.
Box 18. Range of relationships that influence and need to be considered in assessing social change
between local groups and the process facilitator/evaluator (external or internal)
among the local groups, if multiple groups are working together on whatever is being assessed
between local group management/leaders and their members or constituents
between members of the assessment team - whether these are internal/external, local or non-local, etc
between those seeking the assessment and their donors further along the aid chain, e.g. a grant-giving NGO and a bilateral aid agency from which it receives funding
between the values that drive individuals to work on social change and the practical realities of deadlines, earning an income, fatigue, families, peer/social pressure, etc
A direct relationship between a civil society organisation and a donor is nestled within a more extensive hierarchy of accountability, with different emphases placed on accountability or learning at each level in the hierarchy. Each relationship in this hierarchy – or web – has specific accountability and learning needs, each with their own information requirements, timeframe and cycle, etc. So at any one time, multiple accountability relationships and as many learning relationships are at play. Understanding this chain of needs, how information is perceived, and what constitutes ‘evidence’ of social change is important to avoid difficult surprises at a later stage.
Any single relationship is subject to multiple variables – history of relationship, contextual issues, interpersonal connections, competence of those involved, perceived importance of the social change process being funded, etc. Thus what emerges in terms of a shared understanding – or not – of social change and of its assessment at any given point is the result of this force field of variables. Interpersonal relationships are particularly important. The personal connections offer scope for clarifying theories of change and perspectives on assessment but can also create a fragile dependency and may be conducive to corruption.
Anyone’s position in this web of relationships will differ, depending on whether she/he is contracted to assess social change (in an evaluator’s role) or if they are assessing social change as part of a funded programme of activities in which they are actively engaged. It is possible that both roles happen concurrently. For example, the international NGOs are increasingly both a partner in action as well as a source of funding for community organisations. This places them in the role of asking for evaluations as part of contractual agreements on accountability, as well as collaborating in reflections on how to strengthen the work. Therefore, it is important to being conscious of the different types of relationships that are simultaneously at play.
Relationships with donors are particularly problematic when it comes to agreeing what constitutes social change and how to assess it. The risk of this proving to be a constraint in implementation and in assessment will be greater if there is strong dependency on one donor or if interpersonal relationships are not fostered. Furthermore, there is often little time or patience to articulate views on social change in these relationships, nor is there the capacity. In these relationship webs, being explicit about the underlying beliefs and assumptions about social change is critical and requires investment in multiple ongoing conversations. But a lack of appreciation of the importance of this discussion and a task orientation means there is usually little to no investment in such dialogue. This leads to the common situation that clashing visions of social change emerge in formal processes of external evaluation, in which the donor’s vision tends to dominate.
Dealing with Donors
Stereotyping and simplistic assumptions about where power resides and who has which views on social change are problematic. Simplistic views on donor-recipient relationships are increasingly inaccurate. Many actors are both, receiving from elsewhere and disbursing funding to others – even at a very local level when revolving credit groups disburse to members. Hence, issues surrounding (upward) accountability pervades the entire web. Many actors who do not consider themselves a donor, do disburse funding to others. So they play a ‘funding’ agency role somewhere in the web and therefore some of the considerations in this section may be relevant to them as well. Development professionals are usually enmeshed in a cascade of relationships, for example, international NGO head office, national level office, civil society organisations (CSOs), community members – which requires shifting roles as one looks up or down the cascade and being aware of the related shift in information needs.
The core issue in the donor-recipient relationship seems to be the different theories of change that guide decisions and actions. Donors formally tend to approach change as linear and able to be planned and, therefore, tend strongly towards using standardized assessment procedures. Meanwhile, the understanding of social change as discussed here views change as non-linear and adaptive and, therefore, requiring embedded and dynamic assessment processes. Many individuals within donor agencies will recognise this but formal organisational positions and protocols hinder its translation into different practices.
The differences in theories of change has various consequences.
Differing expectations exist of what ‘success’ should be able to occur and how that can be seen - donors wishing/demanding evidence of tangible progress following a projected linear trajectory versus social change organisations seeing incremental changes en route (sometimes unanticipated). Or different understandings of what should and can be considered within the boundaries of influence and therefore for what the recipient can be held accountable.
A strong tendency by those in donors for bureaucratically rigid application of standardised frameworks (logframe or its look-alikes) to assess progress, irrespective of the nature of the initiative being funded (for example, requiring a summative evaluation of a certain type), which do not fit the diversity and non-linearity of many social change processes. These frameworks have emerged from a managerial, accountability need rather than the learning or capacity development need that can help to improve social change (see Dlamini 2006; Reeler 2007).
The inability of bureaucracies to deal with cross-cutting work, integrated change or intertwined phenomena of progress, which sits at odds with their issue-based or thematic nature of reporting, line management, and accountability hierarchies.
Social change is not funded in a broad sense but only as a slice of the pie or one piece of the puzzle. This makes it difficult to achieve the kind of systemic change that is driving the work and certainly sits at odds with the timeframe of such change. As a result, the assessment of social change is even more difficult to get funded.
Such clashes in vision and resulting procedural emphasis are exacerbated by several contextual factors. First, is that a dominant evaluation paradigm that stresses results-based measurement in a particular way - measuring, positivism, efficiency as the main reason for funding empowerment work, etc. The focus on this approach to proving effectiveness puts social change work under even more pressure. Natalia Ortiz explains: “Trying to fit complex themes and realities to limited frameworks that are not well understood by those involved, does not help to articulate in a coherent way the social change process that projects and programs aim to contribute to, and may convert the assessment processes in a contract requirement, far from the interests of the parties involved and not useful for the purpose to better understanding social change and develop local capacities.”
Other factors relate to competency and accessibility within organisational hierarchy. Staff incompetence about assessment generally and about in-house M&E frameworks, is common, as is a general inadequate understanding about social change. In those higher up the accountability chain (donors), there is often insufficient competence to adapt assessment frameworks to social change initiatives. Meanwhile, those closer to the ground in CSOs are often insufficiently competent with M&E to be able to negotiate an assessment process on their terms and issues. Furthermore, there is often no access to certain hierarchies of accountability. Therefore, while one’s direct relationship may be with one step up the system, those individuals or layers in turn have other accountability relationships that mean that one is also caught by those (indirect) demands. This constrains the manoeuvring space to expand the understanding of social change and with it, the conditions for assessing it. Progress with direct contacts may be thwarted by them feeling the need to meet demands imposed on them from the layer above.
Large international NGOs, such as Oxfam and ActionAid International that espouse social change values, face specific and tough challenges. Ritu Shroff explains: “This diversity of stakeholders and learning needs puts an incredible amount of pressure on finding a system that does not take up too much time and effort, yet allows us to provide information, obtain feedback on how we are doing and do some authentic learning. I think other large organisations face similar problems, and either end up insisting on very complex and detailed recording and reporting systems that take up too much time and collect data that may be meaningless at the field level, or keeping recording and reporting systems inconsistent and qualitative, which makes accountability and learning at certain levels impossible”. Balancing standardisation given the dynamics of development is a core challenge, in response to which ActionAid International developed its ‘Accountability, Learning and Planning System’ (see section 3.3.3). This system recognises that change is dynamic and value-driven, yet tries to capture specific moments in a standardised system with as little bureaucracy as possible in order to open time for reflection. Even then, self-reflection on the quality of theories of change was difficult. Ashish Shah refers to ActionAid International, his employer: “I think in AAI we reached a stage where we weren’t necessarily doing critical planning or critically thinking through how change happens, making it difficult for us to articulate our understanding of change to others.”
It is useful to be mindful of a commonly heard simplification when it comes to donor demands for accountability. Marta Foresti stresses that ‘accountability’ is important and essential as part of social change processes. It does not need to be tarred with the brush of ‘dirty word, oppressive practice, necessary evil’ and made the scapegoat of the methodological challenges. She urges viewing and using it as an ally, for example, through ‘downward accountability’: “Ultimately, an assessment demonstrating that a donor X has put money in a project Z that not only did not achieve the expected results, but also that the money was misused and reinforced existing power interests and patronages … could be a very powerful tool for local communities for demanding a different approach in the future, in other words, social change. So … be careful not to dismiss the importance and potentially ‘radical role of accountability for achieving social change, and the mechanism/processes necessary for making a reality of it, including assessment exercises.” There is not always a huge chasm between funding agencies and social change groups. Perhaps there is a common interest, namely ‘social change’.
Managing relationships with donors will rarely be comfortable. Although they are partners, they can also be a source of frustration and interference. Finding the right balance in engaging (with) them is a matter of trial and error, and requires some risk-taking. There are usually individuals within any institution who can become champions, excited and transformed by their engagement. But it takes time and patience to develop these relationships – and given the rapid staff turnover in donors, this may be a wasted investment. Decisions about the energy to invest in donor relations means thinking about one’s own organisational mandate. If it is direct poverty alleviation, that leads to one type of link. But if it is to change the way the development sector, and that particular donor, thinks about development, then donor relationships will require another focus.
The approach taken with donors will depend largely on the donor itself (see Box 19). Cindy Clark, ASC group member says: “Building common understanding with donors (and others) requires an explicit conversation about our assumptions and beliefs about ‘how change happens’ and there is not always room for that in the donor/grantee relationship”. By implication, success will vary and be relationship-specific. But in general, working with donors on articulating their understanding of social change success is important. She continues: “We see shifting how large organisations/agencies think about ‘success’ in their work as an important ingredient to building more effective social change efforts. However, while we’ve been successful in some cases, there have been others where the donor is simply not persuaded of the value of certain ‘intangible’ aspects of social change.”
Different perspectives are inevitable - accepting them and working with that diversity can be healthy (also see 3.3). Therefore, achieving harmonious consensus all the time is unrealistic, although it is important to do so sufficiently to ensure effective work. Ritu Shroff comments on the case of Oxfam: “It is important in a large organisation like Oxfam to first try to understand and to respect this diversity of perspectives. This comes from a belief that Oxfam is a better organisation and likely to be more effective because of this diversity. A second value is to try and build consensus using our organisational mission and purpose as the ‘bottom line’. In my opinion, that is about principled negotiation, rather than each person holding on dearly to their own perspective and opinions and fighting for it, either softly or aggressively.” Coming to a shared understanding requires trust and accepting that change can happen within and in the environment as a result of interactions. So dialogic interaction becomes crucial: suspending judgements, being conscious of language, creating new conversational networks, and transforming dysfunctional human and institutional relationships (Retoloza, personal communication).
Box 19. Engaging with Donors
The initial choice of donors/partners is important. Identify and then live by the non-negotiables in terms of this choice. For example, AAI will not accept money from the WB or USAID.
Engage the donor, in particular ‘educating’ them on interim results and aiming to shift those that have limited definitions of ‘success’ to include an eye for the less tangible.
Foster interpersonal, informal relationships and invest in it being an ongoing relationship. Just connecting at the moment of formal evaluation will be less effective. Such relationships may well be harder for smaller CSOs to invest time in and may be potentially fragile or corrupting.
Make the challenges of assessing social change visible to the donor and others and work together on developing appropriate frameworks , processes and procedures. Engage them in analysing why the social change work does not fit standard patterns.
Try to establish clearly, at the onset, the scope of assessment and key definitions. For example, don’t think that just because you both use the term ‘effectiveness’ or ‘empowerment’, it will mean that there is clarity. This point is also valid for partner organisations and CSOs with whom or in which development professionals work.
Seek to have the local organisations and groups involved in negotiating the Terms of Reference and suggest possible evaluators/facilitators who have a social change vision of development.