Over the past decade, protection in humanitarian crises has grown significantly as a sector. As a recent review pointed out, however, the assessment of protection and activities has not been adequately covered in debates on evaluation of humanitarian action. Although we know evaluation is important, we still do not have clear agreements on what constitutes success and how to measure it. Yet, being able to gauge success (or lack thereof) will clearly have an impact on how we proceed in future crises.

Community based protection in the DRC

Oxfam’s protection programme in the Democratic Republic of the Congo (DRC) has certainly faced challenges in evaluating its effectiveness and progress. Since 2009, Oxfam has provided support to Community Protection Committees (“committees”) in the DRC to identify protection threats, ranging from illegal road barriers, taxes and forced work, to early marriage and others forms of gender-based violence. Once these threats are identified, the committees engage with local civilian and military authorities on actions to address the threats through positive dialogue, local awareness-raising and advocacy actions.

Crucial for evaluation, and particularly  Results-Based Management approaches, is determining what realistic changes a programme is seeking to achieve. 

The evaluation of this work presents a number of pitfalls. Crucial for evaluation, and particularly Results-Based Management approaches, is determining what realistic changes a programme is seeking to achieve. In the DRC, however, what “success” looks like is often a compromise. For committees, this has included negotiating the removal of some, but not all, illegal road barriers in which individuals wishing to pass by were taxed, or agreeing with local commanders that they are only allowed to take produce from farmers one day a week. Despite having an important impact upon individuals’ lives—for example, making a trip to the market profitable or not—such compromises can be difficult to communicate, and are subject to value judgments by an evaluator. Who decides what success looks like?

When positive changes do occur, frequently they are long term and non-linear. Although advocacy may result in the removal of a protection threat, such as a road barrier, a new threat may arise the next day, or the same threat may return the next month. Turnover amongst local authorities means that the committees and local partners may have to start rebuilding crucial relationships from scratch. The long-term nature of change in attitudes and social norms of community members towards local authorities, and issues such as gender-based violence, mean that at any given moment it can be hard to say to what extent change is occurring, and in what direction.

Congolese women attend an Oxfam-led committee meeting on sexual and gender based violence Bweru, DRC.

Tackling common challenges

Outcome Mapping has proved useful in tackling these challenges. With a focus on identifying changes in different stakeholders (called Boundary Partners) that a programme “expects to see”, “would like to see” and “would love to see”, Outcome Mapping has supported Oxfam’s Protection Team in identifying the series of small but meaningful changes in stakeholders. These small changes act as “sign posts” to indicate whether long-term change is happening. This enables the team to be realistic about what they expect to achieve in different time frames (a one vs. three year project for example), helping to identify both what to evaluate, and what to communicate to donors.

Outcome Harvesting has provided a means to then capture specific instances of changes in behaviour. This data has enabled the programme to move beyond simple statements (e.g., “committees are approaching the authorities and undertaking advocacy”), and to log 24 specific cases of advocacy by committees on different protection issues, and what happened as a result, in a three-month period. As this process develops, it will provide a useful resource for learning and evaluation.

Outcome Mapping often relies on observational methods for recording change, such as journals. Some changes however, such as those occurring in attitudes and social norms surrounding traditional gender roles or early marriage, can be difficult to directly observe and log. The specific changes identified through Outcome Mapping have provided a framework to guide the design of different data collection instruments such as surveys and focus groups. Engaging with team members, for example, revealed that a key change the programme needs to achieve is in the responses of community members to cases of sexual violence. Traditionally survivors have faced high levels of stigma, and in the case of unmarried women and girls, may be forced to marry the perpetrator. This is often called an “amicable solution”, reflecting social norms surrounding the acceptability of forced marriage. The programme seeks to challenge these norms through local engagement activities lead by the committees. The team was subsequently able to draw from best practices in measuring social norms to include survey and focus group questions to measure this norm, such as “what would normally happen here if a woman was raped? How people would react? What would they say about the person?” In some cases responses to these questions have revealed reduced stigmatisation surrounding the survivor, and a reduction in the acceptability of forced marriage as an action.  

Ultimately, the simplest methods have often proved the most effective. Discussing the results of focus groups village by village with all staff members has enabled the programme to identify practical, context specific lessons learned, such as emphasising the importance of free medical certificates for survivors of sexual violence. Whether using Outcome Mapping, Results-Based Management, or other approaches, the process of engaging team members in identifying what changes the programme is seeking to achieve, in who, and within what time frame, can be an important spring board.