Aptly enough for an event titled “What counts? Who counts?”, all the speakers at this year’s Association for Research in the Voluntary and Community (ARVAC) lecture agreed that counting and measurement is crucial to proving social impact, but that in and of itself isn’t enough to illustrate it.

Allan Cochrane from the Open University warned that while a quantitative approach to impact offers benefits such as clear measures of success and failure, there is a danger in assuming everything can be counted. There is also the risk of counting for the sake of it, or of focusing on meeting targets instead of improving the quality of the work being done.

Partly for this reason, Allan spoke about the importance of storytelling to supplement figures. In the Q&A after the panel discussion, the Big Lottery Fund’s use of blogs in reporting was referenced as one way of providing context for a process of impact evaluation.

Sioned Churchill from Trust for London said softer approaches to impact measurement were more representative of the diversity and values of the sector. She warned that the influence of big funders and government makes it difficult for charities to push back about how and why they measure impact. It’s important that impact measurement is part of a learning and knowledge-sharing process which aims to improve performance; rather than being exclusively for reporting purposes

Emma Stone from the Joseph Rowntree Foundation pointed out that completely different reports could be produced for the same projects depending on whom carried them out – funder, grantee or external evaluator – and to what end. Whether impact analysis is seen as academic research, an integral part of grant applications or a marketing or organisational strategy tool, the result will vary enormously.

Interestingly, Emma said that JRF’s recent efforts to report on its own impact had made them more aware that the process – which they’d expected from their grantees for some time – could be intrusive and unsettling.

Striking a balance between hard numbers and stories isn’t a zero-sum game, in fact pairing stories with hard data offers a far richer and more nuanced sense of voluntary sector organisation’s work which only helps in getting to grips with the impact its having.

Social impact analysis is not a sexy topic. Yet it has gained increasing profile in the past few months. In the UK, the Social Value Bill is going through the United Kingdom’s House of Lords and is a few steps away from passing into law. Non-profit organisations, foundations and socially responsible businesses are now under increasing pressure to demonstrate their value for money to funders, stakeholders and the wider public alike, as available funding diminishes. There is also a growing buzz around the idea of impact investing, underpinned by interest in the social impact bonds pioneered at in the UK and by initiatives such as the Global Impact Investing Rating System (GIIRS). 

Despite all of this, the practice of social impact analysis remains obscure. The emergence of numerous new methodologies and approaches, often complex and laden with jargon, and the fact that social impact expertise and skills remain scattered across a wide range of sectors, makes it difficult to agree a common set of standards for reporting or assessment amongst practitioners themselves. There are few opportunities for sharing knowledge and information about how best to conduct social impact analysis, or how to put the results to use within an organisation. And social impact analysis as a profession remains little known, unregulated and unsupported.

For funders and charities seeking to assess their impact for the first time, the lack of standards, compounded by the lack of guidance available to them, makes it incredibly difficult to find a social impact analyst, establish their credibility, and choose a suitable methodology.

This is why the Social Impact Analysts Association (SIAA) has been created. SIAA is an international professional body that supports and connects social impact analysts. SIAA’s purpose is to help share and improve knowledge of social impact analysis internationally and across different policy areas, by helping our members learn from one another. The organisation has been  established by a consortium of funders including the Adessium Foundation in the Netherlands, Bertelsmann Stiftung in Germany, New Philanthropy Capital in the UK and PricewaterhouseCoopers Germany.

We launched SIAA at an ‘unconference’ in December 2011, whose agenda was set on the day by our delegates, a majority of whom are now SIAA members. The day ran as a series of open-ended workshops in which delegates identified the most pressing problems we face in practicing social impact analysis; discussed how SIAA and others can help address these issues, for instance through working groups, bespoke research or partnership working between existing initiatives; and committed to working towards these solutions together. We wanted to create a setting in which our members and other thought leaders on social impact could genuinely interact with others and shape the agenda, both on the day and for SIAA going forward.

This is because our members are the real brains behind SIAA. They are the ones with the experience and expertise to define what good practice of social impact analysis looks like, internationally and across different sectors. At the launch, our members told us they wanted more networking opportunities, a chance to exchange knowledge with other professionals in this field, and to learn from developments in different sub-sectors and different countries. A large majority told us they wanted SIAA to represent them as social impact analysts.

Since then, SIAA’s focus has been on engaging with members and developing the practical, professional support they need, as well as raising awareness of social impact analysis, its benefits and limitations. We are working with our growing membership of 130 individuals and organisations – ranging from charities, university departments and third-sector support bodies to grant-makers and large corporations – to expand the conversation about social impact analysis internationally.

Of particular interest to foundations are the networking and outreach events SIAA is holding with the Essl Foundation in Vienna, the Erasmus Centre for Strategic Philanthropy in Rotterdam, and Philanthropy Ireland in Dublin in the coming months. All of these events will explore the implications of the impact agenda for these different contexts and for foundations, who are now facing challenges from national governments and the public about their role in public life and are increasingly looking to social impact as a means of assessing the value of their investments.

Going forward, SIAA will help support the professional development of its members via networking online and through events, by creating working groups and other discussion forums to encourage members to share knowledge and information,  and in the longer term by creating opportunities for training and certification.  SIAA will also tackle the challenge of creating a common language for impact analysis, from principles and guidelines to accepted standards for measurement, analysis and reporting. We have already created a working group on principles to see what gaps currently exist, and help to fill them. We hope in this way to provide information that is useful not just to impact practitioners, but also to the funders and donors commissioning the work.

1. Approach your funders as collaborative partners.  Treat your conversations with your funders about impact evaluation as a constructive dialogue.  Many funders will have a standard reporting template that they ask their grantees to complete on a regular schedule.  That does not mean, however, that you can’t and shouldn’t have a conversation with them about reporting in a different way if you think it would make more sense for your organisation.  Perhaps you have a regular data collection schedule that doesn’t match the reporting time frame they’ve requested.  Or maybe they’ve asked for data that would be hard for you to collect and that, based on your understanding of your theory of change, you don’t think is particularly relevant to your work.  Most funders would be quite happy for you to suggest amendments to their standard reporting framework on a reasonable basis such as this.  Some funders have specific metrics on which they need to report for their own stakeholders, and you should make an effort to be respectful of these and try to accommodate them where possible.  It’s a two-way street.  
2. Plan ahead.  The best time for your first conversation with a funder about how you’ll report back to them is at the time the grant is made (or even before if the funder is willing to engage at this stage), not the day before your first report is due.  It’s hard to re-create data you haven’t been tracking from the start, so getting buy-in on both sides as early as possible about what will be measured is crucial.
3. Build it into your budget.  Evaluation doesn’t have to be expensive, but it’s never entirely without cost.  Most funders expect that there will be a line item in your funding proposal to them for evaluation; be sure that you build it into your plans.
4. Keep it proportional.  In their quest to impress and win the bid, some charities overpromise on what they can deliver in terms of impact evaluation.  Think about what’s really called for from the particular situation and what is realistically achievable.  Most funders will accept a level of reporting back that is consistent with the scale of the commitment they have made to you.

 It’s no secret that the competition among charities for funding has become increasingly intense.  Public sector and increasingly charitable funders are demanding greater evidence of impact from the organisations they support.

Despite a clear business case for greater emphasis on impact measurement and evaluation (“M&E”), many charities, especially small and medium-sized charities with fewer resources, struggle with this area.  Common questions include: “What systems should we be using?”  “How much time and money should we / can we spend on this?”  And in some organisations, the more fundamental question of “Why are we doing this again?” comes up time after time.

At the Impetus Trust, a venture philanthropy funder, we spend a lot of time thinking about impact. It’s a subject that is important to us for three reasons.

1. First, like many other donors, our funding decisions are influenced by investee organisations’ ability to demonstrate the difference that they make.  As Impetus has no endowment, we probably place more emphasis on impact than most other funders, as we ourselves must raise several million pounds each year from our own donors, a mix of grantmaking trusts, corporations and individuals.
2. Our donors are particularly keen on understanding the difference their money makes. This means that Impetus must focus on measuring and communicating our own impact. Time and again, our donors tell us that a major reason they decided to support us was the knowledge that their funds would be well managed.
3. Finally, we commit a substantial level of resource to helping our investees “raise their game” on impact measurement and evaluation, by making clever use of often minimal resources. This is because our primary activity is helping other charities and social enterprises to scale up their operations, and we can only demonstrate the impact of our work if we can get good quality impact data from our investees.

When assessing an application, we don’t expect the charity to have perfect evidence of their impact already. However, we do expect them to have a well-developed theory of change that makes a logically sound link between the work that they do and the results they hope to achieve.  We also look for a cultural commitment to improving impact measurement and evaluation processes throughout the organisation. We want to know that the organisation isn’t interested in M&E purely to please its funders, but because it is genuinely committed to learning what about their service works and what could be improved, so that they can be as effective as possible.

Once we’ve chosen to support an organisation, we work closely with them to help them build their skills and capacity for improved impact measurement.  We invest a lot of time up-front agreeing what measures are really important in understanding the social value that is being created – this usually means focusing on no more than two or three key social impact metrics on the basis of which to assess performance.   We want to measure what matters.

We then work with the organisation to co-develop tools to track these. We don’t think there’s a perfect “one size fits all” tool out there, and we encourage our investees to use the tools that are most comfortable for their organisation. This might include a mix of data collected from surveys, focus groups, observed changes from a dataset collected by a third party, as well as quotes and case studies.

We also help our investees think about their current and future stakeholders and what kind of information about their service or evidence they might require.  As the charity’s operations grow, it may want to be in a position to attract new funders, including through government commissions.  Public bodies and their prime contractors often require very high standards of evidence, and may provide terms to which no charity should agree unless they have good reason to be confident in their ability to deliver the desired results.  So we help our investees understand what would be required of them by potential future stakeholders, and then help them produce this evidence.

In our experience, organisations are often reluctant to invest in impact measurement and evaluation.  Often, this is due to well-intentioned but misguided attempts to maximise the funds available for front-line work with clients. As a charity, your role is to treat a social problem: don’t you owe it to your clients to make sure that you know the solution you’re offering is safe and effective?  What if through an evaluation, you learned that a simple change to what you’re doing could allow your work to have even greater impact, or reach even more people, for the same spending?  Just doing “more of the same” without taking a step back and evaluating the results will never provide an opportunity for improvement.

And ultimately, improving services and resource allocation so that you can make a bigger difference with the money you have is the real reason that organisations should invest in impact measurement.  By building a culture of continual assessment and improvement, combined with honest and transparent communication with your stakeholders, you’ll naturally attract more funding as well.

‘If something isn’t working, do something else’, goes the saying. Though it sounds obvious, we often don’t follow this advice. Many of the social and environmental issues we’ve been trying to address for years persist; clearly, our approaches aren’t working very well.

Hence there is a clear need for innovation in tackling entrenched social problems. However, if we demand evidence of an organisation’s past performance, we in effect discriminate against innovations, which don’t yet have a track record.

And yet there is a way to judge them. Good charities will have good ideas, and implement them effectively. If you’re into formulae, think of it as: Impact = idea x implementation.

Any good organisation will be able to answer the following questions, at the very least:

• What’s the problem you’re trying to solve?
• What activities does the organisation do?
• How do those activities help solve the problem?
• How do you find out whether you are achieving anything? (i.e., what is the research process?)
• What are you achieving? (i.e. what results does that process produce?)
• How are you learning and improving? What examples do you have of learning and improving?

The answer to judging innovations lies in the third question. This, of course, asks for the organisation’s theory of change: the logical route through which its actions are supposed to achieve its intended impact. An established charity should have evidence that connects its activities to its goal: when it distributes chlorine at village water pumps in Kenya, diarrhoea declines, and when it doesn’t, it doesn’t.

A charity with a new or complicated theory of change can’t do that. But any charity should be able to produce some evidence for each stage of its theory of change. This distinguishes it from an organisation taking a random punt on something utterly unknown. The evidence won’t come from its own work, which hasn’t yet yielded fruit; but it will come from analogous circumstances.

For instance, TippingPoint addresses climate change by bringing together performance artists, visual artists and climate scientists in the hope that artists will be inspired to create work in response, raise awareness of the problem and spur action. It has a great long theory of change with loads of links – and obviously it’s too long to be able to demonstrate that every time TippingPoint runs an event, carbon emissions decline.

However, it is possible to produce evidence for each link in the chain. For example, one link is that artistic activity can influence public attitudes and action. This has been powerfully demonstrated for over a century. How the Other Half Lives: Studies Among the Tenements of New York, an essay in photojournalism published in 1890, revealed New York’s slums to many influential people for the first time, and was credited with inspiring political reforms that improved the lives of millions[i]. Similarly, the 1852 novel Uncle Tom’s Cabin by Harriet Beecher Stowe is credited with changing American attitudes towards slavery. Another link is that public awareness drives action, for which evidence might come from public campaigns to discourage drink-driving.

So, though an innovative programme can’t prove its impact, it can (and should) have evidence which supports its theory of change. They’re not taking a wild leap into the dark, but are working on a hunch, which presumably comes from something.

We shouldn’t discriminate against innovation. But we should consider which innovations are likely to work, and which aren’t.

The solution to this problem, as addressed by speakers at recent seminar hosted New Philanthropy Capital and Charity Finance Group, was…well, that there is no simple solution. However, this shouldn’t stop organisations from thinking about impact and how to measure it.

Meredith Niles, Investment Director at the Impetus Trust, encouraged organisations to avoid getting bogged down in picking the perfect tool or approach. She suggested measuring outputs and gradually refining the type of data collected to a handful of carefully chosen metrics. Borrowing a business principle from Facebook, she argued that ‘done is better than perfect’.

Dr. Hugh Rayment Pickard of IntoUniversity presented impact measurement as a journey that is often difficult to begin and takes time. In the case of his organisation, it’s a journey that has been ongoing for over ten years.

IntoUniversity began with simple output metrics like student attendance and the number of programmes delivered, before moving several years later to more elaborate work linking this type of data with academic research to make a more forceful case for their impact. Having gradually built up the complexity from a standing start, they are only now beginning to look at more elaborate strategies using more sophisticated database systems and comparisons with control groups.

The absence of off-the-shelf solution to measuring impact was also made clear by Mind in Harrow’s Mark Gillham. Despite using existing research and frameworks, Mark stressed how much of the process involved adapting questionnaires and to their needs. This is what Dr. Pickard characterised as ‘foraging’ for whatever already exists cand an be adapted to your organisation.

So while there aren’t any ready made answers to many of these issues, small charities should feel confident in their ability to start the process of measuring their impact. After all, they are the ones best placed to explain what they do and the difference they make. Social impact is just a way to help them do it.

For more on the event and some of the key points to think about when embarking on this journey, you can read NPC’s write-up of the event here.

In understanding a charity’s impact, we seek to identify the difference which the charity has made in the world. That is, what has happened which would not otherwise have happened. Though this may sound obvious, impact data rarely actually show it.

For example, imagine a city with poor air quality. A charity works there, trying to persuade drivers to turn off their engines when they’re idling at traffic lights. The charity reports that at the beginning of the year, the air was clean 10% of the time, whereas by the end of the year, it was clean 20% of the time.

Great!

Actually this indicates precisely nothing about whether the charity is doing a good job. Perhaps the improvement was due to the charity; but perhaps it would have happened anyway. Maybe engine technology is improving, or drivers are trying to save fuel because petrol prices are rising. Perhaps more improvement would have happened without the charity: annoying campaigns occasionally provoke people into doing precisely what the campaign is trying to curb.

At this stage, all we have from the air quality charity is ‘before & after data’. So we have an attribution problem. We know what happened but we have no idea why, and therefore we have no clue about the charity’s impact.

To determine the charity’s impact we need to ascertain three things:

1. What happened? In the air quality case, the change from 10% to 20% was pretty clear, but often identifying everything ‘what happened’ is pretty complicated.

2. How is that different from what would have happened anyway?

Since we’re normally allocating scarce resources, analysts usually also need to know:

3. How good are those results relative to other charities’ results?

To answer the second question, we need to understand the ‘counterfactual’: what would have happened anyway. That requires having a ‘control’ – that is, a situation in which everything is the same except the charity’s work. Setting up a control is sometimes easy, often tricky and occasionally impossible. How to do it is a big topic for another day.

Notice that ‘before & after data’ get nowhere near the second question. Yet charities often present results which are in fact just ‘before & after data’. For example, we hear statements such as: ‘awareness of HIV transmission is much higher than when we started’, or ‘following our campaign, the law was changed’. This is no better than saying that ‘before our work, the average height of a child was 1.2 metres whereas afterwards it was 1.3 metres’!

We need to watch out because ‘before & after’ data can be impressively complex or detailed: ‘HIV transmission rates are now 14% in the villages in which we work, whereas they were 20% a year ago’ or even ‘every time we go into a village, the transmission rate drops, and every time we leave, it rises again’, or ‘We use the “Complicated tool” to measure a randomly-chosen sample of 10% of our patients, and we find that in 95% of cases we get a drop in xyz behaviour, with a 15% margin of error and 23% standard deviation’. But complexity and detail are no proof of rigour.

We welcome feedback – please contact us with any comments or to contribute to the next edition.