Sprint Challenge: How can we use technology to predict and measure the impact of projects and investments?

Top Level Summary

Across the public and third sector, organisations have a duty to evidence and demonstrate impact, as a way of justifying the funding and support they receive as well as to support cases future projects, grants, or other types of support. The organisations and programmes in the public sector are multi-sector, multi-disciplinary with a rich breadth of primary and secondary outcomes. And  while evidencing impact from financial or numerical-based indicators is commonly required, what many organisations struggle with is being able to consistently and fully demonstrate impact that relates to societal, economic, cultural or environmental benefits. Further, public sector organisations spend high volumes of money paying for consultants to undertake these impact evaluations, which do not always deliver meaningful or actionable insight.

Fundamentally, this is about making sure that the public and third sectors are able to demonstrate their delivery for Scotland – including its businesses, culture, environment and people – in a rounded, well-grounded and robust manner, and so help to continuously improve that delivery.

Challenge Sponsors Roles

Social Investment Scotland [SIS], and the CivTech programme at the Scottish Government share a similar challenge in their need to find a consistent, holistic and streamlined way to collect, measure, report and communicate impact against designated impact frameworks. With this Challenge SIS will provide a use-case through the development stage, and with a successful outcome it is possible CivTech will become a second use-case. However the solution should be capable of being used by or multiple organisations and initiatives across the public sector and beyond.

SIS and CivTech have partnered with the National Performance Framework [NPF] [Scottish Government, Directorate for Strategic Outcomes & Performance], and while SIS and CivTech have a need to demonstrate impact, the interest of NPF is a little broader. Not only do they need to be able to see how Scotland’s public sector is performing, but a successful solution may also enable them to further shape the framework and how they engage with partner organisations and teams.

Social Investment Scotland impact criteria

Social Investment Scotland [SIS] is Scotland’s leading social and impact investor. Through their ten-year strategy they aim to ‘build an impact economy’ supporting businesses of all kinds to access the tools, investment and support to deliver their business while minimising harm, benefitting stakeholders and contributing solutions to some of society’s biggest Challenges. 

SIS measures impact on 3 levels:

• Enterprise Level [the organisations and businesses they work with];

• Society Level [the difference these organisations make for end users];

• Field Level [how they influence others to join the journey towards an impact economy].

While SIS has developed robust processes for baselining impact across these levels, and to track the difference they make over the period of their investment, or relationships with others, these are currently held in a variety of formats. This is not ideal as the documentation and tools have been developed at a variety of points in time, which can present challenges for SIS staff and their partner organisations. They also provide snapshots in time which doesn’t easily show the current evidence or impact progress. The methods also do not cope particularly well with impact on the society and field levels. By developing this solution it will aid SIS in measuring and reporting impact, as we scale and support more enterprises. It will also help to minimise ‘impact washing’ by establishing processes that provide some level of evidence, based on best practice. 

CivTech’s ‘360 Bottom Line’ methodology

CivTech is the Scottish Government’s flagship innovation programme. It currently runs some 15-16 innovation and tech Challenges every year, with the number set to increase as the programme scales. Challenges will typically have one or two Challenge Sponsors [organisations that want to use collaboration and innovation in technology to solve a problem] each, though there are some Challenges that have more co-sponsors. At the CivTech’s Accelerator stage, each Challenge usually have one company developing a product in response to the Challenge [though again, some Challenges might take more companies through].

The Challenges run by CivTech and its collaborators are multi-disciplinary and broad in their themes, requirements and solutions – ranging from shared repairs in tenement buildings, enabling new employees to the Scottish Government to start their roles more rapidly, increasing the yield of tree seeds, and decarbonising manufacturing. 

As such, CivTech needs a solution for demonstrating impact that is similarly flexible and holistic. The team does not currently have access to a process that readily accommodates reporting on primary outcomes, or on secondary outcomes. 

For example –

Onboarding new employees: a primary outcome might be regarded as the number of new employees onboarded utilising a new pre-employment checks technology, and the associated time/cost savings; and secondary outcomes might involve people joining Scottish Government more quickly and efficiently, and therefore making a swifter impact on, say, their work on Covid projects and pandemic management

Shared repairs in tenement buildings: a primary outcome might be regarded as swifter, more efficient repairs reducing costs at source; and secondary outcomes might involve the environmental impact of repairs that make buildings more energy efficient. 

Similar to SIS, CivTech’s current approach relies on a range of methods including quarterly reporting from alumni companies, surveys, anecdotal evidence, and considerable in-house research as to primary and secondary outcomes. CivTech developed a ‘360 Bottom Line’ methodology, which was further developed with independent consultants. 

CivTech’s research established that existing evaluation approaches would struggle to capture the outputs and outcomes it was identifying for Challenges, and that it would be in everyone’s interests to have an approach capable of handling all benefits of all Challenges, and that could model predicted [potential] benefits before any Challenge was confirmed [so that the Challenge Sponsor could be as confident as possible they would gain value for money] as well as tracking the benefits as they emerged post product deployment. There is a need, not currently being met, to enable reporting against a number of deliberately broad categories. Such a system would then be a robust planning tool as well as a robust reporting and evaluating system capable of measuring delivery throughout the life of a Challenge solution product. 

The National Performance Framework (Directorate for Performance & Strategic Outcomes) Criteria 

An example of the impact frameworks that public sector organisations work towards is the National Performance Framework [NPF] which describes the type of country in which we all want to live . The National Performance Framework is our main vehicle to deliver and localise the United Nations Sustainable Development Goals [SDGs]. The NPF shares the same aspiration for social, environmental and economic improvements, defining our country’s success as more than just growth in GDP. The NPF consists of 11 National Outcomes, aligned to the 17 SDG’s, and we measure progress towards these outcomes by assessing progress being made in 81 National Indicators. The NPF is the “North Star” for the Scottish government and Civil Service, with Ministers required to report on progress at regular intervals and to review the National Outcomes every 5 years, as stated in statute in the Community Empowerment Act 2015. 

Currently, reporting on the 81 National Indicators, as the difference from the last data point, is the only measure of impact that we capture and report on. This falls short of being able to make any assessment or report on progress at the higher Outcome or National level, or to make comment on how this progress translates into progress towards the SDGs. Primarily, National indicators use multiple different data types and criteria for change, making it difficult to establish a suitable method of aggregation for progress at outcome level. 

Similarly, the NPF is very high level, and despite being framed as the vision for all of Scotland, we currently have no way to collect, measure, analyse, report and communicate the impact of multi-sector activities on the National Outcomes. We are asking all public bodies, third and private sector to demonstrate their alignment and impact on the NPF, and have due regard to the outcomes in planning and delivery, but we have no standardized way of assessing their impact. We also have no way to allow others [e.g. charity sector or businesses] to evidence their contribution on the National Outcomes. As the NPF sits above all other frameworks in Scotland, there are multiple levels of aggregation required to capture impact from the delivery of interventions or services on the ground, up to how that is impacting progress towards the National Outcomes. There is also an abundance of other, lower-level frameworks that have outcomes and indicators that the Scottish government must also pay regard too and report their progress towards, making the act of reporting impact disparate, time and resource intensive and completely disconnected. 

NPF report impact on the Societal Level against the United Nations Sustainable Development Goals and Big Society Capitals Outcomes Matrix – so it would be useful to map to these frameworks and indicators – alongside flexibility to define other impact indicators on a case-by-case basis. As we work with a growing number of Scottish businesses on their responsible business practices it will be helpful to map the impacts of their activities to Scotland’s National Performance Framework. ESG [Economic, Social and Governance] and Impact are growing areas of interest, and increasingly growth indicators, for businesses globally. 

Streamlining and standardizing a way to collect, measure, analyze, report and communicate the impact of multi-sector activities, without losing complexity or sentiment at higher aggregations will be a game-changing solution that will be applicable to every sector and organization in Scotland and help to accurately capture Scotland’s progress towards our National Outcomes. 

Key Specific Outcomes include -

• The successful solution must be scalable and suitable for a broad range of organisations – while SIS will form an initial use-case partnered with CivTech and National Performance Framework, a solution should not be designed just for these programmes.

• The successful solution should be intuitive and user-friendly, and enable all stakeholders to use it – from staff in public sector organisations to their stakeholders to the private/ third sector bodies they collaborate with and invest in.

• The solution should provide a streamlined process to measure and predict the impact of prospective investments / projects to facilitate funding and planning future activities – ranging from due-diligence in advance of selecting / undertaking projects through to periodic reviews through the project / investment lifecycle, and beyond to understand the longer term impact.

• The solution should enable a range of evidence to be accommodated / submitted – ranging from reports through to stories through to creative and visual media.

• A solution should accommodate working towards recognised impact frameworks, in particular the NPF. Overall, we share a goal that the National Performance Framework becomes a driver of change, a one stop shop for multi-sector organisations to cyclically evidence their own impact and use the overall, sector or National Outcome impact reporting as a country to drive decision making, policy change and resource allocations – and then be able to continue to evidence the impact using the NPF as a driver for change has had.

• The solution should provide a way to record activities and interventions that progress Building an Impact Economy at the Field Level, for example through sharing expertise, invitations to develop new funds and collaborations.

• The solution should enable a way to collect, measure, analyse, report and communicate the impact of multi-sector activities on the progress towards the National Outcomes [and SDGs] that can be used by policy teams across Scottish Government, as well as external stakeholders to demonstrate their impact on the National Outcomes

• NPF can use impact data to supplement the reporting on indicators to give a bigger, clearer picture of areas that are doing well or areas that may need more support.

• A successful solution should be able to disaggregate impacts on multiple groups of the population, for example those with protected characteristics under the Equalities act [e.g. Age, Gender, Race] - it is important to get the full picture of impacts on various populations or regions

• SIS will be the primary use-case during the development phase.

• Scottish Government – as co-Challenge Sponsors, the solution once developed will be used by Scottish Government, in particular CivTech team [Digital Directorate] and National Performance Framework Team [ Directorate for Performance and Strategic Outcomes]

• Other impact and Economic Social and Governance (ESG) investors

• Other funders of third sector organisations

• Private sector companies/ stakeholders

• Public bodies, private and third sector will use solution to drive and report impacts

• Potentially independent impact evaluation consultants

Social Investment Scotland have looked at a number of solutions and impact platforms, but have encountered a range of challenges. Some have a broad range of functionality, but can be cost-prohibitive, more suitable to larger commercial financial asset managers, not user-friendly / intuitive or require significant buy-in of additional modules / side products that aren’t required. 

The National Performance team have similarly investigated multiple analytical solutions to the problem of aggregating disparate types of data and coming up with a meaningful measure of progress at higher, aggregated levels. However the data is often complex, incomplete and the question of how and if different data sources or contributions should be weighted differently is still unresolved.

At CivTech, work has been done to look at the ‘360 Bottom Line’, as well as reporting on financial outcomes in terms of jobs / revenue created etc. A rudimentary scorecard has been developed to measure the outcome of Challenges but it has been developed in Excel and is admin heavy. 

Transparency in methodology and limitations is key to any solution. Any solution that measures and aggregates impact will need to be analytically sound and be reviewed by Scottish Government analysts and the Chief Statistician to ensure any reporting was not counter to our understanding of the progress being made through current reporting means. Caveats would need to be clear, in plain English and transparent for all. Resource limitations mean as far as possible the solution needs to be straightforward and not time intensive to manage: contributors should be able to manage their own impact information. Checks and evidence would be required in the process to ensure factual and robust reporting of impacts.

No – we’re open to all ideas!