Challenge 10.4

Why does this Challenge need to be solved?

There are thousands of requests for data to be shared across multiple organisations in the Scottish Public Sector, feeding a wide range of different statistics and analysis which in turn provide the evidence to inform public good policy and operational decisions. In order to make sure regular decisions are informed by the latest information the process might be more automated for gaining approval for reusing public data and demonstrating this approval for each subsequent update.  

Organisations at different stages of the data value chain have separate legal obligations for its protection, so that they are obliged to maintain their own processes and infrastructure for managing data access, rather than inheriting from a parent organisation. This means there is a lack of a defined, shared process or standards for approving data sharing requests and a way to document this. Because specific legal obligations and rights may vary, the lawful basis for sharing might ideally be tracked across multiple organisations and transformations of the data. 

The incentives for data owners to approve sharing do not align with the public benefit, causing widespread risk aversion. Meanwhile, often the skills and culture of organisations focus on the risks of data sharing rather than the public benefits and an informed weighing of these. This might be addressed by making best practice, precedents, and the latest tools from the Informational Commissioner’s Office, more readily available and accessible to everyone involved in the approvals process.

How will we know the Challenge has been solved?

Data owners and data protection officers are more comfortable sharing data between organisations, regardless of how technical the use for public benefit. The Challenge Sponsor outcome here will be existing information governance approvers agreeing implementation of the solution.

Once data sharing has been approved, every transfer of data will be handled programmatically, without duplicative manual handling or exchanging messages – enabling near-time sharing via Application programming interface (APIs) for seamless public services. The Challenge Sponsor Outcome here will be automation of an inter-organisational data pipeline – which may already have approvals in place before the Challenge.

Organisations made up of siloed teams are still able to share data in pursuit of tackling big Challenges, like child poverty.  

Privacy preserving protocols can be approved, in place of simple data transfers. The Challenge Sponsor Outcome here will be that the prototype solution is able to describe privacy preserving protocols in a rigorous way that can be automatically checked.

Newer tools from the Information Commissioner’s Office are widely understood and used across the public sector, like delegation of approvals or organisation certification. 

Who are the end users likely to be?

We expect the following end users across Scottish Government, Local Authorities and NHS Scotland, and the wider public sector; 

• Information Asset Owners 

• Data protection officers  

• Analysts supporting policy and operational decisions  

• Product Owners and Software developers who use data programmatically 

Has the Challenge Sponsor attempted to solve this problem before?

Whilst we are aware of solutions that can track data access approvals across organisations, we are interested in proposals that fully meet the Challenge outcomes. 

Are there any interdependencies or blockers?

Careful engagement will be needed throughout, with existing information governance processes and boards across different areas of the public sector.

Will a solution need to integrate with any existing systems / equipment?

Depending on the specific service areas chosen, the solution may have to integrate into

• The Data Architecture team’s chosen metadata cataloguing solution. 

• The Local Government Data Platform 

• The National Services Scotland (NSS) Near-Time Data Service 

• Microsoft in the Cloud (M365) platform, aligning with the ongoing National Information Governance Plan.

Any technologies or features the Challenge Sponsor wishes to explore or avoid?

CivTech is tech agnostic. As long as the proposed solution offers the opportunity to solve the Challenge in question, we will consider it. Integrating with M365 is explained here

That said, any proposed solution must be capable of integrating with existing systems as required by the specific Challenge and its Challenge Sponsor, and if appropriate be compatible with current and developing Scottish Government infrastructure.

Much is currently being made of the potential of advanced AI. In truth, just about all the products CivTech has developed over the past few years have AI as part of the tech stack but there is no obligation on your part to go down this route – either with componentry such as machine learning and pattern recognition, or indeed LLMs.

We are looking for the best solution, whatever technology used.

While not limited to the following technologies, the Challenge Sponsor is particularly interested in proposals that explore the following: 

• AI – including Large Language Models, chatbots, and more conventional Natural Language Processing 

• Distributed Ledger Technology - peer-to-peer, cryptography, Blockchain 

• Multi-Party Computation and other privacy preserving techs

• Cloud-agnostic and hybrid solutions 

We would not like to see a centralised approvals management system, as might be managed individually by Scottish Government for the wider public sector, as this will not meet the nuanced requirements of the Challenge.   

What is the commercial opportunity beyond a CivTech contract?

This problem exists anywhere where separate legal entities have responsibilities for managing agreements for data sharing, and have no trusted third party willing to administer these approvals. 

The UK and global public sector will share the same issues and represent a route to scaling out a Scottish solution. Even if the same infrastructure were shared at these different levels, a business model could still be made from providing services for the users of this infrastructure – with a competitive advantage from intimate knowledge of its development. 

Who are the stakeholders?

The specific service areas involved in the Challenge will likely include one or more of: 

• A justice use case liaising with Police Scotland, Scottish Courts & Tribunal Service, the Scottish Prisons Service, the Justice Division of Scottish Government, and the Judiciary

• A Health & Care use case liaising with the relevant Scottish Government, Local Authorities and NHS Personnel.  

• A National Statistics use case liaising with the relevant Scottish Government and Local Authorities.  

Who’s in the Challenge Sponsor team?

The Challenge Sponsor team will be made up of staff from the following Scottish Government Divisions;

• Chief Data Office

• Data Architecture Team

• The Information Governance team, Digital Citizen Division

Representatives from the specific service areas will also be part of the wider Challenge Sponsor team.

What is the policy background to the Challenge?

The Scottish Government Digital Directorate have made a commitment to Scottish Ministers about exploring solutions to allow a strategic approach to data sharing across the public sector.  

DG Communities have convened stakeholders from across Scottish Government to establish a programme of work leading to a business case delivering a step change in public sector data sharing. 

Local Government is exploring the development of a Local Gov Data Returns Platform to allow more systematic sharing of data between Local Authorities and Scottish Government.