Algorithms are in the line of fire. Boris Johnson recently blamed the exam results fiasco on ‘a mutant algorithm’, after the one used by Ofqual downgraded almost 40% of the A-level grades assessed by teachers. This has added to mounting concerns about their use in government to make decisions about welfare, immigration and applications for asylum. 

But controversy around the use of algorithms isn’t new. A recent Guardian article reported that around 20 councils have stopped using data analytics to flag claims as ‘high risk’ for potential welfare fraud, and these concerns about reliability and bias have been raised in the past.

This is a global phenomenon, and as more and more decisions become automated, we need to be acutely aware of the true effects of algorithms and adopt a framework to ensure that algorithms are effective and user-centred.


What do we need to do to create ethical and impactful algorithms?

This is the question that we asked as we developed an algorithm to facilitate access to free school meals. The project is led by North Lanarkshire Council with support from the Social Innovation Partnership, a collaboration between the Scottish Government and The Hunter Foundation.

The ‘Benefit Identifier’ algorithm combines revenues and benefits data with education data in order to identify the pupils who are entitled to free school meals and school uniform grant but aren’t registered. The new service enabled the council to register an extra 1,841 pupils in a year. The ‘Benefit Identifier’ also has far wider applications, as the data it interprets – and the model it uses to do so – could be modified to apply to the wider benefits system, helping address the £21bn of benefits unclaimed every year in the UK.
The European Commission and the Open Data Institute provide useful frameworks to explore the question of trust in the realms of data. In this post, we share how we tried to apply some of those principles in practice.

Prepare for the worst

We started this project not long after the Cambridge Analytica scandal, which was a vivid reminder of how wrong things can go if safeguards are not properly put in place from the start and we don’t work up the worst case scenarios. 

Throughout the design phase, we asked ourselves: what could go wrong? How might unintended consequences or negative impact occur? How could the work be misused for a different purpose? Regularly asking these questions influenced some of our decisions.

Being cautious also influenced design choices: firstly, the software doesn’t store any data, it only evaluates the entitlement and wipes the data afterwards. Secondly, it sits on a council’s local machine and doesn’t rely on third party services.

Practically: look at the issues that have blighted previous data projects to draw insights on what could go wrong with a project. Pre-mortem sessions allow us to dig deeper into the potential risks.

Invite scrutiny.

Algorithms used in government services are essentially an application of a law or a regulation. Laws are subject to scrutiny by committees while they’re being prepared. They’re then further scrutinised in parliament when they’re passed, and in court when they’re part of the judicial system. 

Things can get more opaque when it comes to an internal department applying regulations, with risk of misinterpretation or bias creeping in. Citizens and lawyers can only raise concerns in court once the application of a regulation has had a negative impact on a number of people, by which point the damage is done. Like laws, data-led decision making should be open to scrutiny in order to hold the public sector accountable. 

The requirement for transparency must be extended to the data sources which the algorithms use. Issues arise when the public sector uses privately held sources, such as credit ratings, which are difficult for the public and external experts to access.

Data-rich approaches offer huge potential for councils to work more openly and effectively by opening up their inner workings, thereby strengthening democracy and fairness. 

Practically: when the public sector funds, procures or develops algorithms, it should impose an Open Government Licence and request that the documentation and code are published openly to invite the scrutiny of the public and experts.

Choose the right data tools for the challenge.

For readers who aren’t already experts in data science (and for those who are experts, please forgive the oversimplification), it’s worth pointing out that not all data analysis is an algorithm. Algorithms are nothing new; the word itself is derived from al-Ḵwārizmī, a name given to the 9th-century Persian mathematician Abū Ja‘far Muhammad ibn Mūsa. The Merriam-Webster definition is: “a step-by-step procedure for solving a problem”, but currently it’s open to a few interpretations. 

In some algorithms, humans define each step that needs to be applied to get to the results. By contrast, in machine learning or artificial intelligence, the algorithm is programmed to ‘learn’ by spotting patterns as it repeats processes over and over. It becomes impossible to pinpoint how a decision was reached. 

The free school meals algorithm doesn’t use machine learning or artificial intelligence. It takes existing, publicly accountable data, and applies transparent rules which can be modified if better data or insights become available, or a legislative change requires it. When the ‘Benefit Identifier’ determines that a child is entitled to free school meals, it specifies under which clause that decision was reached. The algorithm ‘simply’ replicates the decision tree that a staff member would go through to determine entitlement. Similarly, the brief given by North Lanarkshire Council wasn’t to predict whether children would be entitled to free school meals, but to look at the facts – the data that the council holds. In that sense, it doesn’t use the latest technology available but, rather, the most appropriate.

Practically: it’s tempting to jump on the bandwagon of the latest technologies, such as AI or machine learning, but it’s worth considering whether a tried-and-tested approach might deliver equivalent value and present fewer risks. Unless a successful alpha has informed the technology choice, project briefs should be wary of being too prescriptive about the type of technology to be deployed.

Skill up and lead.

The responsibility of implementing regulations falls on the public sector. Therefore, it must have the leading role in the design and delivery of data-rich approaches in government. Many of the algorithms mentioned in the articles linked above as being particularly biased, ineffective and opaque are developed by the private sector. The organisations own the intellectual property, leaving the public and commissioners in the dark as to how decisions are reached.

Other issues arise when the development doesn’t follow the governance, such as procurement, and project/risk management among others. The definition of the challenge should lead the technology development, not the other way around. Teams should be given the opportunity to work on data-rich approaches of increasing complexity.

To develop the free school algorithm, we worked with council teams who were involved in the end-to-end service: from education, catering, legal, IT, children’s services, and revenues and benefits. We worked with them to design the algorithm and to pick it apart for flaws, and they also participated in testing with users. Involving front-line staff in particular was invaluable, as they have an in-depth knowledge of the service, the users and the data. 

Practically: a data-rich service needs to be created as a collaboration with all stakeholders who use and provide the service. The onus is on the business analysts and data scientists to demystify the technical terms to make the process inclusive, and on the public sector to skill up their teams to lead such projects.

Build trust. 

During research with other local authorities, we heard time and time again from citizens who feel that organisations hold all the power and they have very little say in the decisions that affect them. The recurring issues with algorithms consistently discriminating against the most disadvantaged citizens (and pupils) prove that they are right to be worried.

Here are a few pointers on ways to build trust:

  • Start with the ‘positive’ services.

The £4.1bn of benefit overpayment due to fraud and error pales in comparison to the £21bn of unclaimed benefits every year in the UK. And yet, a lot of data-rich innovation efforts are targeted at detecting the risk of fraud and removing benefits. 

By tackling the free school meals award, the interaction between the public sector and citizens starts on a more positive note. This is particularly true in Scotland where, once a pupil is awarded free school meals, the entitlement is not reviewed until the next school year, even if the family’s financial situation changes. The algorithm is therefore designed only to check new entitlement, not to withdraw existing benefits, answering one of the concerns highlighted in research, where families live in fear of seeing their benefits taken away.  

  • Add to the service, but wait until it is proven to work better than the original process before replacing it.

Families are still able to apply for free school meals as they did before. The Revenues and Benefits service evaluates their application manually based on the evidence provided, and using the same criteria as the algorithm. This is particularly important for the families who are in ‘edge case’ situations, such as living and studying in different council areas, or those with complicated custody arrangements. 

  • Inform customers and seek their consent.

The council informs resident families that they have an automated process, and that should they wish to, they can check regularly if they have become entitled to free school meals and/or clothing grants. If the family gives consent, the council reminds them every year that they can withdraw it at any point. The project tested all the communications with customers to make sure they were clear, and that customers could provide informed consent.

  • Test with staff and users to understand impact at a person level.

Even if it’s impossible to pinpoint how machine learning reached a particular decision, it’s possible to test with staff and users whether the decision correctly reflects their situation, before any irreparable damage is done. Testing the algorithm with pupils (not) sitting exams last year would have identified that it discriminated against students from deprived backgrounds. 

Conclusion

It is right for the public sector to review their use of algorithms in light of the bias and unfairness that has been uncovered. However, used correctly, data approaches have tremendous potential to help them deliver better services to their customers, at a time when they need them most. Proper governance needs to be followed to avoid causing harm. It’s not good enough to pass the buck to algorithms – mutant or otherwise – as if they had a life of their own. 

We put forward the idea that the government already has a tried and tested framework to do this successfully: the UK Government Service Standard. An algorithm used in government IS a service and, as such, it should follow the applicable standard. These 14 principles, outlined in the Government Service Standard, would ensure that algorithms are user-centered and effective.

1.Understand users and their needs.

2. Solve a whole problem for users.

3. Provide a joined-up experience across all channels.

4. Make the service simple to use.

5. Make sure everyone can use the service.

6. Have a multidisciplinary team.

7. Use agile ways of working.

8. Iterate and improve frequently.

9. Create a secure service which protects users’ privacy.

10. Define what success looks like, and publish performance data.

11. Choose the right tools and technology.

12. Make new source code open.

13. Use and contribute to open standards, common components and patterns.

14. Operate a reliable service.


Originally written by Anne Dhir – Available on Medium