When algorithms help governments decide, who makes sure they are fair?

Governments are using algorithms more and more to help sort information, identify risks and decide where officials should focus their attention. An algorithm is basically a set of instructions that tells a computer what information to look at and what to do with it. It can help government process thousands of records much faster than a person could. That can make public services more efficient. But it can also create a serious problem: an algorithm can appear neutral while still treating certain groups of people unfairly.

The Netherlands is now trying to reduce that risk with a new technical standard called NTA 8047. Published in June, the standard gives government organisations a practical way to check whether profiling algorithms could lead to discrimination, either directly or indirectly. The issue may sound technical, but the basic question is simple: if an algorithm helps government decide who gets selected, investigated, assisted or given extra attention, how do we know the process is fair?

Algorithms can discriminate without being told to

One of the biggest misunderstandings about algorithms is the belief that because a computer has no personal feelings or prejudices, its decisions must be neutral.

That is not necessarily true.

Algorithms work with information chosen by people. People decide what information matters, how much importance should be attached to it and what outcome the algorithm should try to predict. Those decisions can have unintended consequences. Imagine, for example, that government wants to identify people who are more likely to become unemployed. An algorithm might look at age, education, employment history and where someone lives.

Nothing about using an address may immediately appear discriminatory. But suppose one neighbourhood has a large population from a particular ethnic group. If people from that neighbourhood are repeatedly identified as higher risk, the algorithm could indirectly treat that ethnic group differently even though ethnicity was never specifically programmed into the system.

That is the type of problem NTA 8047 is designed to catch. The Dutch standard focuses particularly on profiling algorithms, systems that place people into categories or use their characteristics to make predictions about them.

First decide what problem you are trying to solve

One of the most important requirements sounds almost too simple: government should clearly explain what it wants an algorithm to achieve before using one. A case examined in the Netherlands shows why that matters.

Researchers looked at GeoMatch, a system being tested by the Dutch Central Agency for the Reception of Asylum Seekers, COA. The system is intended to help identify areas of the Netherlands where refugees who have received residency status might have better chances of finding work.

But researchers raised an important question. What exactly is the goal? Is it to help someone find the best possible job? To place people in municipalities where employment rates are higher? To reduce welfare costs? Those may sound similar, but they are not the same.

An algorithm designed to reduce welfare spending could produce very different recommendations from one designed to find a person the job best suited to his or her qualifications. NTA 8047 therefore says organisations should define the purpose clearly before deciding whether an algorithm is even the right tool for the job. Alternatives should also be considered.

The information you feed it matters

GeoMatch also illustrates another problem. Ideally, a system predicting someone's chances of finding work might use information such as qualifications, professional skills and previous work experience. Researchers said some of that information was not available, so the system also relied on characteristics including gender, age, language and background. That immediately creates questions about fairness.

The researchers argued that the way the system works could reinforce existing differences between groups. COA disputed their conclusions and stressed that GeoMatch is only intended to support employees, with human judgement ultimately taking priority. It shows that an algorithm does not become fair simply because the organisation using it believes it is fair. Its assumptions, information and results need to be tested.

This is not only about artificial intelligence

Another important point is that the problem extends beyond AI. Government agencies may sometimes say that a system is not artificial intelligence, but simply an algorithm or automated process. That does not mean it cannot affect people's rights. A relatively simple algorithm can still determine who gets flagged for investigation, whose application receives extra scrutiny or which residents are considered more likely to present a particular risk.

NTA 8047 was partly created for profiling systems that may fall outside parts of the European Union's AI Act but can still have significant consequences for people. The Netherlands already has experience with what can happen when automated systems are used without enough oversight. Dutch digital government authorities themselves point to the childcare benefits scandal as an example of citizens' fundamental rights being violated through government fraud-detection systems. That history explains why algorithm oversight has become such a serious issue.

So what does the new standard actually require?

In simple terms, it tells organisations to check their work. Before using a profiling algorithm, they should understand what information is being used, what assumptions are built into the system, who could be affected and whether certain groups could be treated differently. They should document how the algorithm was developed and who is responsible for it. Once it begins operating, the organisation should continue checking the results.

Is one group being selected much more often than another? Has the algorithm's behaviour changed over time? Are the results producing consequences nobody expected? The standard also encourages organisations to clearly assign responsibility for monitoring and correcting problems.

NTA 8047 is voluntary. It is not a new criminal law and government agencies are not automatically violating the law simply because they do not use it. But it gives public organisations a structured way to demonstrate that they considered discrimination before introducing an algorithm.

Why St. Maarten should pay attention

The standard was developed in the Netherlands, but the principle behind it is relevant to St. Maarten. As government becomes more digital, more information about residents can be processed automatically. Algorithms can potentially help with everything from detecting irregularities and prioritizing inspections to processing applications and identifying people who may need government assistance.

A small government with limited personnel can use technology to process information more quickly and identify patterns that employees might otherwise miss. But efficiency cannot be the only consideration. If an algorithm eventually helps determine whose business gets inspected, whose application is flagged, who is considered high-risk or who receives additional government attention, citizens should be able to know that the system has been tested for fairness.

There should also be somebody who can explain how it works. “The algorithm decided” cannot become a convenient answer when a person wants to know why government treated him or her differently.

‍