Government services are increasingly being automated and technology is relied on more and more to make crucial decisions about our lives and livelihoods. This includes decisions about what type of support we can access in times of need: welfare, benefits, and other government services.

Technology has the potential to not only reproduce but amplify structural inequalities in our societies. If you combine this drive for automation with a broader context of criminalising poverty and systemic racism, this can have disastrous effects.

A recent example is the ‘child benefits scandal’ that brought down the Dutch government at the start of 2021. In the Netherlands, working parents are eligible for a government contribution toward the costs of daycare. This can run up to 90% of the actual costs for those with a low income. While contributions are often directly paid to childcare providers, parents are responsible for them. This means that, if the tax authorities determine that any allowance was wrongfully paid out, parents are liable for repaying them.

To detect cases of fraud, the Dutch tax authorities used a system that was outright discriminatory. An investigation by the Dutch Data Protection Authority last year showed that parents were singled out for special scrutiny because of their ethnic origin or dual nationality.  “The whole system was organised in a discriminatory manner and was also used as such,” it stated.

The fallout of these ‘fraud detection’ efforts was enormous. It is currently estimated that 46,000 parents were wrongly accused of having fraudulently claimed child care allowances. Families were forced to repay tens of thousands of euros, leading to financial hardship, loss of livelihood, homes, and in one case, even loss of life – one parent died by suicide. While we can still hope that justice for these families won't be denied, it will certainly be delayed: this weekend, it became clear that it could take up to ten years to handle all claims. An unacceptable timeline, given how precarious the situation will be for many of those affected.

In addition to the individual hardship, there are also significant structural issues at play. There are more than 200 government ‘blacklists’ in use in the Netherlands. Blacklists like these could soon be turbo-charged through technology: in December, a legislative proposal was adopted by the Dutch parliament that would give the government algorithmic-profiling powers that exceed the ‘preventative fraud detection’ system (SyRI), which was designed to detect false benefit claims and which the courts declared in violation of human rights only a year ago. Allowing for data sharing between government and private companies, it would be an Orwellian nightmare come true. A parliamentary inquiry has been scheduled for the summer of 2022, which will likely be too late for any learnings from the tax benefits scandal to feed into the discussion on the draft law currently ongoing in the Senate.

The automation and further entrenchment of institutional racism is not just happening in the Netherlands, but across Europe and the world. It is happening in all areas of life imaginable. From social welfare to racist predictive policing tools to assigning organs for transplant on the basis of race, sex and class, and the classist, prejudiced system that was used to assign A-Level test results to students in the UK after their exams were cancelled due to COVID: technology is a part of every aspect of our lives. 

The urgency for civil society to further step up its efforts to address issues of automating racism and discrimination is therefore greater than ever. Public awareness raising, advocacy, and litigation will need to go hand in hand with in-depth engagement with policy makers and legislators. An instrument like the recent EU draft regulation on Artificial Intelligence, for example, does not sufficiently take into account that context matters just as much as the technology itself when it comes to the detrimental impact it can have. No matter how much we are promised that the tech is accurate, its deployment in and of itself can have a negative impact on especially marginalised and racialised individuals. Even a perfectly working tool of oppression is still a tool of oppression.

When this negative impact materialises, the groups and individuals that are affected will need to be able to seek redress directly. This means that concrete enforcement actions for human rights violations resulting from the use of technology –– currently missing from many regulatory initiatives, including the draft regulation –– should be possible. This will make possible the kinds of strategic legal cases that civil society can unite around to bring about change and lead to increased protection of human rights in the digital context for everyone.

While technological development can sometimes feel like an unstoppable rollercoaster, this is a false narrative. Technology is developed by humans. We decide what we create, where we deploy it, and how we regulate it. The only urgency we should feel is in the realisation that the decisions we make now will have consequences for generations to come.