Owen Lin/Flickr. CC (by-nc-nd)

Four years ago the Walk Free Foundation (Walk Free) published the first ever Global Slavery Index (GSI), with the goal of spurring the international community to action by quantifying human exploitation. For an issue to exist, Bill Gates apparently advised Walk Free’s founder Andrew Forrest, it must be measurable. And if something can be measured, so can the world’s progress be towards eliminating it.

This first attempt, published in 2013, drew harsh methodological criticism. The data underlying the index was much too flimsy for the foundation of a global ranking. In the attempt to attract attention, critics argued, Walk Free had prioritised shocking headline figures over due diligence. Walk Free acknowledged these criticisms but stuck to its guns by broadening the GSI empirical basis for the 2014 and 2016 editions. However, the key weakness remained: surveys about the prevalence of modern slavery existed only for a minority of countries around the world, and the GSI simply extrapolated from the ones it examined to the rest of the globe.

Download this report as a PDF

2017 marks a new departure for the Global Slavery Index, and a huge boost in legitimacy for Walk Free. With its advocacy, it has attracted global attention to the plight of millions of people around the world. The GSI in particular has proven hugely effective in focusing attention on the issue. So for the first time this year, its index is fused with data from the International Labour Organisation (ILO). With additional support from the International Organisation for Migration, Walk Free and the ILO issued a new joint report in September: the Global Estimates of Modern Slavery (GEMS).

The stamp of approval from these two UN-associated institutions places the GEMS at the centre of planning around the sustainable development goals (SDGs) – the global master plan to promote human well-being and sustainability of the planet’s ecosystems – as it directly relates to target 8.7 on the elimination of forced labour, modern slavery and child labour. According to the ILO, “[the] 2017 Global Estimate of Modern Slavery will provide benchmark figures against which progress of global efforts to eradicate modern slavery can be measured”. This is remarkable. Within four years the work of Walk Free, despite continuing and vocal reservations over its quality, has become an official policy tool of the global agenda.

This elevated status of the GEMS invites debate and scrutiny on a new level. As the report notes, “[to] be effective, policies and programmes must be grounded in the best possible understanding of the root causes of modern slavery at both the national and global levels" (p. 15). I couldn’t agree more, and therein lies the problem. We must hold tools like indices and indicators to the highest standards because they are intentionally designed to shape the behaviour of governments, international organisations, and citizens around the world. GEMS will act as a benchmark for the future evolution of modern slavery, so the scope for continually updating the methodology has shrunk – otherwise figures over time would be impossible to compare.

So as long as significant data problems persist – and important ones do – it remains a deficient yardstick for progress. If GEMS measurements are skewed, the policy prescriptions based on them will be skewed as well. The report’s authors should expect governments to take notice of its findings and to consider policies that will make them look better – so if the measurements themselves are off, policies may equally veer in an undesirable direction. The stakes in the struggle against exploitation are too high to tolerate misleading conclusions.

Limited source data and extrapolation

One of the most serious criticisms of the early reports centred on the global extrapolation from random sample surveys that existed for only 19 countries. How do we know that findings for some South East Asian countries hold true for others? The short answer is of course that we don’t. We can make informed guesses, based for example on the economic profile of countries. But those are guesses, not more.

The current GEMS features survey data for roughly a quarter of all countries (48 out of approximately 200). It leaves unclear how global and regional estimates were generated from that amount of data. The methodological annex in the report itself offers no detail, and the separate methodology guide promised on the Alliance 8.7 website is still “coming soon” at the time of writing – several weeks after the report itself had spread around the world.

But even without a methodology paper there is reason to treat the results with caution, as the GEMS itself repeatedly suggests that its underlying data may be rather weak. It states, for example, that data on "[forced] labour imposed by state authorities was derived from validated sources and systematic review of comments from the ILO supervisory bodies with regard to ILO Conventions on forced labour" (p. 11-12). That is vague, to say the least. It remains unclear what these sources are, how they were validated, and by whom. After all, even governments themselves – officially committed to the fight against forced labour – will be loath to incriminate themselves. At a time when getting the issue on the agenda was the main goal, that may have been good enough. With GEMS’s direct link to SDG 8.7, it no longer is.

Within four years the work of Walk Free, despite continuing and vocal reservations over its quality, has become an official policy tool of the global agenda.

Equally, the report offers caveats about the coverage of some parts of the world: "The regional figures are important but should be interpreted with care, bearing in mind critical gaps and limitations of the data. This is especially the case in Central Asia and the Arab States, where few surveys have been conducted despite numerous reports of forced labour and forced marriages occurring. Far more research and survey work is required at the national level to provide a more comprehensive picture".

On the one hand, such honesty is laudable. On the other hand, it should set off readers’ alarm bells. Despite acknowledging “critical gaps and limitations in the data” for the Arab States, as the grouping is called, the report nevertheless lists figures for the prevalence of forced marriages and forced labour in them. These figures, in turn, feed into the global estimates, thereby tainting the accuracy of the headline numbers as well.

Reports with as much political weight as the GEMS should not be papering over data gaps, playing fast and loose with extrapolation, or adding bad data to good simply because nothing better was available. Would it not have been appropriate simply to leave countries or regions with “critical gaps and limitations of the data” blank on the map? It would have made the map – and the global estimate – incomplete, but it would have more accurately reflected what was believed to be known. And how critical are these data gaps anyway? At present, there is no way for me as a reader to find out.

There is a bigger problem, however. In the reporting of statistics more generally, disclaimers about data quality and other warnings to data users quickly get lost. Activists, politicians and journalists are interested in ranks, headline figures, and neat maps, not in the ‘ifs’ and ‘buts’ of data collection. When an organisation publishes a snappy report, it should be aware that the figures contained therein will lead a life of their own, without health warnings attached.

Search “global estimates modern slavery” on Twitter and – unsurprisingly – you will mostly find the 40 million aggregate number. (Even the future, 280-character limit on the platform does not allow much more nuance than that.) As it stands, it is not clear that the authors are themselves convinced that the published figures are solid enough to stand on their own, without all the qualifiers. It is thus fair to ask how responsible it is to publish them – in our world of instant and unstoppable digital propagation – especially now that they are linked to policy through the SDGs.

Where do we draw the line on forced labour?

The most immediate association many people have with modern slavery is forced labour. The GEMS draw on the internationally recognised 1930 Forced Labour Convention to define the latter, which revolves around work that is performed either involuntarily or under menace of penalty. Intuitively this makes sense, but deciding where to draw the lines for each of these terms is much, much trickier.

When does work in a capitalist economy – where everybody but the extremely wealthy are compelled at some level to work or starve – become involuntary? Many people’s options for earning money are extremely limited. When your livelihood hangs by a thread, you hardly have a choice about anything. Until starvation becomes the more appealing option, you must embrace whatever promises to get you through the day, week, year – exploitative work conditions, an abusive partner, criminal activity. If, in your town, you can choose between two exploitative employers, does that mean you ‘voluntarily’ signed up with the one for which you opt eventually?

Street work in Kolkata, India. Eric Parker/Flickr. CC (by-nc)

The point is simple: millions and millions of people on this planet – I don’t have specific numbers – are structurally screwed and have no options to improve their situation. Many of them fall outside the purview of the GEMS because of unspoken assumptions about what constitutes voluntary labour. Similar questions can be asked about “menace of penalty”. There is, for nearly everybody, some sort of penalty for not working. When does that go from the everyday to the exceptional, and should that transition matter?

One of the topics the GEMS include within the umbrella of modern slavery is debt bondage. It defines this as “being forced to work to repay a debt and not being able to leave, or being forced to work and not being able to leave because of a debt”. Does “not being able to leave” include situations where what little you own will be confiscated by creditors if the debt is not repaid? What if that creditor is not a shady man in a dark alley but a legitimate bank or the state? Perhaps the consequences of walking away from the debt make continued work the better choice. But does that make it ‘voluntary’?

Perhaps the consequences of walking away from the debt make continued work the better choice. But does that make it ‘voluntary’?

‘Modern slavery’ suggests an image of domination of a clearly identifiable ‘master’ over the ‘slave’ in question. But social and economic structures can limit the exit options for disadvantaged women and men perfectly well without a single ‘master’. How many people are there around the world whose debt-load forces them to devote their existence to paying it off? We really don’t know – but then, that’s the point precisely.

You could say that the full breadth of these grey area scenarios are not what the GEMS were designed to measure, so it is unfair to fault Walk Free and the ILO for its absence. But once an index is elevated to the status that the GEMS has now reached things are not so easy. The GEMS are meant to attract attention and shape the discussion – that is their whole point. So it is fair to ask whether they steer debate about labour exploitation in the right direction.

Think here of the parallel with gross domestic product (GDP). Critics have long lamented that it ignores all those things that make life worth living. The statisticians have a valid answer: GDP was never meant to measure the quality of our human existence; it is a mundane yardstick for the quantity of goods and services our economies churn out. But the discussion doesn't end there, because GDP growth has become the central yardstick for country progress in many politicians' eyes. And in that way, its mere existence may block other worthy policy goals from sight. It's unfair to blame the statisticians for the economistic slant of GDP, but that doesn't mean its effect on public debate is innocent.

The same it true for the GEMS. Unwittingly or not, they draw a line between forms of exploitation that deserve opprobrium, and those that are legit. If we care about the stark inequalities in today’s global economy and how it structurally violates the human rights of men and women whose labour is exploited day in, day out, then the GEMS casts the net too narrowly. And this narrow definition of the problem may seem to exonerate all those who profit from labour exploitation, even when it does not constitute modern slavery.

There is room for genuine discussion here: there are good arguments in favour of fencing “modern slavery” off from other forms of exploitation. But whatever position you take, that discussion itself is crucial. It would be a great loss if a new officially sanctioned measure would appear to settle that issue.

The sense and non-sense of regional averages

Just as we must be careful about what does and does not get included within the GEMS definition of modern slavery – and the consequences of this for policy – we must also be careful about lumping diverse settings together into ‘regional averages’ simply because they are near each other on the globe. Averages become murky when values are not randomly distributed. Average wages in a country, for example, do not tell us whether loads of people are actually quite miserable – because a minority is doing so well – or whether everybody is doing more or less okay. Similarly, when vastly different populations are averaged together it is easy to get a number that relates well to neither. The average size of cucumbers and tomatoes together, for example, tells us nothing about either.

Do certain countries belong in the same basket simply because they roughly share the same longitude?

The same is true when we start to compare and combine countries. Following convention at the ILO, the GEMS clusters countries into regions. “Africa” tops the rankings with 7.6 per thousand people in modern slavery; then comes “Asia and the Pacific” (6.1); “Europe and Central Asia” (3.9); the “Arab States” (3.3); and finally the “Americas” (1.9). At first blush the Americas have done rather well, but does it really make sense to put Canada and Colombia in the same basket? Or, for that matter, Uzbekistan and Germany? Japan and Bangladesh? Maybe it does, but in the 2016 Global Slavery Index the countries paired together here had completely different scores. Do they now belong in the same basket simply because they roughly share the same longitude?

Future versions of the GEMS, or a new version of the GSI, may well return to per-country figures, and thereby avoid the arbitrary regional clustering. That might reduce but not necessarily solve the problem, as even national figures are fraught with difficulties. They imply that the phenomenon in question is somehow a homogeneous property of a country, in the sense that it applies to one corner as much as to another, to one sector as much as to another, to one social stratum as much as to another. It may easily be the case, however, that forced labour is concentrated in certain regions, sectors, or social strata. Erasing this level of nuance to create a simple set of numbers in big, bold print may – once again – be good for getting attention, but the efficacy of the policy response may well suffer because of it.

Forced marriage and forced labour – apples and oranges?

As the report makes abundantly clear, the 40 million modern slavery headline figure combines forced labour and forced marriages. To my mind, that is an unfortunate choice. What unites forced labour and forced marriages is that the people caught up in them are, in a very direct sense, unfree. But there are also obvious differences, the phenomena have diverse roots, and – crucially – they require disparate policy responses. Why is it useful to fuse them in an aggregate figure while child labour, in contrast, is covered in a separate report, with separate figures? (Forced child marriages, in contract, are included in the GEMS.)

The discussion about the parallels and differences between forced marriages and labour is everything but philosophical. Once we start comparing the “regions” that the report covers, we find that headline prevalence rates of modern slavery hang crucially on whether we integrate forced marriages with forced labour or not. “Africa” is reported as having the highest prevalence of modern slavery. But that is due primarily to high numbers of forced marriages. If we consider only forced labour, both “Asia and the Pacific” and “Europe and Central Asia” in fact score worse than “Africa” does. These details are in the report, but the section headline nevertheless proclaims “The Prevalence of Modern Slavery is Highest in Africa” (p. 26). For a casual reader, that’s a definitive statement, even if she might object to the mixing of labour and marriages upon further consideration.

To make things worse, the report notes that the methodology underlying forced marriage figures is weak: "It is important to note that the measurement of forced marriage is at an early stage and both the scope and the methodologies are likely to be further refined. Accordingly, the current estimates should be considered to be conservative" (p. 43). But if the methodology is shaky, how do we know that the estimates are at the conservative end? Does that not hang on the conceptual haziness of the “forced marriage” concept as much as the measurement problems?

‘Consent’ or otherwise to a marriage is hard to capture, especially when the victims are children. If children – or adults, for that matter – consider arranged marriages the norm, they may well ‘consent’ to their parents’ choice of a spouse. But that does not mean they had a meaningful choice. Furthermore, more than two million of the roughly 15 million people found to be in situations of forced marriage by the GEMS are men. I am no expert on forced marriage by any stretch. But in my mind, for a situation of ‘modern slavery’ to exist there must be some form of exploitation. Is the suggestion here that those two million men are exploited by their (presumably female) spouses? That may well be the case. Equally plausible, however, is that customs forced these men into their marriages. Yet rather than being exploited themselves, they – as dominant partners in arranged marriages – may well exploit their female spouses in the end.

The long and short of it is that we don’t know, and as a reader, I cannot critically monitor the choices that have gone into the aggregate numbers. In light of the figures’ weaknesses, it is not obvious that it is responsible to put them out there and use them as a benchmark for future developments and policy initiatives.

Now what?

Modern slavery is a disgrace to twenty-first century humankind. Walk Free’s and the ILO’s drive to eradicate it deserves our support. That is the spirit in which the critical remarks above should be understood: as part of a discussion about both the pitfalls of our current approach and the best way forward.

Four problems stand out: (1) empirical foundations still too weak to support the global claims based on them; (2) a definition of forced labour that might shut down rather than promote debate about labour exploitation; (3) a questionable aggregation of countries in regional averages; and (4) the dubious fusion of forced marriage and forced labour in a single figures.

What should be done in light of these problems? Albert Einstein is credited with the aphorism that things should be made as simple as possible, but not simpler. A similar maxim applies here: quantify, aggregate, and extrapolate as much as you reasonably can, but no further than that. To my mind, white spots on the map – countries without reliable data – should remain white. Regional averages should simply be omitted when, as is the case here, they are misleading at best. And why not have two separate reports for forced labour and forced marriage?

I understand the temptation to publish comprehensive global figures under a heading as emotive as “modern slavery”. In the early stages of Walk Free’s struggle, that may have been a useful strategy. But now that the goal shifts from attention-grabbing to targeted policy interventions, we should substitute nuance, caution and yet more diligence for shaky aggregates. The long march to justice for the women, men and children suffering from severe exploitation can only benefit.