The Lib Dems’ 8,000-vote lead in yesterday’s Chesham and Amersham by-election sounds very impressive. So does the party’s vote share of 57% to the Tories' 36% – but majorities and percentages can be quite misleading.

The Lib Dem vote jumped from 14,627 to 21,517 – a rise of 6,890 votes. But dig a little deeper and it turns out that much of that was managed by squeezing the Labour vote, which dropped by 6,544 to 622.

The real story of the election was one of Conservative collapse, as its vote dropped from 30,850 to 13,489 – a fall of 17,361. It was turnout that did for the Tories.

If just over two-thirds of the existing Tory vote in the area had turned up, the party would have held the seat. Instead, some 17,000 Tory voters stayed at home.

This looks like a classic example of a Lib Dem voter suppression by-election campaign. On a normal turnout, the party’s poll of 21,517 votes would have merely been enough for a distant second place.

This is not Georgia, but voter suppression takes many forms. Here, a ton of data is gathered, identifying voter affiliations. Tory voters are identified and are bombarded with more than 30 leaflets in a short campaign, so hostile voters are left exhausted by daily leaflet drops and door-knocking. As a result, many grow sick of the whole campaign, and opt to stay at home come polling day.

The most insidious part of this type of active voter suppression is that most of the people doing it, eagerly offering their time as volunteers, have no idea why they are doing it – it is never explained to them.

Is this the start of a progressive alliance?

If there is a wider lesson to be learned, it is around the importance of tactical voting in ousting Conservatives. Remember, the Labour vote collapsed by 6,544 (with only 610 Labour voters left), while the Lib Dem vote rose by 6,890. It is hard to see this as anything other than a straightforward transfer of tactical votes, and it seems to have been with the Labour leader’s blessing – Keir Starmer did not even visit Chesham and Amersham.

The Lib Dems could have won without the tactical Labour votes – the Tory vote slumped to 13,489, which is 1,148 fewer votes than the Lib Dems polled in 2019 anyway. Nonetheless, it is hard to see the Lib Dems having won so convincingly without these squeezed Labour votes, and if they are to stand any chance of holding on in an election where the Tory vote turns up again, then they will need every one of these tactical votes.

What all the opposition parties must wake up to is how the Conservatives are currently the only truly national party in UK politics. Labour as well as smaller parties such as the Lib Dems, Greens, Plaid Cymru and the Scottish National Party have become more like regionally based coalitions than national parties. To win under those conditions in the first-past-the-post system, some element of progressive alliance is not only possible, but necessary. As Lewis Baston says in a shrewd piece, there are many problems, not least of culture:

Both perspectives have a point. Nonetheless, some form of multi-party cooperation is necessary. It's the only viable path to overturning the Conservative majority at the next election.

It is often overlooked that the 1997 general election – the last time a Tory government was ousted – was itself the product of extensive Lib-Lab co-operation. There was policy coordination, aligning around shared goals – which resulted in the Freedom of Information Act, devolution for Scotland and Wales, and proportional representation for European elections. There was also an electoral understanding: nothing as crude as candidates standing down, but the parties shared their lists of target seats.

Instead of going after one another and letting the Conservatives win by default on a rump one-third of the vote, they gave one another a clear run, making it easier to squeeze one another's local voters. In fewer than 10 seats did the two parties seriously go head-to-head. The result was the first Labour government elected in 23 years, and the highest number of Liberal MPs returned since the 1920s.

Lib-Lab realignment alone did not win Chesham and Amersham – this was a seat won on voter suppression. But the 8,000 majority was made possible through Lib-Lab realignment; and if the Lib Dems are to stand any chance of holding the seat, and to gain other Tory-held seats, it is their only hope. Much of their electoral success against Tories in areas like the West Country was built on realigning the anti-Tory vote, drawing tactical support from Labour. Yet parties will need to relearn that cooperation works both ways – whether Lib Dem voters return the favour in Batley and Spen’s by-election next month remains to be seen.

The Lib Dem by-election machine

Liberal Democrat by-election gains are rarer than they used to be. There was one in the run-up to the summer of 2019 in Brecon and Radnorshire – which was then lost at the general election four months later. Before that, there was one one was five years ago, in the affluent commuter seat of Richmond Park in 2016. Before that, we have to look back 15 years to see the Lib Dems gaining a seat at a by-election.

The circumstances of that 2006 victory, in Dunfermline and West Fife, were particularly unusual – the Lib Dems had no leader then, with the late Charles Kennedy having just been forced out. Arguably, with Ed Davey, the Invisible Man of British politics, the party has no leader now, either – though leaders in politics are often overrated in their impact.

Yesterday's by-election felt like a return to the 1990s, when the Lib Dems scored a series of upsets in safe Tory constituencies which the Labour Party could never touch. These were Eastbourne in 1990, Ribble Valley (1991), Kincardine and Deeside (1992), Christchurch (1993), Newbury (1993), Eastleigh (1994) and Oldham East and Saddleworth (1995).

This impressive-sounding run dried up under New Labour, with only one more gain from the Conservatives, the already-marginal Romsey in 2000.

After that, Lib Dem efforts against the Tories fizzled out, most notably in a doomed ‘decapitation’ strategy in 2005: an attention-grabbing publicity stunt which claimed that top Tory MPs would be unseated. They weren't. By 2010, the boot was on the other foot, and Lib Dems were losing seats to Tories. By 2015, all the by-election gains against the Tories had evaporated.

The rise of New Labour wrong-footed the Lib Dems, and the party's by-election operation took several years to re-emerge in the wake of the Iraq War. For a brief period, the Lib Dems reinvented themselves as a left-of-New-Labour alternative in the inner cities, before being decimated after the loss of Charles Kennedy in 2006.

While this latest result looks very much like a 1990s-style Lib Dem by-election gain from an unpopular Tory government, the turnout factor is the big difference. And that is why voter suppression was also key.

Sticky voters

The Lib Dems' anti-Tory by-election victories of the 1990s used this tactic sparingly: Newbury in 1993 had a 22,055 majority on a 71% turnout; Christchurch had a 16,427 majority on a 74% turnout in the same year. It was when fighting Labour in the 2000s that the Lib Dems started using voter suppression far more in by-elections: the Labour vote was found to be ‘stickier’, and so if voters couldn't be converted, they could be turned off from voting altogether.

Thus the Lib Dems picked up Brent East in 2003 by 1,118 votes on a 36% turnout, Leicester South in 2004 by 1,654 votes on a 42% turnout, and Dunfermline and West Fife in 2006 by 1,800 votes on a 48% turnout. Yesterday's by-election came in at a 52% turnout, down a third from 77%.

Does Chesham and Amersham herald the future of the Lib Dems? History doesn’t hold out too much hope. The 1962 Orpington by-election had very similar figures to this – a Tory majority of 15,000, rather than 16,000, also converted into a Liberal majority of 8,000.

Many column inches were expended on the phenomenon of ‘Orpington Man’, and how he was meant to represent a kind of voter found elsewhere. Margaret Thatcher was allegedly set to lose her demographically similar Finchley constituency, as was possibly even the then prime minister himself, Harold Macmillan, in neighbouring Bromley. None of this came to pass.

However, Chesham and Amersham does tell us something about the Lib Dems returning as a viable force to take on the Tories in some affluent south-eastern English seats.

The Remain-voting commuter belt outside London is becoming more small ’l’ liberal, as young professionals move out of the capital; we saw this in 2019 with St Albans going Liberal for the first time in over a century, and Dominic Raab may yet have cause for concern in Esher and Walton (especially after boundary changes).

Yet as past elections show, this is not a recipe for the Lib Dems picking up swathes of seats – elections like those in 1974 and 1983 showed Liberals picking up plenty of respectable second places in the affluent south-east, yet still failing to break through.

The longer-term problem for the Lib Dems is that this by-election magnifies the party's dependence on nimbyism and opportunistic campaigning.

The Lib Dems are, theoretically, a left-of-centre environmentalist party committed to house-building and greater infrastructure. They won yesterday's by-election by pretending they were against HS2, the new national high-speed railway, (when all their MPs voted for it). They also claimed they were committed to propping up middle-class house prices and opposing new local developments (when the party is committed to a national policy of house-building). The party’s naked nimbyism even prompted some Lib Dem associations to refuse to campaign.

As a party, the Liberal Democrats have an ideology drawing on liberalism and radicalism, stretching back centuries. Yet the party’s own electioneering tactics can seriously undermine this, and be both short-sighted and short-termist. The result is that the party’s “finest hours” in by-elections are, in reality, rarely examples of its finest moments.

Seth Thévoz has previously compiled psephological analyses of Lib Dem election results, from the 2015 general election, through the party’s funding of its election efforts, to the patterns of Lib Dem by-election gains.

Update, 18 June 2021: When this article was first published, it got the date of the last Lib Dem by-election victory wrong. This has now been corrected.