David Cameron is playing with fire, and he knows it. A much spun speech he will deliver today already dominates the headlines and the BBC's UK news site as he attacks Labour "not delivering" on immigration that he promises to "cut back". There are huge numbers of people without secure, full-time work in Britain, and an election is looming. Out comes that old Conservative trump card: scapegoating immigrants.

Ever since the advent of the mass franchise, when the Tories realised they couldn't win elections simply by being a political gravy train for the rich, foreigner-bashing has been a staple of Conservative electioneering. It's a technique that stretches back to 1904 when they introduced the restrictive Aliens Bill, tapping into the then-growing backlash against Irish and Jewish immigrants.

"For too long, immigration has been too high," Cameron says to party members today. Mass immigration has led to "discomfort and disjointedness" in neighbourhoods because immigrants aren't integrating. As accusations of gutter politics fly, he will undoubtedly protest that he praises the contributions of immigrants, but this well-trained PR man knows how the right-wing press will present it ("Cameron: migration threatens our way of life," says The Telegraph, describing his speech as "his most forthright on the issue") and - above all - this is about timing as the Tories prepare for a drubbing in next month's local elections.

In his speech Cameron takes on "forced marriages", cynically conflating the issue with immigration when the vast majority of people entering the country reject it as much as anyone else. There were 1,735 cases of potential or actual cases reported to the Government's forced marriage unit last year - disturbing, yes, but to be treated separately from a debate on immigration. Cameron's intention is clear: to excite the popular imagination into believing the idea that hundreds of thousands of people are arriving on our shores who reject the "British way of life" and that he is the leader to "defend" us from this invasion.

Cameron is the latest in a long line of politicians and journalists to attack the alleged cultural practices of immigrant communities. As well as forced marriages, there have been a number of other attacks on so-called "honour killings". While brutality must be opposed regardless of its source, the left should challenge arguments made by those bigoted right-wingers who say that horrors such as “honour killings” are representative of wider communities. They are not. For example, a 2007 Gallup poll revealed that 49 out of every 50 British Muslims surveyed opposed the barbarous practice.

Similarly, the left (and not just the left) should support women of all backgrounds fighting for their rights, but not in ways that are counterproductive and make people defensive. When Labour's Jack Straw railed against the veil as a “visible statement of separation” in 2007, it was his turn to grab the headlines. He may have felt it a success to have triggered an outpouring of tabloid Islamophobia – but sales of the veil in his constituency increased.

Although Cameron must be condemned for his dangerous political posturing at the same time, as a Labour Party member, I know it is important to engage in a debate about immigration, and not try and shut down discussion of it as if any objection to it is inherently racist.

The anti-immigration backlash in this country is real. Polls consistently show that a majority of people think there is too much immigration. But at the same time all the evidence shows that that racist attitudes are weaker and less widespread than ever and the most successful racist party in British history should not be flourishing.

I’m sure it’s a story familiar to every Labour activist. On one chilly day canvassing in North West London before last year's general election, I encountered a middle-aged woman with a lot on her mind. “My son can’t get a job,” she said angrily. “But there are all these immigrants coming in and they’re getting all the jobs. There are too many immigrants!” I had to listen carefully to what she was saying – because she had a thick Bengali accent.

The current 21st century anti-immigrant backlash is born from New Labour’s refusal to address working-class frustrations at the consequences of unfettered free-market fundamentalism. There aren’t enough affordable homes to go round – with or without immigrants – because the Tories and New Labour refused to replace council housing stock depleted by Thatcher’s right-to-buy policies. The result? Millions languishing on social housing waiting lists. Housing shortages are caused by the failure to build – not foreign-born people (who make up just one in twenty social tenants) taking the diminishing stock. At the same time, successive governments allowed millions of skilled manufacturing jobs to vanish. In much of the old industrial heartland they were replaced by fewer, poorer quality, and badly paid jobs.

As the personal struggles for ever scarcer resources intensified, politicians and media commentators who aimed fire at immigrants enjoyed more of a hearing. Because the far-reaching social reforms necessary to increase good jobs and affordable homes were kept off the agenda, demands to prioritise what little there was ‘for people like us’ seemed to many to be little more than commonsense.

Last year, I interviewed a range of people in the former BNP stronghold of Barking and Dagenham. Thousands of skilled jobs at the old Ford plant have gone, the housing crisis is acute, and there has been a bigger influx of immigrants than anywhere else in the capital. With no other explanations on offer, all local issues were seen through a racial prism. “They’re getting the houses, and our people, our children can’t get the houses,” one retired care worker told me. “Foreigners come in here and get places… I never got that. My children never got it.” As one local trade union official told me: “I think if Labour would have carried on building houses in this area, you wouldn’t have half the trouble with the BNP.”

Atthe same time immigration does impact on jobs and wages. One 2008 study by economists from Oxford University and the Bank of England found that it was those in the semi-skilled and unskilled service sector who felt the squeeze. According to their estimates, a 10% rise in the proportion of immigrants would take their wages down by 5%. But rather than sealing our borders, we can deal with this by introducing a living wage and preventing foreign workers being hired on poorer terms and conditions.

Politicians and the media have used immigration-bashing as a smokescreen. In the years before the recession hit, corporate profits were booming but wages were already stagnating for the bottom half of workers. In the case of the bottom third, wage packets were actually shrinking. The lack of trade union rights and the consequences of globalisation were far bigger explanations than immigration – but mainstream politicians did not want to ask questions that challenged some of the most basic assumptions of neo-liberalism. Instead, they focused attention on an issue that has a much smaller impact but has the advantage of exploiting people's direct experience and appealing to prejudices, as well as enjoying the vociferous backing of the right-wing tabloids.

Sure, it’s not all economics. If you’ve lived in a homogenously white community all of your life and have had little interaction with other cultures, a sudden influx of immigrants can be disorientating or even alarming. But history shows that this hostility dissipates as mixing takes place. Take Hackney, where I live: it was a stomping ground for the National Front in the 1970s, but the far-right barely exist today in one of the country’s most diverse boroughs.

Like other right-wing commentators, David Cameron has also recently attacked multiculturalism. There are certainly grounds for objecting to how it has been implemented. Faith schools, for instance, represent an appalling attempt to segregate children. And, while it has been fashionable to understand inequality in racial terms, class has been tossed to one side. This has encouraged some white working-class people to develop notions of ethnic pride similar to minority groups, promoting an identity based on race to gain recognition in multicultural society. The BNP has tapped into this disastrous redefining of white working-class people as, effectively, another marginalised ethnic minority.

But these aren't the sort of objections raised by Cameron, who - with clear echoes of hard-right propaganda - argued that multiculturalism was helping to breed extremism in ethnic minority communities. 

Because of the failure of New Labour to give answers to market-driven housing shortages and a lack of good jobs, immigrant-bashing has filled the vacuum. We need to talk about building homes; providing skilled jobs through an industrial strategy; introducing a living wage; strengthening workers’ rights – all things that will take the intensity out of the anti-immigration backlash. Let’s focus on  tax evasion by the wealthy which costs our country £70 billion; on companies upping sticks to exploit cheaper workers in poor countries; and on the fact that this year the top 1,000 richest Britons saw their wealth rise by a third while everyone else was facing pay freezes and job cuts. Let’s not let allow the Prime Minister and his Coalition partners to turn the victims of their market fundamentalism against each other.

After all, we’ve already seen public sector workers and benefit recipients (who also get a kicking in Cameron's speech) scapegoated for an economic crisis caused by the greed of the bankers. Are we really going to allow immigrants to be added to the list?