‘We are in an information war’,declared Hillary Clinton, addressing a congressional committee in March 2011, ‘and we’re losing that war.’ The US Secretary of State was struggling to persuade a reluctant Congress to approve her $47-billion budget for diplomacy and development in 2012. Her blunt admission reinforced what countless think-tank reports have concluded in the decade since 9/11: that the American superpower is losing the ‘war of ideas’ .

1.

After the attacks of 9/11, the United States found itself fighting an idea as well as an enemy. It was not enough to confront Al-Qaida and its global network. America also had to engage in an ideological struggle – a ‘battle for Muslim hearts and minds’ – to undermine the appeal of Al-Qaida and the violent jihadists. This was the longer-term challenge. As the 9/11 Commission warned, those inspired by Al-Qaida ‘will menace Americans and American interests long after [Osama bin Laden] and his cohorts are killed or captured’. Defeating Al-Qaida physically was no guarantee of ultimate victory in the ‘long war’.

But since 9/11, despite much talk of winning hearts and minds, and the outpouring of billions of dollars, two administrations with very different approaches to the issue have both been defeated by it. Hillary Clinton is not alone in believing the United States is losing the ‘information war’. During six months of research in Washington in 2010 and early 2011, I found virtually no one who thought the effort had succeeded.

However, the debate about why it has failed, and how that failure can be remedied, has had an unduly narrow focus. The conventional explanation is that, in the field of ideological warfare, the United States ‘unilaterally disarmed’ at the end of the Cold War. Its closure of the US Information Agency (USIA) – its principal global propaganda weapon against communism – was, according to this view, a blunder from which American ‘public diplomacy’ (its ability to reach out and influence opinion in foreign countries) has never recovered.

This seems to me only part of the story. What has been under way since 9/11 is an ambitious long-term effort to weaken and marginalise Islamist extremism, using all the elements of American ‘soft power’. It is about more than public diplomacy. Its varied instruments include diplomacy, intelligence, broadcasting, scholarships, and economic aid and development. It is the work of soldiers and spies, cops and contractors, as well as diplomats and bureaucrats. It is covert as well as overt. It involves deeds as well as words. It is about policy, not just the marketing of policy.

The unprecedented nature of this ideological struggle – its long-term character, its global scope, its cast of thousands – imposes great strains on a large and cumbersome bureaucracy such as that of the United States. Even the Cold War, that other ‘long twilight struggle’ played out on a global canvas, was in some ways less daunting. Communism may have been the enemy but it was familiar: it came from within the Western intellectual tradition. Since 9/11, in contrast, policy-makers have had to confront a religious/cultural world (Islam) and a family of ideologies (Islamism) with which they are unfamiliar and uncomfortable. What’s more, in the confrontation between NATO and the Warsaw Pact, the West was essentially fighting governments: the people were allies, or potential allies. Now it is the other way round: the West finds itself allied with unpopular Muslim governments against Islamist movements which draw support from the people. This is a much weaker position from which to wage a struggle for hearts and minds.

The nature of the task poses unusual challenges for the policy-maker. How to co-ordinate the activities of so many players, in so many roles, in so many parts of the world? How to measure success in the ideological struggle? Through opinion polls (unscientific at the best of times)? Or by whether the number of suicide attacks goes up or down (a short-term measurement, at best)? Complicating matters further, there is little agreement on the most basic terms. Journalists speak of ‘hearts and minds’, diplomats of ‘public diplomacy’, the Pentagon of ‘strategic communication’. The Bush administration believed it was engaged in ideological warfare; hence the ‘war of ideas’ – a term the Obama administration shuns.

2.

From his first speech, on his first day as president of the United States, Barack Obama committed himself to a new relationship with the Muslim world. He did so with particular authority. He embodied multiculturalism. He instinctively understood oppression and humiliation. He was not George Bush.

During those first five months – from his inauguration in January 2009 to the Cairo speech in June – he seemed to walk on water. I had a modest stake in Barack Obama. As a visiting fellow at Princeton in the autumn of 2008, I had for the first time watched an American presidential election up-close. I did what journalists do: I talked to people in bars and on trains and buses, eavesdropping on lively debates about the personalities and the issues. I loved the all-American bumper-stickers (‘Old White Woman for Obama’) and admired the way young activists working for the Obama campaign got voters registered and motivated.

But even during the spectacle and rhetoric of inauguration day, I could not help wondering if this young, eloquent, dignified president could fulfil the expectations that Americans and people around the world were investing in him. Most of those who voted for him had domestic concerns: they blamed Bush for the economic crisis the country was in, and wanted Obama to safeguard their jobs, homes, and livelihoods. Others were painfully aware that during the eight years of the Bush administration America’s reputation around the world had been tarnished. Bush’s character had been divisive. The Iraq war was almost universally seen around the world as a disastrous mistake – ‘illegal, immoral, and unnecessary’, as a retired British diplomat described it to me in 2003, expressing quiet fury that this misadventure was being aided and abetted by a British government.

Obama had a clean slate. As the senator from Illinois he had opposed the war. Now he was committed to ending it. On the Greater Middle East – the place of Bush’s most signal failures – Obama pledged a new approach which he began to spell out in a series of well-crafted speeches. First in his inaugural address, and then in Ankara, where he spoke before the Turkish parliament, and then, memorably, in Cairo – capital of the Arab world’s most important and most populous country – he offered Muslims around the world the dignity of respect. He directly addressed their grievances (over the Israeli-Palestinian issue, Iraq, Guantanamo Bay), but just as important was his respectful tone, his use of Arabic phrases, his quotations from the Qur’an. He had come, early in his presidency, to one of Islam’s historic capitals and proffered an olive branch.

I watched the Cairo speech on television in a hotel room in The Hague. I was attending a conference addressing the issue policy-makers had wrestled with since 9/11: how to combat jihadist ideology. Now, at last, our Dutch host declared excitedly, the American president had produced the ‘counter-narrative’ that had eluded us for so long. It was as if we had found the Golden Fleece. Part of the skill of the Cairo speech was indeed that, item by item, it unpicked the Al-Qaida narrative – the idea that had taken hold in much of the Muslim world that the West was remorselessly at war with Islam (an idea Bush had unwittingly reinforced) and that it was the duty of all Muslims to rally to the defence of their faith, their lands, and their identity by joining the global jihad against the ‘Crusader-Jewish alliance’. Obama’s message to Muslims was that this was a dead-end.

It was a moment of hope and possibility, but it was short-lived. Within months, Obama’s efforts to transform US foreign policy bogged down. He tried and failed to kick-start the moribund Middle East peace process, to shut down the infamous prison at Guantanamo Bay, to normalise relations with Iran and Syria, to shake off the Bush administration’s tendency to view everything through the distorting lens of the ‘global war on terror’. A widening gap opened up between promise and performance, between the hopes engendered in Cairo and the disappointments of real-world policy.

3.

Bush’s legacy was, to be sure, toxic. He had made the fundamental mistake of mischaracterising the enemy. Key players in his team – Cheney, Rumsfeld, Rice, and others – had come to foreign policy with preconceptions shaped by the Cold War. They saw America’s enemies, in conventional terms, as hostile states. Communism had been a global force but with a discernible centre, Moscow. In contrast, the new enemy (Al-Qaida) was a non-state actor, a kind of malign NGO with transnational reach. Clobbering countries where it was based or thought to be based would not eliminate it. When struck, it dispersed like quicksilver.

America’s failure to understand its enemy or the world its enemy inhabited seriously undermined the ‘war of ideas’. The question that haunted America after 9/11 – ‘Why do they hate us?’ – tended to provoke the wrong answers. It was falsely comforting to believe that America’s enemies did not understand it, and that the remedy was to harness the skills of Madison Avenue to demonstrate that the United States was a benign actor, a force for good that had brought the world science and technology as well as Hollywood and hamburgers.

After 9/11 the Bush administration appointed a Texan advertising executive, Charlotte Beers, to run its public diplomacy. Her claim to fame was that she had successfully marketed Uncle Ben’s Rice and Head & Shoulders shampoo. ‘Charlotte Beers thought you could sell anything,’ says Paul Pillar, at the time a senior CIA official. ‘I don’t think that’s the way it works. Whether you’re selling foreign policy or toothpaste, the product is ultimately more important than the advertising.’ Beers found it tough going, and her successors in the job scarcely fared much better.

Complicating the task, a rift opened up within the administration over the root causes of Islamist extremism. Officials in the State Department, and Pillar and some of his colleagues at the CIA, tended to see extremism as a reaction to foreign occupation of Muslim lands (Palestine, Kashmir, Chechnya) and as being rooted in poverty, under-development, and resentment of local dictators allied to the West.  On the other hand, neo-conservatives such as Elliott Abrams in the National Security Council and Douglas Feith at the Pentagon saw the phenomenon as being rooted in a culture of hatred and intolerance nurtured by bigoted imams, pernicious schoolbooks, and Saudi money.

When I interviewed Abrams and Feith in early 2011, it was clear they still regarded the State Department under Colin Powell and his deputy Richard Armitage with disdain. ‘Powell and Armitage were suspicious of ideas and the people who carry them,’ remarked Abrams. Fighting a ‘war of ideas’ was a task for intellectuals. (‘That’s a snarky statement,’ growled Armitage when I relayed it to him. Were we wrong, he asked rhetorically, to work so hard to bring about an Israeli-Palestinian agreement? The old animosities still smoulder.)

The pragmatists were sidelined. They watched, largely powerless, as the ideologues – enthusiasts for Israel who knew very little about the rest of the Middle East – produced policies tailor-made to alienate Muslims rather than win their hearts and minds. The invasion of Afghanistan in 2001, and then of Iraq in 2003, unwittingly reinforced the jihadist narrative of an aggressive West invading, occupying, and pillaging Muslim lands. Even if American public diplomacy had been more effective, nothing could have persuaded the Muslims of Iraq or elsewhere that American actions were benign. The Coalition Provisional Authority, the body led by Paul Bremer that governed Iraq for the first year after the invasion, made clumsy efforts to communicate with ordinary Iraqis, using what Ali Allawi, an Iraqi government minister at the time, called an ‘alien rhetoric’.

As a test-bed for Bush’s ‘freedom agenda', Iraq was hardly an unqualified success. The dictator had been overthrown, and a form of parliamentary democracy introduced, but faith in the ‘new Iraq’ was undermined by high levels of violence, frequent power cuts, and widespread corruption. Photos of the abuse of prisoners at Abu Ghraib tarnished America’s pretensions as an exemplar of human rights. The stain of Abu Ghraib, as General David Petraeus later acknowledged, was ‘non-biodegradable’.

American efforts to counter jihadist propaganda had two basic flaws, says Bruce Riedel, a former CIA official now with the Brookings Institution. ‘Flaw number one was George Bush. You couldn’t possibly have a public diplomacy that could counter the negatives of Bush-Cheney diplomacy – invading Iraq, Abu Ghraib, Guantanamo Bay, torture, you name it. Secondly, we’re not Muslims, and when we speak to the Muslim world we don’t sound like we’re all that credible – we sound like we’re Crusaders.’ 

4.

One day in late 2004, as he browsed the website of an American embassy in Africa, David Kaplan discovered that the United States had got into the madrasa business. As a seasoned investigative journalist, he was intrigued. Since 9/11, madrasas (Muslim seminaries) had come to be seen as factories of jihad. And, in any case, given its strict constitutional separation of religion and state, why was the United States funding the construction of religious schools?

Kaplan’s chance discovery led him on a journey into the hidden world of America’s battle for Muslim hearts and minds. What he found out was that the Bush administration – now entering its second term – had embarked on the biggest and boldest campaign of ideological warfare since the Cold War. The campaign involved all the major arms of government – the State Department, the Pentagon, the National Security Council, the CIA, the US Agency for International Development – and a huge army of contractors and bit-players. It was costly, classified, and controversial. It was also chaotic. Who, if anyone, was co-ordinating this vast effort was unclear, and it was all too apparent that the endeavour was plagued by rivalries, turf wars, and red tape. Huge sums of money were being thrown at the ‘war of ideas’, with little to show for it.

In April 2005, after four months of research, Kaplan published his findings in a 12-page article for US News & World Report. Based on over a hundred interviews and an examination of a mass of official reports and documents, the article (with the apt title ‘Hearts, Minds, and Dollars’) was the first in-depth investigation of the American government’s labyrinthine efforts to win over Muslim opinion around the globe. Kaplan mapped the intricacies and frustrations of the ‘inter-agency process’, and tracked the startlingly wide geographical scope of government initiatives since 9/11 – the restoration of mosques and Islamic manuscripts, the building of schools (and, yes, three madrasas in the Ugandan capital Kampala), the quiet funding of Muslim think-tanks and NGOs fostering tolerance and human rights – and a host of other activities, covert and overt, in more than two-dozen countries.

Kaplan’s most controversial finding was that George Bush had committed the administration, through a presidential directive, to taking sides in what was deemed to be the civil war between ‘moderates’ and ‘radicals’ in the Muslim world. He had decided that, faced with an unprecedented threat from Islamist extremism, the United States had no choice but to intervene and, in effect, re-make Islam from within.

The new initiative was launched, under the innocuous title Muslim World Outreach, in July 2004. Spearheading it was Elliott Abrams, one of Condoleezza Rice’s deputies at the National Security Council and a prominent neo-conservative. Abrams wanted to wage ideological warfare with the same zeal that had produced victory in the Cold War. Muslim World Outreach involved a succession of ideas, meetings, and proposals designed to persuade all the relevant arms of government to play their part in implementing the president’s directive. By the time Kaplan completed his research he felt that, after the false starts of the preceding four years, the administration was at last on the right track.

But the optimism was premature. The initiative, like those that had gone before, ran into the sand. I asked Kaplan what had gone wrong. ‘Everything,’ he replied with a harsh laugh. For one thing, policy-makers were often uncomfortable dealing with religion. This was not normal policy terrain. Was it legitimate for the American government to try to influence one of the world’s great faith communities – indeed, to hasten it along the path towards Islamic reformation? Moreover the new enemy, radical Islam, was harder to grapple with than communism. How, for example, could one be sure who was a ‘good Muslim’ (to be supported and encouraged) and who was a ‘bad Muslim’ (to be weakened and marginalised)?

Besides, the rift between ideologues and pragmatists had not healed. Abrams and Feith recall meetings where they would stress the need to combat an ‘evil ideology’, while Powell and Armitage stressed the need to tackle Palestine and poverty. There was little meeting of minds.

5.

In Bush’s second term, the balance of power between the two camps began to shift. A key factor was the violence in Iraq, which reached such a pitch and so alarmed the American public that Bush was forced to change course. ‘The ideologues had had their day,’ David Newton, former US ambassador to Iraq, told me. ‘Their ideology had been implemented and the implementation had proved disastrous – and finally the pragmatists, the people who normally run events, were needed to try to get [us] out of this mess.’ The new men who took the helm and brought a much less ideological tone to policy-making included Robert Gates as Defence Secretary, Ryan Crocker as the new ambassador in Baghdad – and the new guru of counter-insurgency, General Petraeus.

To rewrite your war-fighting doctrine in the middle of a war that is going badly is no small undertaking. But that was the task Petraeus entrusted, in December 2006, to a team of experts led by one of his brightest protégés, Colonel John Nagl. The result – a new counter-insurgency manual, produced in record time – became a surprise bestseller. Whereas Rumsfeld had regarded armies as fighting machines and disdained anything that smacked of nation-building, Petraeus argued that America was fighting a new kind of war in which Muslim hearts and minds were the main prize.

The Petraeus doctrine was put to the test in the Iraqi capital Baghdad and the western province of Anbar. The timing was propitious. The Iraqi wing of Al-Qaida had discredited itself by its brutal violence against Iraqis as well as foreigners. Many of the Sunni tribesmen of Anbar who had allied themselves with the jihadists to fight the foreign invaders now switched sides in return for American dollars. The new approach helped weaken Al-Qaida and stabilise Iraq – buying time (so it was hoped) for the Iraqi politicians to set their house in order and pave the way for an eventual withdrawal of Western forces.

By the end of the Bush era, much of the groundwork had been laid for the Obama inheritance. While adhering to the term ‘war on terror’ – which he had unwisely attached to the post-9/11 campaign against Al-Qaida – Bush at least curbed some of its excesses. He ended waterboarding, agreed to put detained Al-Qaida suspects on trial, and announced his intention to shut down Guantanamo. Much of this was forced on a reluctant president by the courts and by the pressure of public opinion. But, as a result, there was a much greater degree of continuity between the Bush and Obama presidencies than at first appeared.

The Obama team, understandably, sought to play up the differences rather than the similarities. A buzz-word of the new administration was engagement – reaching out to the world to repair the damage done during the Bush years and stress America’s renewed commitment to multilateralism after the go-it-alone adventurism of Bush. Central to this was the idea of common interests. This became an Obama theme tune. Hence, in his Cairo speech, he argued that defeating Al-Qaida, bringing about Israeli-Palestinian peace, and reining in Iran were common concerns rather than self-interested American obsessions.

The administration created an infrastructure of global engagement, much of it with a Muslim focus, directed by the White House and the National Security Council. Officials promoted initiatives in science, technology, and business (such as the much-trumpeted Entrepreneurship Summit in Washington in the spring of 2010). But while the desire to build up networks of common interest was worthy, there could be no substitute for progress in the policy arena. The Cairo speech had contained its own check-list of commitments and on most of them (the Palestinian issue, Afghanistan, Guantanamo) progress was either slow or non-existent.  I asked Richard Armitage what he thought of the Cairo speech. It was a good speech, he said, but without follow-up it was just ‘pixie dust’.

6.

In November 2010, on the eve of Thanksgiving, I took the train down to Fort Bragg, North Carolina, one of the biggest military bases in the United States. A vast, sprawling compound with housing complexes, a sports stadium, an ice rink, petrol stations, supermarkets, hotels, and inter-denominational places of worship, the base is home to 50,000 soldiers.

Young men who had just returned from a year in Afghanistan told me what the Petraeus doctrine meant to them in practice. They were all familiar with the counter-insurgency manual produced four years earlier by Colonel Nagl and his team. But few had studied it in classrooms at Fort Bragg. They had learned the new arts of counter-insurgency the hard way, on the battlefield in Iraq, where some had fought during earlier assignments, or in 2009-10 in Afghanistan, when their units had suffered the highest casualties of any Western forces in the country.

Obama had decided to divest himself of one of Bush’s wars (Iraq) but plunge deeper into the other (Afghanistan). His fateful decision, at the end of 2009, to send an additional 30,000 troops to Afghanistan – chronicled in unremitting detail in Bob Woodward’s book, Obama’s Wars – weighed heavily on a president who was, at best, a reluctant warrior. But his defence and security policies pushed him closer and closer to Bush, to the unsuppressed glee of Republicans who saw this as vindication of their approach. Many liberals who had voted for Obama, on the other hand, watched aghast as he quadrupled the number of Predator drone attacks in Pakistan, extended the ‘war on terror’ to Yemen, and kept open Guantanamo and the controversial military commissions associated with it. There was an inescapable contradiction between the administration’s intentions and its actions.

7.

In the decade since 9/11, the United States has failed to win Muslim hearts and minds. Two administrations, beginning from very different starting-points, have been unable to produce a coherent strategy for the ‘war of ideas’. This raises two sets of questions. First, is this a failure of organisation or of understanding? Second, even if America has failed to win the ‘war of ideas’, is Al-Qaida nevertheless in the process of losing it? Is the Arab Spring – the wave of protest that broke out across the Middle East at the start of 2011 – final proof of Al-Qaida’s irrelevance?

There has certainly been a failure of what is known euphemistically in Washington as the ‘inter-agency process’. Time after time, initiatives have run into the sand because of red tape, turf wars, and the complexities and neuroses that bedevil any big bureaucracy. Above all, it has never been determined, once and for all, who should lead – and drive through – a complex, across-the-board effort to counter radical Islamist ideology. What is abundantly clear is that an under-secretary of state, even one with the ear of the president, has neither the clout nor the money to do the job effectively.

What exacerbated the bureaucratic gridlock, especially during the Bush years, was a fundamental rift over the root cause of extremism. For the pragmatists, it was a set of ill-conceived Western actions and policies which sparked a pervasive sense of Muslim grievance ripe for exploitation by the extremists. For the ideologues, it was a deep-rooted culture of hatred, intolerance, and incitement to violence – a culture tolerated for their own reasons by individual Muslim governments, and fuelled by petrodollars. This rift hobbled policy-making.

Inheriting these problems, the Obama administration, for all its good intentions, has failed to resolve the central issue of who should take charge of the effort. The State Department has continued to lack the means to do so effectively, and public diplomacy has continued to languish – despite Hillary Clinton’s advocacy of the merits of ‘smart power’ and intermittent pressure from Congress, think-tanks, and private-sector bodies for a new and more energetic approach. Some of the Obama team have engaged in fresh thinking about ‘knowing the enemy’, including the vexed question of whether the United States should, under certain conditions, engage with Islamists rather than shunning them. But policy success has remained elusive, and as time has gone by it is no longer possible for the Obama team to blame everything on their predecessors. Many of their mistakes are  their own.

Has Al-Qaida nevertheless lost the ‘war of ideas’? ‘It’s too soon to tell,’ says Armitage (quoting what Chou En-lai is supposed to have said when asked about the upheavals in Europe in 1968). It would certainly be ironic if, despite America’s failure to win the ideological war, Al-Qaida were to self-destruct. It is possible that, in time, it will. Its indiscriminate use of violence has alienated many of those who initially regarded Bin Laden as a hero. Two recent events have been taken as proof of Al-Qaida’s marginalisation. The first is the Arab Spring. The young protestors who have taken to the streets of Tunis, Cairo, and elsewhere have found – or thought they found – a new model for regime change, very different from Al-Qaida’s. The second is the killing of Bin Laden in an American raid on a compound in Abbottabad, Pakistan, on 2 May. This has deprived the global jihadist movement of its iconic figurehead. But it is too soon to write the movement’s obituary, however tempting it is to do so. Even if Al-Qaida has been weakened, its ideological influence persists. The idea lives on.

In confronting that idea, American policy-makers face an inescapable dilemma, regardless of which administration is in power. Like it or not, the United States is seen as the new imperial power. Americans are reluctant imperialists. They want to be regarded as liberators rather than as oppressors, even when their actions belie their words. Islamism feeds on the perception that a once-powerful Muslim world has been brought low by the strength, technology, and culture of an all-conquering American-led West. This view may be exaggerated. It may be ruthlessly exploited by demagogues and bigots. It may produce an unhealthy culture of victim-hood. But it persists, and without a sea-change in Western attitudes and Western policy it will retain enough truth to be persuasive. Changing the perception requires changing the reality.