That Tony Blair regrets introducing the Freedom of Information Act has been known for some time. But the force with which he reproaches himself in his new autobiography for doing so is truly remarkable:

"Freedom of Information. Three harmless words. I look at those words as I write them, and feel like shaking my head till it drops off my shoulders. You idiot. You naive, foolish, irresponsible nincompoop. There is really no description of stupidity, no matter how vivid, that is adequate. I quake at the imbecility of it.

Once I appreciated the full enormity of the blunder, I used to say - more than a little unfairly - to any civil servant who would listen: Where was Sir Humphrey when I needed him? We had legislated in the first throes of power. How could you, knowing what you know have allowed us to do such a thing so utterly undermining of sensible government?"

Tony Blair, A Journey, Hutchinson, September 2010

What a distance Blair has travelled since proclaiming his belief in FOI at the Campaign for Freedom of Information’s 1996 Awards. Then he said:

"it is not some isolated constitutional reform that we are proposing with a Freedom of Information Act. It is a change that is absolutely fundamental to how we see politics developing in this country over the next few years…information is power and any government's attitude about sharing information with the people actually says a great deal about how it views power itself and how it views the relationship between itself and the people who elected it"

Presciently, he added:

"People often say to me today: everyone says this before they get into power, then, after they get into power you start to read the words of the government on the screen and they don't seem so silly after all."

Read the full text of his speech here and listen to him giving it here.

FOI’s setbacks under Blair

Mr Blair’s strong dislike of FOI helps to explain its troubled history. FOI had featured in Labour’s 1997 manifesto - the sixth successive time that the party had promised it to the electorate since the early 1970s.

Months after the 1997 election, the government published its well received FOI white paper. The proposals were produced by Dr David Clark, the Chancellor of Duchy of Lancaster, strongly backed by Lord Irvine, the Lord Chancellor, one of the most influential of Blair’s ministers.

Yet 7 months later, David Clark had lost his job and FOI had been placed under the notably less sympathetic wing of Jack Straw, the Home Secretary. The Home Office later produced a uniquely awful draft bill. There was a voluntary public interest test, which the Information Commissioner could not rule on. Safety information was protected from disclosure by no less than three broad exemptions. If no grounds could be found to block a request which had been received, a new exemption could rapidly be created without primary legislation. Most bizarre was the “right to pry and gag”. Authorities would be able to insist on knowing why someone wanted information - and to disclose it on condition they did not share it with anyone else.

Fortunately these and many other weaknesses were later removed. The final, much improved Act, was passed in November 2000.

But that was not the end of FOI’s troubles. Although Jack Straw and Derry Irvine (who later took over responsibility for FOI) both wanted to bring the Act into force for central government after 12-18 months, Tony Blair’s personal intervention ensured it was delayed for over 4 years. The right of access did not take effect until January 2005.

Just 18 months later, Mr Blair struck again. New regulations were proposed, drastically limiting the right of access. Requests can be refused if the cost of answering them exceeds certain limits, but only the time spent finding and extracting the information can be counted. The government proposed to also include the time officials spent thinking about the decision. Any complex request, or one raising new issues, would involve considerable 'thinking time' and so would be likely to be refused on cost grounds - freezing progress on openness. Another provision would have rationed the number of requests anyone could make within a 3 month period, limiting campaigners, and entire organisations such as the BBC, to perhaps one or two requests a quarter to any one authority.

Weeks later, yet another assault: a private member’s bill to remove Parliament from the Act’s scope. Although introduced by a backbench MP, it clearly had government support.

Instead of taking pride in its creation, the Blair administration was trying to smother its infant law.

Fortunately, both initiatives failed. The private member’s bill was killed off by overwhelming public and press hostility. The FOI restrictions were rejected by Blair’s successor, Gordon Brown.

Why the change?

What changed Tony Blair’s once ardent support for FOI? In his book, he suggests it was disillusion with the media’s use of the Act:

"The truth is that the FOI Act isn't used, for the most part, by 'the people'. It's used by journalists. For political leaders, it's like saying to someone who is hitting you over the head with a stick, 'Hey, try this instead', and handing them a mallet. The information is neither sought because the journalist is curious to know, nor given to bestow knowledge on 'the people'. It's used as a weapon."

But the prime minister had been resisting FOI long before any journalist’s request was made. It was apparent in his decision to delay the Act’s implementation for over 4 years.

Mr Blair offers another explanation, that FOI undermines the government’s ability to discuss issues “with a reasonable level of confidentiality”. This is an issue which the Information Commissioner and Tribunal have taken into account in their decisions. Where they believe disclosure would harm decision making or collective responsibility, that is treated as a substantial factor against disclosure.

But Mr Blair himself acknowledges that his concern about frankness:

"didn’t impact much in 2005. It was only later, far too late in the day, when the full folly of the legislation had become apparent, that I realised we had crossed a series of what should have been red lines, and strayed far beyond what it was sensible to disclose."

If Mr Blair’s rejection of FOI occurred long before the Act became law, what actually caused it?

Labour’s skeletons

The book provides a significant clue. In opposition, Mr Blair says, Labour exploited the sleaze allegations which had dogged the Tories. He now regrets doing so and says he should not have given the impression that a Labour government would be “purer than pure”.

Remarkably, he then links his regret at introducing FOI to the possibility that it could be used to reveal Labour’s own scandals. The passage is worth reading in full:

"What I failed to realise is that we would also have our skeletons rattling around the cupboard, and while they might be different, they would be just as repulsive. Moreover, I did not at that time see the full implications of the massive increase in transparency we were planning as part of our reforms to 'clean up politics'. For the first time, details of donors and the amounts given to political parties were going to be published. I completely missed the fact that though in Opposition millionaire donors were to be welcomed as a sign of respectability, in government they would very quickly be seen as buying influence. The Freedom of Information Act was then being debated in Cabinet Committee. It represented a quite extraordinary offer by a government to open itself and Parliament to scrutiny. Its consequences would be revolutionary; the power it handed to the tender mercy of the media was gigantic. We did it with care, but without foresight. Politicians are people and scandals will happen. There never was going to be a happy ending to that story, and sure enough there wasn't. The irony was that far from improving our reputation, we sullied it."

The message is explicit. “Scandals will happen” he says. The problem with FOI is that it can be used to expose them.

The book goes on to discuss two of Labour “scandals” from 1997: one a personal matter, involving Foreign Secretary Robin Cook’s affair with his then assistant, and the other the Ecclestone saga.

Bernie Ecclestone, the Formula One boss, had secretly donated £1 million to Labour just before the 1997 election. After the election he held private talks about a further donation. At the same time, he successfully lobbied Mr Blair to exempt motor racing from the proposed ban on tobacco advertising in sport. The affair led to allegations that Blair had lied to Parliament and to calls for his resignation. It was Labour’s - and Tony Blair’s - first scandal, deeply damaging to the new prime minister.

It was also an archetypal example of the kind of issue that FOI might help uncover. Years after the event, FOI disclosures demonstrated what had been denied at the time. Within hours of meeting Ecclestone, Mr Blair instructed the Health Secretary, Tessa Jowell, to look for ways of exempting Formula One from the ban. At the time Mr Blair said the decision was not taken till 2 or 3 weeks later.

The fact that these documents did not emerge till 2008, after Mr Blair had left office, is the direct result of his decision to delay the FOI’s implementation to 2005.

The Ecclestone affair blew up in November 1997 as David Clark was putting the finishing touches to his bold FOI white paper. Within months, the reform had had the momentum kicked out of it.

So it was not the use of the Act to seek information about the 2003 Iraq invasion that soured Tony Blair’s view of FOI. His hostility was evident long before then.

The memoirs suggest that Mr Blair’s objection to FOI is that it undermines candid discussions in government. But they also offer an alternative franker explanation. He and his government - like all prime ministers and all governments - sometimes behaved in ways they should not have done. He regrets giving the public a legal right to probe those shortcomings.

This piece is cross-posted from the Campaign for Freedom of Information blog.