The Edinburgh of Scotland’s late summer is awash not just with rainstorms but a plethora of festivals and happenings: the International Festival, the Fringe, the Book Festival, Television Festival, and even a Festival of Politics in the Scottish Parliament.

If all this sounds like an expression of the Scottish ‘democratic intellect’ or a modern day ‘Enlightenment city’, while conversations, deliberations and cultural happenings cover a multitude of concerns, there is usually an absence of connection to the host city and anything seriously Scottish.

This year the Book Festival has tried to overcome some of this with a range of packed political discussions – international, British and Scottish; Gordon Brown on social justice; considerations on Scottish independence; and a provocative international writers’ conference. One such discussion before a sold out Saturday audience was myself and The Guardian’s Polly Toynbee examining what future there is for British Labour.

It was a genuine conversation and exchange, which established real common ground and difference; it was an occasion where Polly Toynbee, an influential figure on the British centre-left who is close to senior Labour circles, not only engaged in a serious way with views different to her own, but also surprised many, certainly myself, with some of her perspectives in the direction she articulated for the official opposition.

Toynbee outlined a prospectus for Labour under Ed Miliband declaring that Labour should be “the party of conservatism”, of “preserving and restoring”, arguing that we shouldn't be obsessing with the appeal of “the new”. Instead, she said “what Labour has to do is blindingly obvious” against what she called the most “repugnant, malicious government” in British history – which surely has to count as a colossal overstatement (what about the Chamberlain government for starters?).

“Labour only needs to remember the good it did in office”, Toynbee announced, reading at one point a very long, detailed list of Labour achievements under Blair and Brown, while at the same time only awarding them an overall six of ten. She then stated that what Labour has “not got to do is to have a serious intellectual argument” about what it stands for, its politics, values or vision.

Taking a different standpoint, I stressed the importance at the outset of taking a long view of how New Labour had originated; that it hadn’t appeared out of a vacuum, or as some on the left postulated as the result of an elite coup d’état (with the assumption that the party body politic was all fine, radical and healthily social democratic). There was a crucial need to see New Labour in all its complexities including its undoubted achievements in office, now made harder by the actions of Blair himself post-office; and at the same time, acknowledge the ultimate car crash and damage New Labour had done, trashing social democratic language, values and credentials.

There have been thirty years of post-war Labour Governments in four distinct periods: 1945, 1964, 1974 and 1997. This is a substantial enough period for us to be able to assess and pass judgement on Labour’s reforming and radical credentials. Over this timespan Labour in office has done much many of us are proud of, changing Britain in lots of ways for the better, but the omissions, caution and wrong steps have been hugely significant and damaging.

Four periods of Labour in office and nine popular mandates since 1945 and the Labour Party hasn’t challenged the power nexuses and networks of establishment Britain or touched the institutions of ‘the Conservative nation’, which give succour and support to privilege and status. Nor has it, with the exception of the Attlee administration, substantially reduced inequality in any of its periods of office.

Then there is Labour’s lack of understanding of the British state, its inability to embrace a democratic politics of liberty, pluralism and constitutionalism which breaks with Westminster absolutism. It didn’t happen in 1997 when the opportunity and support was there, thanks to eighteen years of Tory Government, and it shows no signs of happening under Ed Miliband.

Polly Toynbee concluded her talk, before the question period, by making a special plea to the Edinburgh audience. “Don’t leave us” she said. “We need you to be there”, portraying a permanently Tory England as one of the consequences of Scottish independence.

In response to this I asked Polly not to fall into the trap of believing the myth of a Tory England or a conservative England: a story which New Labour peddled and which entrapped it, and many of the left have consistently fallen for (Eric Hobsbawm being a good example). As a point of fact, the Tories have only ever won a majority of the English vote once – 1955 – the same year they won a majority of the Scottish and Northern Irish vote (only Wales proving immune to the Tory charm!).

In response to Polly’s politics of conservatism, I argued that this misread how Labour needed to change while also playing into a politics which many in Labour and the left have, of nostalgia, of living in the past, believing in a golden era pre-Thatcherism/New Labour, and at worst, thinking you could turn back the clock to simpler, better times.

“Polly’s prognosis”, as I called it, gave the future and the politics of change to the free marketeers and market fundamentalists who would need no encouragement in seizing it as the recent Tory postulating from some of the 2010 parliamentary intake of Britain as a nation of “idlers”, the work shy, lazy and unproductive indicated.

Instead, a radical centre-left politics has to be about change, challenging the status quo and vested interests, and should never align itself with establishment forces anywhere, whether in the public sector, public realm or corporate capitalism. A left politics which didn’t have the confidence to be about the future, would become stuck in timidity, defensiveness and constant retreat – having to time and again cede political territory to its opponents.

Toynbee and myself talked about the politics, alliances and values needed to change a Britain scarred by inequality, insecurity and a lack of fairness and justice. She talked about the need for a “politics which starts at a human level” and which is about “feelings and emotions”. We acknowledged common ground on this and more, reflecting on how hope and anger could find collective voice.

Both of us had some optimism about Ed Miliband trying to distance himself and the party from the legacy and limits of New Labour. Toynbee was more prepared to invest in Miliband than myself with her call for “restoration” and “conservatism” which seemed to suggest she believed he could pose as ‘continuity Labour’, the inheritor of the tradition of social democratic Britain 1945-75.

I argued that Labour’s journey to New Labour had been a long one of hollowing out social democracy over many decades; and that to turn a new page Labour needed dramatic action and something of the scale of a national Truth and Reconciliation Commission on the damage done to our politics, democracy and public life.

It was only one debate, but both Polly and myself felt that we - at least as genuinely as we could - engaged with huge, challenging issues about the state of politics and democracy for which there are no easy, sloganeering answers. This seemed an appropriate response to the times we live in and the multiple crises which British, Scottish and Western societies find themselves in.

This was an honest, respectful conversation, listening and exchanging serious ideas and considered positions, and one that both of us reflected on afterwards had been worthwhile. This was an opening and hopeful dialogue, but we have to wonder whether Westminster Labour with all its pressures, siren calls and the narrow bandwidth of British politics, will have any real prospect of changing the political weather, and posing daring, difficult and hopeful questions for the age we live in.

In that sense, I cant help thinking that Polly Toynbee has done some of us a favour, by laying out so candidly, the limited, cautious politics which will be on offer from Ed Miliband and his team: continuity, minimal Labour, at best, a brief interlude in the storm and problems engulfing the global economy and society.