Keeping the Tories united: The battles without and within

The Conservatives were heading into conference season in as good a mood as they could possibly be in. David Cameron had, by hook or by crook, managed to get his party into a fighting mood, and he and his MPs were keen to do everything within their power to at least put up the best fight they could against Ed Miliband.

Then Douglas Carswell defected to Ukip, and it all changed. This has left Tories scratching their heads, wondering whether being sent back into opposition is now inevitable, and whether they shouldn't start looking for jobs to keep them busy when their party is out of government and no one wants to hear from them. Carswell's defection opened up the split in the political right, and rather than just a repair job, that split now needs a big steel bridge to span it.

Cameron might have been comforted by the sight of Eurosceptics lining up to condemn Carswell's decision to defect, and he might be encouraged by the fact that the rest of his party seems keen to remain loyal. Even troublesome backbenchers such as Andrew Bridgen have fallen into line, saying that they want to fight Labour, not their party colleagues. Other big beasts whose remarks have previously stirred up talk of a leadership contest, such as David Davis, struggle to get the same hearing these days, having lost many of their supporters.

The summer reshuffle, which saw controversial ministers packing their bags so that the Tories can try to sell their reforms and soothe rows over education and planning reform, demonstrated that the party doesn't care about policy any more: it wants to fight a general election. A number of ministers are now ready to be deployed on the airwaves as part of a slicker and more controlled broadcast operation. Grant Shapps is assembling an army of activists keen to knock on doors and persuade voters to back the Tories.

All this activity was impressive. But the problem is that it may have lulled David Cameron into a false sense of security about his standing in the party. His MPs are bunching together not because they have suddenly discoveredthat they love their party leader. Rather, they have suddenly realised that if they don't put up a jolly good fight, they could risk the humiliating spectacle of being beaten by Ed Miliband, who they don't believe deserves to win.

Cameron might be quite happy to accept this rather superficial loyalty for the time being, just as charities are happy to take donations from attention-seeking celebrities. But he should beware that it will evaporate once the polls have closed. It is a given that if the Tories lose next year, their current leader is off. If they secure a majority, then the grumblers in his party will have to concede that Cameron has pulled off something truly impressive. But if the Tories return as merely the largest party, Cameron faces all manner of troubles.

Time to go it alone? Attitudes to coalition and Europe

The prime minister knows that while his instinct is for another coalition with the Liberal Democrats, his party would prefer a minority. With today's MPs, though, a minority would be difficult. Cameron would not just be trying to patch together deals with other parties to pass legislation: he'd need to patch together deals with his own independently-minded backbenchers. This process would be exhausting for a man who views maintaining relations with his MPs as a necessary but unpleasant task, a bit like cleaning the bath.

There are high-flying ministers, too, who like Cameron but don't want another coalition. Many of them are fed up of the day-to-day grind of working with the Lib Dems; some have had their pet ideas blocked. Some of these ministers would prefer to vote against another partnership if Tory MPs are offered a secret ballot on whether to approve a coalition.

Assuming that Cameron manages to survive next summer, he will then start renegotiating Britain's relationship with Europe. And so around this time, influential backbenchers will start jostling to lead the 'out' campaign within parliament. Liam Fox and Owen Paterson are two prime candidates. Both were deeply hurt by their treatment in the recent reshuffle: Fox was offered a ministerial position below his capabilities, and Paterson was sacked from the job he loved at Defra. Both are fierce Eurosceptics, and appeal to a certain branch of the Conservative party that Cameron has always struggled with.

Whoever does decide to lead the 'out' camp will create an alternative centre of gravity within the Conservative party, which will destabilise it. Beyond the intellectual arguments about the EU and British sovereignty, it is this potential to destabilise the Tories that appeals to those who feel that Cameron will never be a really good party leader.

Two Tory tribes: optimists and pragmatists

As for most backbenchers, they presently fall into two camps: the optimists and the pragmatists. The optimists are MPs who see reasonably positive newspaper coverage, and a small national poll lead for Labour, and scent victory. They're not necessarily die-hard Cameroons: many of the optimists are former rebels who think their party is simply better than Labour and deserves to win. Then there are the pragmatists – often City types who specialise in numbers and disregard feelings. They look at the odds offered by bookies for the various outcomes next year, and pore over Lord Ashcroft's detailed polling of marginal seats, and conclude that yes – their party is winning the air war, but in the seats that matter it's looking like Labour is going to romp home as at least the largest party.

The pragmatists have been growing in number since Carswell's defection. They see a groundswell for Ukip that will affect more than just Clacton: the party that some commentators have written off at least three times in the past year has come back stronger than ever.

For Cameron, the most important thing is to try to keep the optimists feeling cheerful about their party's prospects right up to polling day. He doesn't want too many converts to pragmatism in the party over the next few months: wild optimists are much more useful.

To do this, he will need to show that he has a plan to reunite the right, and that he does sincerely want radical reform of the EU. To stand any chance of being believed by voters or backbenchers, though, the prime minister must first sincerely believe in both of these things himself. Therein lies his greatest challenge.

Isabel Hardman is assistant editor of the Spectator and edits the magazine's Coffee House blog. She also writes a weekly column for the Daily Telegraph.

This article appears in edition 21.2 of Juncture, IPPR's quarterly journal of politics and ideas.