Relaxing one evening last week at the Cuckoo's Cafe, a rooftop restaurant in the heart of the eastern Pakistani city of Lahore, Barack Obama's special envoy to Pakistan and Afghanistan seemed on the point of causing a major incident.

Afghanistan's President Hamid Karzai (R) shakes hand with U.S. Special Representative to Pakistan and Afghanistan Richard Holbrooke in Kabul February 15, 2009. REUTERS/Omar Sobhani (AFGHANISTAN) As ever in the region, there had been no warning. The weather was just right, a warm late winter evening. The view was even better - unmarred by the security subtly positioned on surrounding buildings. From his table, Richard Holbrooke, 67, the diplomat charged with calming what fellow members of the administration call the most dangerous place in the world, looked out over the giant Badshahi mosque and the imposing Lahore Fort, both more than 300 years old. Carefully invited politicians, writers, human rights activists and journalists from Lahore's liberal elite chatted at tables around him.

It was not that Holbrooke did not enjoy the barbecued spicy kebabs, Lahore's speciality, it was just he had one special request. He wanted daal, the plain lentil curry that is the humblest dish in South Asia. For such a distinguished guest, none had been prepared. "The bulldozer", credited with negotiating an end to the war in the Balkans in the 1990s, usually gets his way and this time was no exception. Daal was soon on its way.

Tonight Holbrooke will land at the Palam air force base, adjacent to the main civilian airport in New Delhi. It will be the last stop on a journey that has led the diplomat across the broad swathe of territory stretching from central Afghanistan to Pakistan's Indus river. Call it the central front of the global "war on terror", the fulcrum of the "arc of crisis", Pashtunistan or simply, in the most recent neologism, "AfPak", no one doubts that this is the biggest foreign policy headache for Obama's new team.

"The situation there grows more perilous every day," Admiral Mike Mullen, chairman of the American joint chiefs of staff, told journalists earlier this month. Holbrooke reaches for the ultimate comparison: "It's tougher than Iraq."

First, there is the local situation. Since launching an offensive in 2006 the shifting alliance of insurgents which make up the Taliban in Afghanistan have established control - or at least denied government authority - over a large part of southern and eastern Afghanistan. British foreign secretary David Miliband last week spoke of a "stalemate" - something senior generals and security officials have known for some time.

Local Afghan forces are still far from able to take on the insurgents without assistance from the 73,000 Nato troops now in country. The government is corrupt and ineffective. Opium production has exploded. Across the border in Pakistan, despite continuing military operations, authorities seem unable to push the Islamic militants on to the defensive. And somewhere in the mess is al-Qaida, though few can say exactly where.

Then, there is the regional situation. There is little love lost between Pakistan, India and Afghanistan. The two former countries have been at loggerheads since splitting in the aftermath of independence from Britain. Kabul's relationships with New Delhi are warm, a cause and consequence of their mutual animosity towards Islamabad.

"Both India and Pakistan would justify their involvement [in Afghanistan] as a deterrent against the other," said Chietigj Bajpaee, South Asia analyst for the Control Risks group.

Finally, there is the global situation. "AfPak", or more specifically the area dominated by the Pashtun tribes around the border mountains, has become the "grand central station" of global Islamic militancy, intelligence sources told the Observer. Young westerners head up to the tribal areas, the semi-autonomous zones which line the Pakistani side of the porous frontier, to visit makeshift al-Qaida training camps to learn how to blow up trains or planes back home. British intelligence track about 30 individuals of high risk through Pakistan each year. Others are known to be fighting with the Taliban against Nato troops.

It is this hideous puzzle that Holbrooke has been sent to sort out. If he can. "It is not too late. If they get the approach right and make an effort to really understand the problems, they can still do it," said Hekmat Karzai of the Centre for Conflict and Peace Studies, Kabul.

Holbrooke will not do it alone, however. Obama has assembled a powerful team of new and old faces entirely to revamp the American "AfPak" strategy. On a global level, Hillary Clinton, the new secretary of state, will take charge. Holbrooke will work on the region and the political track. On the military side, David Petraeus, the general credited with turning Iraq around, is now tasked with winning Afghanistan too. He has been clear that engaging with the largely Pashtun tribes, who bear the brunt of the fighting and provide most of the support for the insurgents, is an essential part of his strategy. As those tribes stretch across the border into Pakistan - a frontier which they cross more or less at will - Petraeus has focused on Afghanistan's neighbour too.

The complexity of the problems is forcing what UK diplomats call a "recalibration" of objectives. The Americans are more blunt. Defence secretary Robert Gates said the aim is not to build a "central Asian Valhalla". Creating a liberal, democratic and prosperous Afghanistan has been, at the very least, postponed.

"We have certainly pulled back from the aims of a nice, happy, Scandinavian-style democracy,' said Steve Cohen, at the Brookings Institution policy research centre, Washington.

The priority now is stabilisation. "There is a recognition that before... nation building, you have to clear the ground," said Seth Jones, of the US-based Rand Corporation thinktank. For Waliullah Rahmani, of the Centre for Strategic Studies, Kabul, "until Afghanistan is stabilised, you can't have good governance, development or democracy."

First stop on Holbrooke's "listen and learn tour" was Pakistan. As he travelled, the militants sowed death. In Peshawar, the Pakistani frontier city, last Wednesday a member of the provincial parliament was killed by a roadside bomb, the first elected politician to die in the current violence. The same day, Afghan Taliban launched an attack on government buildings in Kabul which involved eight suicide bombers and killed 28. The Afghan government blamed it on Islamabad's spies.

In Pakistan, those Holbrooke met were impressed by the envoy's apparent desire to hear what Pakistanis had to say. In Lahore, Jugnu Mohsin, a newspaper publisher, described how when told how Lahore was once known as a tolerant city where all religions thrived, Holbrooke, who backpacked through the region as a young man, wanted to know if it had become more conservative.

"He wanted to know about the Badshahi (mosque), who built it. He was interested in the culture and history of the place," said Mohsin. "He was basically there to learn, to inform himself, not to tell us what was what."

Others agreed, though pointed out that Holbrooke's open mind might have revealed a lack of detailed knowledge. "He is candid... and not given to the pro-India fixation of the Bush administration," said Ikram Sehgal, an analyst who briefed Holbrooke on the security concerns of Pakistani businessmen. "We've turned a real corner."

Washington has poured an estimated $1bn a year in military aid into Pakistan since 2001 and is worried that it is not getting value for money. There are also persistent question marks over the Pakistan security establishment's possible support of some Taliban elements.

Indians make frequent accusations. "We have no illusions in India that Pakistan is a major player in Afghanistan," says MK Bhadrakumar, a former Indian diplomat. "Pakistan estimates that at some point the US will withdraw ... [so] it can't let the Taliban go out of its hands."

Islamabad denies this, accusing New Delhi of joining with Kabul to foment violence amid separatists in Pakistan's south-west province of Balochistan and of spying from two consulates they have established along the border. "The Pakistanis have real concerns about Indian activities such as road construction or building the national parliament," said Jones of the Rand Corporation.

Holbrooke was taken on an aerial tour of the restive Pashtun tribal areas, flying by helicopter over Waziristan, the epicentre of militancy, to see the rugged and remote terrain. Yesterday, a missile fired from an American drone destroyed a house and at least 20 Taliban fighters in areas the envoy flew over, the latest in a series of highly controversial strikes.

Holbrooke stopped in the Khyber Pass, a key supply route for troops in Afghanistan and under attack in recent months, for a briefing with local commanders. Impressed, local observers pointed out that neither Pakistan's president, Asif Zardari, nor prime minister Yousaf Raza Gilani, have dared to do the same. Holbrooke had met both in Islamabad.

Then it was on to Lahore for meetings with former prime minister, Nawaz Sharif - who said Holbrooke had admitted that there had been "mistakes" in past US policy - and the rooftop dinner.

Then Holbrooke was on the move again to a frozen, snowy Kabul. The gritty, depressing, grey weather reflected the mood of the visit. Not only is it widely recognised that the Afghan project is in deep trouble but the Obama team believe President Hamid Karzai is at least in part responsible. Relations have deteriorated badly since the halcyon days when the Afghan tribal leader seemed the perfect man to lead his country. Obama himself is said to regard Karzai as unreliable and ineffective. Hillary Clinton has called his country a "narco-state".

Holbrooke arrived last Thursday and did not see the Afghan president until yesterday. Kabul was quiet - on account of the weather, power cuts and a national holiday celebrating the anniversary of the withdrawal of Soviet troops from the country 20 years ago.

Obama has long promised to put 30,000 more US troops into Afghanistan as part of a wide-ranging review of American policy and the first soldiers are expected to arrive before late spring. John Nagl, a senior fellow at the Centre for a New American Security, in Washington, believes the US commitment could eventually rise to 100,000 troops.

"The immediate problem is to stop the bleeding. The 30,000 troops is a tourniquet ... [but] that is all we have," he said. "If Obama is a two-term president then by the end of his time in office there may only be marine embassy guards in Iraq. But there will still be tens of thousands of US troops in Afghanistan."

There is also the matter of Afghanistan's coming elections, already postponed once. Some experts believe the polls might solve America's "Karzai problem". "Karzai will either improve his performance or he will be ex-president Karzai. That is the wonderful thing about elections," said Nagl.

Diplomats in European capitals fret about a weakened, re-elected Karazi with no real mandate. Sultan Ahmad Bahin, an Afghan government spokesman, said that Holbrooke had reassured the Afghan government of continuing American co-operation and of the new focus that Obama will bring.

Few locals showed much interest in the visit. "He's going to do what for us? These people just go backwards and forwards for nothing," said Karim, 34, a shopkeeper. "The Taliban have been killing us for seven years now."

For Bashir, a Kabul taxi driver, the Americans would leave. "The Soviets couldn't stay in our country. How can the Americans stay?" he asked.

A preoccupation for Obama and the Europeans is domestic public support for the war in Afghanistan. White House strategists believe it will hold up much better than the conflict in Iraq. "The polling has been very supportive. Iraq was a phony war but al-Qaida really is in Afghanistan and Pakistan," said Cohen.

That makes the job of persuading Americans that the war needs to be fought much easier. It is not hard to point out the genuine threats of a region where there are thousands of Islamic militants, nuclear weapons and where the 9/11 plot was hatched. "The main task will be to persuade the allies, especially the Europeans," said Cohen.

Finally on to New Delhi, where Holbrooke will step into a diplomatic atmosphere poisoned by November's Mumbai terrorist attacks. India holds Pakistan responsible for the three-day siege which left 179 people dead and many more injured. Relations with Islamabad are at their lowest ebb since the two nuclear-armed neighbours nearly went to war over Kashmir in 2001 and 2002.

Bajpaee, the analyst, argues that Holbrooke's best hope is to convince India to take a step back in Afghanistan to calm Pakistani concerns. Delhi may just be happy to let the US turn the screws on Islamabad. The Indians say they intend simply to "listen" to Holbrooke. The envoy too is going to be listening. The encounter may be much quieter than "the bulldozer" likes.

Divided Pashtun Nation

Which nation with homogenous ethnic make-up, a common language, religion and values is not a nation? The answer: Pashtunistan.

The Pashtuns, of whom there are now an estimated 40 million spread from south-western Afghanistan through to central Pakistan, (plus communities in cities such as Karachi and abroad in the UK), were divided on lines drawn by Sir Mortimer Durand in 1893, when he separated the British Indian Raj and the Kingdom of Afghanistan.

Throughout the 19th century the Pashtun tribes fought ferociously, following their honour code of revenge. In Afghanistan, they dominated the emerging state.

But it was not all war. Pashtun culture, particularly poetry and a famous love of flowers, also flourished.

In the post-colonial era, an educated elite campaigned for a nation state but with little popular support. In the past decade, Pashtun identity has fused with more global, radical Islamic strands. Experts, however, warn against branding current violence a 'Pashtun insurgency'.

The Pashtun world

• The world population of Pashtuns is estimated at 42 million, and they make up the majority of the population of modern-day Afghanistan.

• Pashtun tradition asserts they are descended from Afghana, grandson of King Saul of Israel, though most scholars believe it more likely they arose from an intermingling of ancient Aryans from the north or west with subsequent invaders.

• Pashtuns are predominantly Sunni Muslim.

• The largest population of Pashtuns is said to be in the Pakistani city of Karachi.

• Pashtun culture rests on "Pashtunwali", a legal and moral code that determines social order and responsibilities based on values such as honour (namuz), solidarity (nang), hospitality, mutual support, shame and revenge.

Reporting by Jason Burke in London, Yama Omid in Kabul, Paul Harris in Washington, Saeed Shah in Islamabad, and Gethin Chamberlain in Delhi.