Former Iraqi leader Saddam Hussein made billions of dollars smuggling oil in defiance of sanctions policed by the US and Britain, the UN chief said.
Mr Annan was recently criticized over his son's work with the program
A UK government minister told the BBC that an interim report had criticized the UN, not national governments.
The $60bn (£32bn) UN program allowed Iraq to sell oil in order to buy civilian goods and ease the impact of UN sanctions.
US Senate investigators have alleged that the Iraqi regime received some $4bn (£2.13bn) in illegal payments from oil companies involved in the program
The BBC's Michael Voss in New York says this figure is dwarfed by the $14bn (£7.5bn) that allegedly came from "sanctions-busting" - illegally selling oil to neighboring states such as Jordan and Turkey.
"The bulk of the money that Saddam [Hussein] made came out of smuggling outside the oil-for-food program, and it was on the American and British watch," Mr Annan said.
"Possibly they were the ones who knew exactly what was going on, and that the countries themselves decided to close their eyes to smuggling to Turkey and Jordan because they were allies."
Oil shipments were openly sent from Iraq to Jordan and Turkey during the 1990s and were not intercepted, despite the US maintaining forces in the Gulf area.
The overland route from Iraq to Turkey was a very busy oil route, very clearly officially sanctioned by Turkey, says the BBC's Jonny Dymond in Istanbul.
It is difficult to believe that the large US and UK embassies in Turkey would not have known that a large quantity of Iraqi oil was being smuggled across the border, our correspondent adds.
Mr Annan partly excused the smuggling to Jordan and Turkey, accepting that countries not under sanctions had a right to be compensated for any loss of trading income.
Allegations of kickbacks and illegal trading under the oil-for-food program are being investigated by an independent committee headed by former US Federal Reserve chairman Paul Volcker.
In two interim reports, the committee has questioned the ethical conduct of the program's head, Benon Sevan, and the involvement of Mr Annan's son, Kojo Annan, in a company linked to the oil-for-food deals.
UK Foreign Office Minister Bill Rammell said the UK took "vigorous actions" to ensure that the sanctions regime was not undermined.
Pointing to the criticism of the UN in the two interim reports, he added: "I think the UN has to learn those lessons.
"I know Kofi Annan wants to do that and we want to work with him to achieve it."
On Thursday a Texan, a Briton and a Bulgarian were indicted for bribery.
Texan David Chalmers Jr, Bulgarian Ludmil Dionissiev and British oil trader John Irving are accused of paying bribes to Saddam Hussein's regime.
© 2005 BBC News