They demanded the government intervene to ensure the barrier is demolished.
The US military said the wall would reduce sectarian violence and stop the movement of weapons and militants.
Many Iraqis reacted angrily in April when US troops began building a barrier around the Baghdad Sunni enclave of Adhamiya to prevent it being attacked by Shia militants.
Tuesday's demonstration saw many residents of Shula and Ghazaliya march between the two areas before gathering in front of the first completed section of the 2km (1.25 mile) wall.
Carrying banners reading "No to the dividing wall" and "The wall is US terrorism", the protesters issued a statement demanding that Iraqi authorities intervene.
"The wall is in accordance with al-Qaeda's plans," the statement said, adding that it would "separate family from family".
Hassan al-Tai, a leader of the Sunni Tai tribe, demanded the Iraqi government act against those "planting division and sectarianism amongst Iraqis".
"The wall is dividing small neighbourhoods and will lead to the partitioning of Iraq," he told the AFP news agency.
A Shia cleric at the protest, Abdul Baqir al-Subaihawi, insisted the wall would provide neither security nor stability.
"The government must maintain security in Baghdad rather than separate its neighbourhoods," he added.
© 2007 BBC