The BBC’s Mohammed Olad Hassan says this is only the second public demonstration against al-Shabab, which controls much of southern Somalia.
The protesters shouted their support for the UN-backed government.
Mohyadin Hassan Afrah, who helped organise the protest in one of Mogadishu’s few government-controlled districts, says people were upset at a move by al-Shabab to destroy the tombs of revered Sufi clerics.
Al-Shabab follows the strict Saudi Arabian-inspired Wahhabi interpretation of Islam, rather than the Sufi Islam of many Somalis. "We call for a holy war against them," said Sheikh Somow, from the Sufi Ahlu Sunnah Wal Jama group, which recently stuck a deal with the government.
Mr Afrah also said he was marching to protest at al-Shabab’s use of foreign fighters. Our reporter says fighters from Pakistan, Yemen and North Africa have travelled to Somalia to join al-Shabab.
Dozens of government troops watched the march and fired shots into the sky. Some of the demonstrators carried posters with slogan such as "Down with al-Shabab" and carried "Support Peace and Government".
"We have been forced out of our houses because of the violence instigated by al-Shabab. We are here to support the government and make our voices against them heard," said one of the marchers, Hawo Abdulle Aden.
About half of Mogadishu’s population have fled their homes.
The country has been torn by conflict since 1991.
Source: BBC