The rebels said that a nationwide curfew was in force and that the constitution had been suspended.
The renegade troops had staged a mutiny on Wednesday, trading gunfire with soldiers loyal to the government.
The mutineers say the government is not giving them enough arms to tackle a rebellion by ethnic Tuaregs.
On Wednesday the mutineers had taken over the state radio and TV broadcaster in Bamako and took it off air.
After several hours of footage of traditional Malian music and dancing, a group of soldiers appeared on screen early on Thursday morning, with a caption identifying them as the “Committee for the Re-establishment of Democracy and the Restoration of the State”.
A spokesperson for the rebels, identified on screen as Lt Amadou Konare, said they had ended the “incompetent regime” of President Amadou Toumani Toure.
Lt Konare condemned the “inability” of President Toure’s government to “fight terrorism”, and said the soldiers would look to hand over to a democratically elected government.
There has not yet been any reaction from President Toure to the announcement.
The unrest began on Wednesday as the country’s defence minister started a tour of military barracks north of the capital.
Soldiers fired in the air during the inspection, prompting an immediate strengthening of security around the presidential palace.
Troops are upset with the government’s handling of a Tuareg rebellion in the north of the country, and are also reportedly opposed to any potential talks with the rebels.
There was heavy gunfire in Bamako throughout Wednesday, and armoured vehicles had moved in to protect the presidential palace. Gunshots reportedly continued to ring out overnight.
A member of the presidential guard described the fighting to AFP news agency.
“We are in control of the presidential palace. People are shooting towards us and we are returning fire,” he said.
The BBC’s West Africa correspondent Thomas Fessy says it is unclear whether President Amadou Toumani Toure was inside the palace at the time.
In the northern town of Gao, young recruits were said to have begun rioting at a military base, according to the Associated Press.
Both the US and France have urged the soldiers and government to resolve their dispute through peaceful means.
The Tuaregs have forced the army out of several northern towns in recent months.
Earlier this week Ecowas, the West African regional organisation, urged its member states to support Mali with military equipment and logistics.
A presidential election was due to take place in the country in just under a month.
The government had so far refused to postpone the poll, despite the unrest involving Tuareg-led rebels.