The prize was given for his story Stickfighting Days – the judges said it presented a heroic culture that was “Homeric” in its scale and conception.
They described Olufemi Terry as a talent with an enormous future.
Terry was born in Sierra Leone, grew up in Nigeria, was a journalist in Somalia and Uganda, and now lives in Cape Town.
His book is about Raul, a boy who lives in a dump and uses sticks to fight with other boys.
The Caine prize, of £10,000 ($16,000), is given annually for a short story published in English by an African writer.
Terry, however, told the BBC he thought it was “unhelpful” to see writers from the continent as a distinct category.
“There is a danger in seeking authenticity in African writing,” he told the World Today programme.
However, he said he was glad to have won the prize, as it would help him get his first novel published.