Ms Pillay said forces sent to quash violence in the northern state were reported to be targeting Muslims.
The UN refugee agency (UNHCR) says about 80,000 people have been displaced following inter-communal violence.
The agency says most of those displaced are living in camps and more tents are being airlifted in to help them.
The latest violence in Rakhine state began in May when a Buddhist ethnic Rakhine woman was raped and murdered by three Muslims.
On 3 June, an unidentified mob killed 10 Muslims.
Ms Pillay’s office says that since then at least 78 people have been killed in ensuing violence but unofficial estimates are higher.
“We have been receiving a stream of reports from independent sources alleging discriminatory and arbitrary responses by security forces, and even their instigation of and involvement in clashes,” Ms Pillay, the UN High Commissioner for Human Rights, said.
“Reports indicate that the initial swift response of the authorities to the communal violence may have turned into a crackdown targeting Muslims, in particular members of the Rohingya community.”
She welcomed a government decision to allow a UN envoy access to Rakhine state next week, but said it was “no substitute for a fully-fledged independent investigation”.
‘Scared to return’
The UNHCR says that about 80,000 people had been displaced in and around the towns of Sittwe and Maungdaw.
Spokesman Andrej Mahecic said that many were too scared to return home while others were being prevented from earning a living.
“Some displaced Muslims tell UNHCR staff they would also like to go home to resume work, but fear for their safety,” he said.
Burmese opposition leader Aung San Suu Kyi recently called for laws to protect the rights of ethnic minority groups.
In her first statement in parliament, she said such laws were important for Burma to become a truly democratic nation of mutual respect.
Burma has undergone a series of political reforms initiated by the military-backed government.
But some parts of the country are still hit by conflict and unrest, most recently Rakhine state.