Mr Modi was invited to speak via video-link at a conference at the University of Pennsylvania this month.
He has been criticised for his handling of Hindu-Muslim riots in 2002.
Riots erupted after 60 Hindu pilgrims died in a train fire, which Muslims were blamed for igniting.
Many Western government broke off contact with Mr Modi’s government after the riots. The US continues to refuse Mr Modi permission to enter the country.
Mr Modi was invited to speak via video-link at an annual student-run conference on India at the Wharton School of the University of Pennsylvania later this month.
“Our team felt that the potential polarising reactions from sub-segments of the alumni base, student body, and our supporters, might put Mr Modi in a compromising position, which we would like to avoid at all costs, especially in the spirit of our conference’s purpose,” the Wharton India Economic Forum said in a statement quoted by the Press Trust of India news agency.
Reports said that a group of teachers and students had expressed outrage to learn that Mr Modi had been invited to deliver the keynote address at the meeting in Philadelphia later this month.
“This is the same politician who was refused a diplomatic visa by the United States State Department on 18 March 2005 on the ground that he, as chief minister, did nothing to prevent a series of orchestrated riots that targeted Muslims in Gujarat,” they said in a letter.
Mr Modi, who belongs to the Hindu nationalist Bharatiya Janata Party (BJP), has not reacted to the development.
Under his leadership, Gujarat has been turned into one of India’s economic powerhouses, but he is also seen as a divisive figure and accused of not doing enough to stop the riots.
Mr Modi has always denied any wrongdoing in connection with the violence, but has not expressed remorse or offered an apology.