Authorities in Bangladesh have started transferring thousands of Rohingya refugees to an isolated island despite calls from human rights organizations for a halt to the process.

The United Nations is concerned the refugees won't be allowed to make a free and informed decision about whether to transfer to the low-lying area in the Bay of Bengal.

Humanitarian workers said that some refugees had been coerced - despite government claims none would be forced to go.

Two said their names appeared on lists collated by government-picked local leaders without their knowledge - while aid workers said authorities threatened people to leave.

Police officers escorted the refugees in 11 buses from Ukhiya for the journey to Chittagong port and then on to Bhasan Char - a 52-square-kilometer island.

Authorities said the refugees were taken on seven boats with two more carrying supplies. Around a thousand have already been relocated, regional police director Anwar Hossain told Agence France-Presse.

The island can accommodate 100,000 people only. A million Rohingya Muslims who have fled oppression in their native Myanmar. They are now living in a network of congested camps in south-eastern Bangladesh.

According to Mohammad Shamsud Douza, the deputy Bangladesh government official overseeing the refugees, the transfer was voluntary.

Refugees International said the move was "nothing short of a dangerous mass detention of the Rohingya people in violation of international human rights obligations," Al-Jazeera reported.

The UN said it had been given "limited information" about the transfers and wasn't involved in preparations.

"We don't want to end up living an isolated prison-like life," Nurul Amin, one of the refugees, said. 

"They've taken us here by force," a 31-year-old man told Reuters.