AFP quoted a Malaysian maritime official as saying 13 were killed and another 13 were rescued by local fishermen.
The news agency reported that the passengers onboard were believed to be Indonesian migrants.
Other reports have suggested up to 100 could have been on the boat.
Officials said they have deployed ships and an aeroplane to search for survivors.
The local head of the Malaysian Maritime Enforcement Agency, Mohamad Aliyas Hamdan, told AFP: "We are not sure if the migrants were attempting to land in Malaysia or trying to leave Malaysia illegally."
Thousands of Indonesians work illegally in plantations and other industries in Malaysia. They often risk dangerous sea journeys to return home.
In June 2014 two overloaded boats carrying Indonesian migrants capsized in the same area, killing at least 15 people at that time.
South East Asia saw a migrant crisis earlier this year when Thai police clamped down on human traffickers. This prompted thousands of Rohingya Muslims fleeing Myanmar, and who were heading for Malaysia and Indonesia, to attempt dangerous crossings in rickety boats.
A number of mass graves have also been discovered in southern Thailand and northern Malaysia in camps where the migrants were being held by the traffickers.
More about:
