Most areas suitable for growing the oil crop are key habitats for primates, a study suggests.
The researchers, who are examining palm oil's possible effect on Africa's biodiversity, say consumers can help by choosing sustainably-grown palm oil.
This may mean paying more for food, cosmetics and cleaning products that contain the oil, or limiting their use.
"If we are concerned about the environment, we have to pay for it," said Serge Wich, professor of primate biology at Liverpool John Moores University, and leader of the study.
"In the products that we buy, the cost to the environment has to be incorporated."
What is palm oil?
Palm oil comes from the oil palm tree, which is native to West Africa. However, most palm oil is currently grown in Indonesia and Malaysia.
Environmentalists say the region's forests have paid the price, with native trees cut down to make way for palm trees.
Oil palm expansion is a major driver of deforestation, which in turn threatens wildlife, such as the critically endangered orangutan of Borneo.
However, the International Union for Conservation of Nature (IUCN) says replacing palm oil with other oil crops is not a solution, as these crops have lower yields and would take up more land.
They say to reduce the impact on biodiversity, palm oil needs to be produced more sustainably by avoiding deforestation and by reducing the use of palm oil in products other than food.
Palm oil is found in many supermarket products, including soaps and cosmetics. A huge amount is now also being used in biofuel.
Where is further expansion likely?
Many companies growing palm oil are looking to expand into Africa.
This is a worry for conservationists, as potential plantation sites are in areas of rich biodiversity.
They are particularly worried about Africa's primates. Nearly 200 primate species are found in Africa, many of which are already under threat.
Habitat destruction is one of the main reasons why all great apes are at the edge of extinction. The introduction of palm oil plantations to Africa is expected to accelerate the habitat loss.
The latest research is published in the journal Proceedings of the National Academy of Sciences (PNAS).
The study found that while oil palm cultivation represents an important source of income for many tropical countries, there are few opportunities for compromise by growing palm oil in areas that are of low importance for primate conservation.
"We found that such areas of compromise are very rare throughout the continent (0.13 million hectares), and that large-scale expansion of oil palm cultivation in Africa will have unavoidable, negative effects on primates," said the research team.
To put that figure into context, 53 million hectares of land will be needed by 2050 to grow palm oil in order to meet global demand.
Dr Giovanni Strona of the European Commission Joint Research Centre in Ispra, Italy, worked on the study.
He said primates are already in steep decline in Africa due to habitat loss and poaching.
"The main message is that, due to the large overlap between areas that are suitable to grow oil palm and areas that host many vulnerable primates, it will be extremely challenging to reconcile oil palm expansion and African primate conservation," he explained.
What can be done to tackle the problem?
The IUCN says effective policies are needed to stop the clearing of native tropical forests for new oil palm plantations.
In existing oil palm plantations, companies should manage their land to reduce impacts on biodiversity.
Consumers can also help by choosing products that use sustainable palm oil and cutting down on the amount of palm oil they buy.
More about: palmoil