Harvard researchers presented their study of nearly 19,000 IVF cycles at the European Society of Human Reproduction and Embryology.
The findings contradict the idea that male fertility goes on forever.
In fact, whether conception is natural or assisted, sperm mutations and a decline in sperm count in older men are thought to reduce the chances of pregnancy.
Previous research has also shown that older sperm is more prone to genetic errors, and this has been linked to the development of autism and schizophrenia in children.
But the age of the woman still has a larger impact on fertility than the man's age.
In this study, scientists found that men aged 40-42 were linked with a 46% lower chance of having a baby by in vitro fertilisation (IVF) than men aged 30-35, when the female partner was under 30.
Thirty-five year old women had significantly more success with a male partner under 30 after one cycle of treatment, than with a man in his mid-30s.
Dr Laura Dodge, from Israel Deaconess Medical Centre and Harvard Medical School, said the reasons behind reduced male fertility in older age were not completely clear.
"While the effect of female age on fertility is overwhelmingly due to increased rates of chromosomal abnormality, the proposed mechanism in the effect of male age on pregnancy are more subtle."
Dr Dodge said: "In the absence of clear evidence of the mechanism, the best pre-conception advice we can offer is to maintain a healthy lifestyle."
Nick Macklon, professor of obstetrics and gynaecology at University of Southampton, said: "This may encourage male partners to get a move on.
"It also offers new insights into the dynamics between the man and woman - it's not just down to the age of the woman."
'Errors creep in'
Dr Raj Mathur, consultant gynaecologist and clinical lead for reproductive medicine at Manchester Fertility, said the issue of men's age and its impact in IVF needed to be researched further in a larger database of couples.
But he said: "We should start taking male age into account."
Women who used donor sperm were not included in the study of nearly 8,000 IVF couples.
Dr Gillian Lockwood, medical director, Midlands Fertility Services, said there were distinct differences between women's and men's fertility.
"Women are born with all their eggs and don't make any new ones, but men make new sperm every morning.
"That's where errors can creep in as a result of cell division," she said.
She said that there was more chance of the woman becoming pregnant if both partners were young and added: "But if they both wait until they are older, then that could be more problematic."
More about: #IVF
