Sadly, as the business of science has grown, more researchers have been competing for a limited amount of cash, reducing the incentives to do thorough, careful work. For instance, a recent effort—called the Reproducibility Project—selected 100 high-profile psychology studies and successfully reproduced the results of only 36 of them.

But scientists are known for thinking up creative solutions to pressing problems. When Swedish economist Anna Dreber heard about the Reproducibility Project, she dreamt up a futures market that might help predict which studies could be reproduced.

Even though science is a mostly fact-based profession, researchers depend on gut instinct to move the field forward. Dreber thought that a futures market could be a shortcut to harness the wisdom of the crowds to identify better solutions than what any individual scientist could come up with on their own.

Here’s how the market worked: Each of 92 scientists involved in the Reproducibility Project were given $100. They could use that to buy or sell stocks in 41 of the 100 studies that were being reproduced by the same group of scientists. To avoid conflicts of interest, scientists could not buy stock in the study they were trying to replicate.

At launch, each stock was worth $0.50. If the study was successfully reproduced, the scientist who bought the stock would be rewarded $1. If not, he or she would lose their money. As scientists traded their shares, the stock price for each study changed.

The results of the experiment led by Dreber have now been published in the Proceedings of the National Academy of Sciences. She and her colleagues found that the futures market was able to predict the outcome of 71% replications.

As a control, each scientist was surveyed about a study’s likelihood of replication. The survey results showed that individuals could only predict the outcome of 58% replications, which is only slightly better than chance. As with so many other things, putting money on the line focuses the mind.

Although a futures-driven success rate 71% is a lot better than 58%, it is not a perfect solution. So the realistic use of a futures market for science would be to filter out most bad studies and identify those with a greater likelihood of being replicated. And that, ultimately, is a useful pointer for scientists to direct their efforts to the areas that show the most promise and potential for significant breakthroughs.

More about: