His decision comes after French paper Le Monde claimed the former Namibia sprinter received a payment days before voting on the 2016 Olympics host city, allegations he denies.
The International Olympic Committee (IOC) is investigating the claims.
As head of the IOC evaluation commission, Fredericks was to lead an inspection visit to 2024 candidate cities Los Angeles and Paris.
The 49-year-old said he was stepping down so as not to be a distraction, adding: "It is important that the taskforce's mission is seen as free and fair with no outside influence."
The IAAF says Fredericks' place on the five-person taskforce has been taken by IAAF Athletes' Commission chairperson Rozle Prezelj.
Russia's athletes were banned last year following the McLaren report, which outlined a state-sponsored doping programme.
More about:
