The 89-year-old was discovered unresponsive at the maximum security facility shortly after being transferred there from a Florida jail.
A prison union official told the Associated Press news agency the death was being investigated as a homicide.
He was captured in California in 2011 after a 16-year manhunt.
The former leader of South Boston's Winter Hill gang inspired the film Black Mass featuring Johnny Depp, and The Departed, which won the Academy Award for best picture in 2007.
His death came on the same day that he was transferred to the Hazelton penitentiary in West Virginia, which houses 1,385 inmates, according to reports.
An inmate with mafia ties is under investigation for Bulger's killing, three sources briefed on the incident told the Boston Globe.
Bulger was severely beaten by one or more of his fellow inmates shortly after arriving at the prison, law enforcement sources told CBS.
Boston-based WFXT-TV, a Fox affiliate, reports that Bulger was killed hours after being admitted to the general inmate population.
The Bureau of Prisons declined to comment last week on why Bulger - who was serving a life sentence - was being moved from the Florida penitentiary.
But according to the Globe, Bulger was transferred to Florida in 2014 from another prison in Arizona after his relationship with a female psychologist who was counselling him attracted the attention of prison authorities.
The details of his death have not been announced, but a local union official for correctional workers at the prison told CBS that a "homicide" had occurred on Tuesday morning.
"This is the third homicide in seven months at our facility," Richard Heldreth said in a phone interview. "We are very understaffed, we are short about 40 officers."
A Department of Corrections statement on Tuesday confirmed his death in custody, and said the FBI had launched an investigation.
The statement made no mention of the cause of death, but said that "no staff or other inmates were injured, and at no time was the public in danger."
US Attorney for the Massachusetts district Andrew Lelling said that his office had been informed of Bulger's death, adding: "Our thoughts are with his victims and their families."
Who was James 'Whitey' Bulger?
Bulger was born in 1929 as one of six children in an Irish-American family.
Raised in Boston's Irish-Catholic South Boston neighbourhood, he became involved with The Shamrocks gang as a young man, starting off by stealing cars and soon escalating to robbing banks.
Bulger at a christening for the son of his hitman, in this undated handout photo
He was first arrested for delinquency as a young teenager, and eventually went on to mastermind a criminal empire of loan sharking, gambling, extortion, drug dealing and murder.
After being convicted of armed robbery and hijacking, he was sent to the infamous San Francisco prison-island Alcatraz.
He apparently liked Alcatraz so much, he visited and posed as a tourist - while he was on the run.
The moment fugitive Whitey Bulger brazenly posed as a tourist in Alcatraz... where he had once been...
Brazen holiday snaps of fugitive James 'Whitey' Bulger have emerged showing the former mob boss visiting his old haunt Alcatraz - just weeks into his 16-year run from the FBI.
During his criminal career, Bulger attempted to provide weapons to the paramilitary Irish Republican Army in Northern Ireland.
He once strangled two women with his bare hands, and on another occasion tortured a man for hours before using a machine gun to riddle his head with bullets.
Bulger is said to have disliked his nickname Whitey - which derived from his light blond hair - preferring to be known as Jimmy.
His underworld activities were enabled by a rogue FBI agent who tolerated his crimes so he would supply information on other gangsters.
The many faces of Whitey Bulger in an FBI wanted poster
Bulger's brother William was a powerful local Democratic politician who became president of the Massachusetts State Senate in 1978 and later the president of the University of Massachusetts.
The politician always denied any knowledge of his brother's crimes, but maintained that he loved his brother and could never betray him to law enforcement.
Bulger went on the run in 1995 after the FBI agent tipped him off to an impending indictment.
The mobster was arrested in 2011 in Santa Monica, California, where he had been hiding out with his girlfriend Catherine Greig, who remains incarcerated at a women's prison in Minnesota.
He was convicted in 2013 of a litany of federal crimes, including participating in 11 murders across the US throughout the 1970s and 80s.
The US government paid more than $20m in damages to his victims on the grounds he had committed his murders while under government supervision.
In a 2015 letter to a group of students who had written to him for a history project, Bulger appeared to express remorse for his past.
"My life was wasted and spent foolishly," he wrote.
More about: gangster