The startup will be led by Musk, the CEO of Tesla and owner of Twitter, who has said on several occasions that the development of AI should be paused and that the sector needs regulation.
"Announcing formation of @xAI to understand reality," Musk said in a tweet on Wednesday.
The website said xAI will hold a Twitter Spaces event on July 14.
The team at xAI includes Igor Babuschkin, a former engineer at DeepMind, Tony Wu, who worked at Google, Christian Szegedy, who was also a research scientist at Google and Greg Yang, who was previously at Microsoft.
Musk in March registered a firm named X.AI Corp, incorporated in Nevada, according to a state filing.
The firm lists Musk as the sole director and Jared Birchall, the managing director of Musk's family office, as a secretary.
The billionaire had said in April that he would launch TruthGPT or a maximum truth-seeking AI to rival Google's Bard and Microsoft's Bing AI that tries to understand the nature of the universe.
Generative AI caught the limelight with OpenAI's launch of the popular chatbot ChatGPT, which came in November last year, ahead of the launch of Bard and Bing AI.
Dan Hendrycks, who will advise the xAI team, currently serves as the director of the Center for AI Safety and his work revolves around the risks of AI.
Musk's new company is separate from X Corp, but will work closely with Twitter, Tesla, and other companies, according to the website.
xAI said it is recruiting experienced engineers and researchers in the Bay Area.
Musk previously criticized Microsoft-backed OpenAI, the firm behind chatbot sensation ChatGPT, for "training the AI to lie." He said OpenAI has now become a "closed source," "for-profit" organization "closely allied with Microsoft."
He also accused Larry Page, co-founder of Google, of not taking AI safety seriously.
"I'm going to start something which I call 'TruthGPT,' or a maximum truth-seeking AI that tries to understand the nature of the universe," Musk said in an interview with Fox News Channel's Tucker Carlson.
He said TruthGPT "might be the best path to safety," that would be "unlikely to annihilate humans."
"It's simply starting late. But I will try to create a third option," Musk said.
The idea, Musk said, is that an AI that wants to understand humanity is less likely to destroy it.
Musk said he's worried that ChatGPT "is being trained to be politically correct."
More about: ElonMusk