WhatsApp accuses the company of sending malware to roughly 1,400 mobile phones for the purposes of surveillance.
Users affected included journalists, human rights activists, political dissidents, and diplomats.
NSO Group, which makes software for surveillance, disputed the allegations.
In a court filing, WhatsApp said NSO Group "developed their malware in order to access messages and other communications after they were decrypted on target devices".
It said NSO Group created various WhatsApp accounts and caused the malicious code to be transmitted over the WhatsApp servers in April and May.
"We believe this attack targeted at least 100 members of civil society, which is an unmistakable pattern of abuse," WhatsApp said in a statement.
The affected users had numbers from several countries, including Bahrain, the United Arab Emirates and Mexico, according to the lawsuit.
WhatsApp said it is seeking a permanent injunction banning NSO from using its service.
The firm, which was acquired by Facebook in 2014, said it was the first time an encrypted messaging provider had taken legal action of this kind.
WhatsApp promotes itself as a "secure" communications app because messages are end-to-end encrypted. This means they should only be displayed in a legible form on the sender or recipient's device.
NSO Group said it would fight the allegations.
"In the strongest possible terms, we dispute today's allegations and will vigorously fight them," the company said in a statement to the BBC.
"The sole purpose of NSO is to provide technology to licensed government intelligence and law enforcement agencies to help them fight terrorism and serious crime."