Populist parties have campaigned for a "no" vote and Mr Renzi has said he will resign if he loses.
Opinion polls suggest he faces defeat.
A "no" vote for Mr Renzi would follow the UK`s vote in June to leave the European Union, as well as the rise of the anti-immigrant Front National in France and populist parties elsewhere, along with Donald Trump`s unexpected victory in the US presidential election.
The "no" campaign in Italy has been spearheaded by the the anti-establishment Five Star Movement, led by Beppe Grillo. It wants a referendum on whether Italy should keep the euro.
The Five Star Movement and the anti-immigrant Northern League would receive a boost from the prime minister`s defeat.
Many voters are fed up with years of economic stagnation.
But the possibility of Mr Renzi, 41, falling from power has reignited concerns about financial stability in the eurozone`s third largest economy.
The reforms include reducing the power of the senate. Its members would be cut from 315 to 100 with most drawn from mayors and regional representatives.
Mr Renzi says the reforms would speed up the cumbersome law-making process in Italy, which has seen 60 governments since 1948.
Opponents say the proposals would concentrate too much power in the prime minister`s hands.
Some 50 million Italians have the right to vote in the referendum. An opinion poll in November gave the "no" vote a lead of at least five percentage points. But many Italians are thought to be still undecided.
If Mr Renzi does lose, it is still not entirely certain that he will be out of power.
President Sergio Mattarella could ask him to form a new government or appoint a technocratic prime minister to serve until elections due in 2018.
Voting began on Sunday at 07:00 (06:00 GMT) and ends at 23:00, with results expected early on Monday.
More about:
