Ajax played well and had a Nicolas Tagliafico goal ruled out by the first Champions League video assistant referee consultation.
Holders Real took the lead when Karim Benzema blasted into the top corner after fine work by Vinicius Jr.
Hakim Ziyech levelled from 10 yards but Marco Asensio scored the winner from Dani Carvajal's cross.
The Dutch side had the game's best chances, with Dusan Tadic hitting the post in the first half and David Neres, who set up Ziyech's goal, shooting straight at Thibaut Courtois in a one-on-one chance.
Substitute Kasper Dolberg missed a big chance in injury time as he slipped while shooting in the box, with Courtois able to get a hand to it.
VAR introduced at wrong time for Ajax
The video assistant referee system has been brought into the Champions League for this season's knockout phases - and it got its first major usage at the Johan Cruyff Arena.
Ajax thought they had taken the lead when Tagliafico headed the ball in following a goalmouth scramble. Lasse Schone's corner found Matthijs de Ligt, whose header was scooped by Courtois into the air. Tagliafico headed the ball into the air over the keeper and Tadic and into the back of the net.
But after a consultation with the video assistant referee team, referee Damir Skomina went across to watch the incident again on his screen and disallowed the goal.
The official had decided Ajax forward Tadic was marginally offside and interfering with play - standing in front of Courtois at the exact moment Tagliafico got his header in. It was the correct decision technically, however without VAR the goal would have stood.
On the balance of play, Ajax had more than deserved Ziyech's equaliser, which came with his 30th shot of the season - more than any other player in the Champions League.
They failed to make the most of their 19 shots and now face a real task on Tuesday, 5 March at the Bernabeu.
This was Ajax's first Champions League knockout game in 13 years and with star midfielder Frenkie de Jong already agreeing to join Barcelona this summer, and teenage captain De Ligt among other star players expected to leave, they could have to wait a while before getting to this stage again unless they can overturn their deficit in Madrid.
BBC
More about: football
