Thousands had marched to the border at the start of a six-week protest, dubbed the Great March of Return.
The Israeli military said soldiers had opened fire after rioting.
UN Security Council members meeting in New York have called for an investigation into the violence.
Palestinians have pitched five camps near the border for the protest. They are demanding that refugees be allowed to return to homes that are now in Israel.
Palestinian President Mahmoud Abbas urged the UN Security Council to help provide "protection for our Palestinian people".
"I... place full responsibility on the Israeli authorities for the loss of the martyrs who were killed today," he said.
What happened at the border?
The Israel Defense Forces (IDF) said there were about 17,000 Palestinians in five locations near the border fence. It said it had "enforced a closed military zone" in the area surrounding Gaza.
Although most protesters stayed in the encampments, some groups of youths ignored organisers' calls to stay away from the fence and headed closer to Israeli positions.
Image copyrightAFPImage captionIsraeli soldiers maintained a close watch on the border protests
The IDF said troops were "firing towards the main instigators" to break up rioting that included petrol bombs and stones being thrown at the fence.
A spokesman said all those who were killed had been trying to breach or damage the border fence, the Jerusalem Post reports.
Israel said it had targeted sites of the Hamas militant group.
Israel deployed tanks and snipers. Witnesses said a drone had been used in at least one location to drop tear gas.
Will the protests lead to a military escalation?
The death toll from Friday's rally is the largest since the last Israeli war on Gaza in the summer of 2014. Since then Gaza has seen a long period of calm but difficult economic conditions and the Israeli blockade may be the final chapter in the four-year truce between Hamas and Israel.
Despite the call for peaceful demonstrations, the confrontations involving angry Palestinian youths were not surprising. Young men have been demonstrating near the border with Israel on numerous occasions, but this time Israel's response was exaggerated.
Tomorrow, the Palestinians will bury their dead and head back to the border with Israel to throw stones at the soldiers. The important question that remains is to what extent will peaceful demonstrations succeed in stopping an impending war, or will the protests lead to military escalation?
Hamas, the militant group that has controlled the Gaza Strip since 2007, does not recognise Israel's right to exist but last year said it was ready to accept an interim Palestinian state limited to Gaza and the West Bank.
Addressing protesters on Friday, Hamas senior political leader Ismail Haniyeh said "we will not concede a single inch of the land of Palestine".
He said: "There is no alternative to Palestine and no solution except to return."
Palestinian health officials said at least 400 people had been wounded by live ammunition. It said one of those killed was a 16-year-old boy.
Image copyrightREUTERSImage captionHamas leader Ismail Haniyeh reiterated that his organisation would not recognise the Israeli state
What have the Israelis said?
The Israeli military oversees a no-go zone along the Gaza border, citing security concerns, and has doubled its troop presence for the protest. It fears the protest could be an attempt at a mass breach of the border.
The Israeli foreign ministry said the protest was a "deliberate attempt to provoke a confrontation with Israel" and responsibility for any clashes lay "solely with Hamas and other participating Palestinian organisations".
BBC
More about: Gaza
