The US said its planes had halted the attack in Deir al-Zour when informed of the Syrian presence.
A spokesman for the US administration expressed "regret" for the "unintentional loss of life".
An emergency meeting of the United Nations Security Council, convened by Russia, is discussing the crisis.
The Russians earlier said the current ceasefire in Syria was in danger of collapse and the US would be to blame.
The cessation of hostilities does not include attacks by the US on IS or other jihadist groups.
The US Central Command statement said the coalition believed it was attacking positions of so-called Islamic State and the raids were "halted immediately when coalition officials were informed by Russian officials that it was possible the personnel and vehicles targeted were part of the Syrian military".
It said the "Combined Air Operations Center had earlier informed Russian counterparts of the upcoming strike".
It added: "Syria is a complex situation with various military forces and militias in close proximity, but coalition forces would not intentionally strike a known Syrian military unit. The coalition will review this strike and the circumstances surrounding it to see if any lessons can be learned."
Russia`s defence ministry earlier said that if the US air strikes did turn out to be an error, it would be because of Washington`s refusal to co-ordinate military action with Moscow.
Syria conflict: US air strikes `kill dozens of government troops`
18 September 2016
Read: 1108
The US-led coalition has admitted its planes carried out an attack in eastern Syria that the Russian army says killed at least 62 Syrian troops fighting IS.
