He called for Mr Trump to unite the party and do more to adhere to conservative principles.
An hour later, the New York businessman responded by saying: "I am not ready to support Speaker Ryan`s agenda."
"Perhaps in the future we can work together," he added.
Mr Trump said he hoped he and Mr Ryan could come to an agreement about what was best for the American people.
"They have been treated so badly for so long that it is about time for politicians to put them first!"
There has been persistent speculation that Mr Ryan, the 2012 vice-presidential nominee alongside Mitt Romney, may make a last-minute run for the top job.
But he has frequently sought to dispel those claims and did so again on Thursday.
In explaining his refusal to back Mr Trump, he said: "I think what a lot of Republicans want to see is that we have a standard bearer that bears our standards."
Analysis - Katty Kay, Washington
Paul Ryan is giving Donald Trump a lesson here - you have to unify the party, you`ve got to tone it down - and inviting him to do so in whatever way he can.
It`s like a carrot-and-stick move from Paul Ryan.
He`s hoping Trump will listen to this and will realise that if he wants to bring the bulk of the Republican party - and Paul Ryan definitely represents quite a lot of Republicans - on board with his candidacy and give him the chance to win the election in November, he will have to do something about his tone and some of the more inflammatory things he has said in this campaign.
In December 2015, Mr Ryan harshly criticised Mr Trump`s proposal to ban Muslims from entering the US.
He said it was "not what this party stands for and more importantly it`s not what this country stands for".
Although several senior Republicans like Mr Romney have already voiced their objection to the nominee, Mr Ryan is the highest ranked to do so.
Former presidents George W Bush and George H W Bush said on Thursday they would not endorse the controversial presumptive nominee.
Both Bushs, plus former nominees Mr Romney and Senator John McCain, have said they will not go to the party convention in July when Mr Trump will be formally confirmed.
Protests have plagued appearances by Mr Trump, with particular focus on his plan to build a wall on the Mexican border and deport 11 million undocumented immigrants.
More about:
