Queen's prerogative powers
The Queen has the right to authorise the forming of a new government and the appointment of a Prime Minister. So, theoretically, she can and has the power to say 'no' to the request.
The Queen only has the power to authorise the formation of a new government and appoint a new Prime Minister. There aren't any special powers in the event of a hung parliament.
According to this Parliament's research briefing, it's part of the Queen's prerogative powers:
3 The Crown’s personal prerogative powers
There are three main prerogative powers recognised under the common law which still reside in the jurisdiction of the Crown.
Firstly, the appointment of a Prime Minister; the sovereign must appoint that person who is in the best position to receive the support of the majority in the House of Commons. However, this does not involve the sovereign in making a personal assessment of leading politicians since no major party could fight a general election without a recognised leader.
However, if after an election no one party has an absolute majority in the House (as in 1923, 1929 and February 1974) then the Queen will send for the leader of the party with the largest number of seats (as in 1929 and 1974) or with the next largest number of seats (as in January 1924). Alternatively, the sovereign would have to initiate discussions with and between the parties to discover, for example, whether a government could be formed by a politician who was not a party leader or whether a coalition government could be formed.
2017 General Election
That being said, it's very high unlikely that that would happen, especially when May likely has the support of the DUP and is able to form a workable majority.
As of declared results, the Conservative Party will have at least 318 seats and the DUP has 10 seats. That makes 328 seats in total and a simple majority is just 326. Thus, it's mathematically impossible for the Labour Party to reach a majority, even if all the other parties form a coalition with them.
What role does the Queen play?
The leader of the party that can tell the Queen they have a workable Commons majority is the one Her Majesty will authorise to form a government.
By convention, the Queen does not get involved in party politics, so there are no circumstances in which she would choose the prime minister.
There have been suggestions that she may not deliver the Queen's Speech in person if there's a question mark over whether it will get voted through.
This Mirror article – "'Can the Queen say no to Theresa May?' Prime Minister seeks permission to form government from monarch" mentions this too:
The simple fact is, the monarch can't refuse permission to form a government from a party leader if they have the required number of seats to make up a majority.
If May has managed to cobble together a coalition and goes to the palace to ask the Queen to form a government, Her Majesty can't exactly say no.
What happens when a minority party tries to force its way to form a government?
It's also worth noting that should a minority party try to form a government without having the votes in the House of Commons, it will likely fail to pass the Queen's Speech. This will then force the Prime Minister to resign.
The Queen's Speech is thought of as the first test of a minority or coalition government. If any amendments to the Queen's Speech are passed by the House of Commons, or if the vote on the speech itself is lost, the Prime Minister must resign.
Tory Prime Minister Stanley Baldwin's 1924 King's Speech was defeated by 72 votes - he resigned the next morning and Ramsay MacDonald formed the first Labour government...which itself only lasted nine months.
The incumbent Prime Minister also advises the Queen on who to appoint as the next PM.
Currently, when a Prime Minister resigns, he or she advises the Queen on whom she should appoint as the next Prime Minister. The established convention seems to be that the Monarch is not obliged to take the advice of the outgoing Prime Minister, and may take advice from other sources, although if the resignation of a Prime Minister follows a general election in which another party has won a single majority in the Commons, there will be in practice no question about who should become the new Prime Minister.
The Political and Constitutional Reform Committee of the UK Parliament published a Select Committee Publications on the "Lessons from the process of Government formation after the 2010 General Election". It describes in detail constitutional rules and the conventions and is worth reading.
Just a note, the government formation process is mainly based on precedent and there is no exact constitutional process, as mentioned in the committee publication linked above.
Government formation takes place within a constitutional framework which is largely unwritten and based on precedent.