As is widely reported now, Jill Stein's campaign is requesting recounts in two states (and likely soon a third). Ideally, the recounts will closely (or exactly) match the reported totals, and the country can move on, knowing that Trump did, in fact, win those states, even if it's by a very small margin.
It's also possible that the recounts will find some significant but non-systematic issues, such as a particular precinct's votes being accidentally recorded twice. It's feasible that there's enough of these to change the result in the state if every correction favors Clinton, but as long as the recount is done in time, the "what happens" is straight forward: The Democratic Electors in that state will cast their votes for Clinton, instead of the Republican ones for Trump.
However, I don't know what could happen if there's evidence of major, systematic problems that make it impossible to determine what the actual vote for a given state was. As a hypothetical example, what if it's discovered that every Sequoia voting machine in PA reported exactly 725 votes for Trump and 400 for Clinton, regardless of how many votes were actually cast? Or someone "misfiled" ballots in all-optical scan Michigan, and now a portion of the state's vote is just gone?
In other words, what if it becomes impossible to say who won a state? Obviously, if it's attributable to a human, they will be tried and punished according to their intent and culpability, but that won't get back the missing votes. Is there precedent for anything like this?