In addition to the valid points made in the other two answers, the pardon power is a natural extension of discretion that the President has in the criminal justice and military domains. It also reflects the changing nature of the criminal justice process which was very different when the President was given the pardon power than it is today.
The Pardon Power Is A Natural Extension Of Executive Branch Criminal Justice Discretion
In the common law legal tradition of which the United States is a part, the executive branch has always had broad discretion to determine in what manner criminal law will be enforced by law enforcement officers on the street prioritizing which criminal laws scarce resources need to be applied towards, and by prosecutors, deciding which crimes that have been investigated by law enforcement to charge and what compromises to reach in the criminal justice process.
Likewise, in circumstances where the judicial branch has sentenced more people to criminal punishments than the executive branch has the funding and capacity to carry out as judicially ordered, it has been a long standing understanding that the executive branch must use its discretion to figure out what subset of judicially authorized punishments (imposed in the first place only because the executive branch asked for them in any case), will be carried out.
The executive branch's power to permanently refrain from enforcing a criminal law (which is equivalent to a pardon), and to commute sentences already imposed for reasons of correctional resource scarcity, naturally flow into the full fledged pardon power. And, if the executive branch can do something through the back door, its legitimacy is enhanced by formally authorizing it to do the same thing through the front door.
The Pardon Power Is Pertinent To The Commander In Chief Role
While it is little used these days, the Founders, who spent their formative years fighting the American Revolution, followed by the recent experience of Shay's Rebellion under the Articles of Confederation, which had concluded just two years before, were very attuned to issues pertinent to invasions and insurrections.
Executive branch pardons and commutations were critical tools in bringing the more than four thousand people who engaged in Shay's rebellion including several hundred who were prosecuted criminally for their conduct in the uprising to a peaceful conclusion which re-integrated the rebellious territory into the United States on a peaceful and loyal basis. Of the eighteen ring leaders sentenced to death in connection with this rebellion, only two were actually executed, while pardons and commutations prevented most of the other death sentences from being carried out and creating martyrs whose deaths could have reignited the conflict.
This model of post-insurrection use of pardons to end a civil war and reintegrate the rebellious territories back into the United States was, in fact, used in precisely the same manner in the way of the U.S. Civil War during the Reconstruction period.
Also, keep in mind that the modern Geneva Convention concepts of the right of prisoners of war to be released upon the end of hostilities did not exist in 1789 when the pardon power was inserted into the Constitution. Many people who would have been classified as prisoners of war and automatically released upon the close of hostilities in modern wars would have been considered criminal offenders who could only obtain relief at the end of hostilities via a pardon at the end of the conflict in the 18th century legal understanding of the status of losers who were captures in insurgencies.
Thus, the pardon power, in addition to having an ordinary criminal justice role, also served an important role as Presidential authority that would allow a President acting as commander-in-chief to secure peace after leading a war. Congress can declare war, but the Senate can only approve a peace treaty if a President can negotiate one to present to it. Constitutionally, then, primary responsibility for peacemaking following a war is vested in the President and the pardon power is a natural tool that the President can used to carry out that responsibility.
The Evolution Of The Judicial Power
A third consideration, expanding on the notion in the other posts that the pardon power operates as a check or balance on the judicial branch's authority made in other posts, is that there was much more of a need to check unsound trial court criminal convictions at the time that the pardon power was established in the constitution, because appellate review of criminal convictions was very primitive.
It is also worth recalling that from 1789 to 1890, the first century under the United States Constitution, there were no direct appeals from criminal convictions in federal trial courts, and the scope of the review of criminal convictions afforded by a writ of habeas corpus, which was the only way to legally challenge a federal criminal conviction at the time, was much narrower than the scope of a modern appeal from a criminal verdict. Also, law school trained lawyers themselves didn't even exist until the 1870s, so the kind of expertise necessary to write appellate briefs in criminal cases was much more scarce at the time.
In part, a lack of a direct appeal made sense in this era because accurate verbatim transcripts to use as an appellate record on appeal were much more difficult and expensive to produce, and much less reliable, than their modern equivalents which form the basis for all modern direct appeals of criminal convictions. A trial court record would have been a clerk's summary of the proceedings in the style of meeting minutes, rather than a verbatim record of the relevant testimony, evidence and argument, so the pardon process which focused not just on the nature of the crime and the proceedings, but also on evidence of innocence unmoored from the procedural complexities of the actual criminal trial and supplemented by the post-conviction conduct of the criminal defendant, may have been a more workable way to review wrongful convictions under the circumstances.
Appealing a 19th century federal criminal conviction was something done primarily via a writ of habeas corpus which was more like a request for relief from an arbitration award in a civil case today (which is possible only for very narrow grounds going to the soundness of the process rather than to issues like mistakes of fact or law by the decision maker), and less like an true "appeal" of the trial court proceedings. Thus, the pardon power provided a critical safety valve to provide relief from flawed criminal convictions that was only institutionalized in a more formal matter within the judicial branch a century later.
Interestingly, since the judicial branch made it possible to appeal criminal convictions and even to collaterally attack them as well after losing a direct appeal in some circumstances, the rate at which executive branch officials have granted pardons has plummeted. This corroborates the notion that in the early Republic, a much larger share of cases where the pardon power was exercised amounted to plain old appellate review of trial court criminal convictions.