While the actual motives of the decisionmakers involved cannot be known with certainty, there are several factors that almost certainly contributed to the choices made regarding the U.S. stance towards Iran.
- Iran's military presents a credible threat to the lives of U.S. servicemembers.
Although staggering differences in capacity exist between the militaries of each nation, the recent wars in Afghanistan and Iraq have taught the U.S. that they are not invincible, and in particular they are very bad at keeping servicemembers safe in territories where they are not welcomed by the local populace. There is a severe political cost at home for military adventurism, and both of these wars have been important issues for voters during elections. The decision to invade is ultimately up to civilians in the United States: Congress to declare war, and the President to give the order, so regardless of the tactical or strategic situation - military officers don't get to make the call.
- Iran is part of the Middle East, a region with an extremely delicate international relations landscape.
The United States has a vested interest in the stability of the Middle East and is already a controversial presence there. A full-scale invasion of Iran would create political and diplomatic turmoil in other, neighboring nations, who would see it as a message of "step out of line, we step on you."
- An invasion is a commitment to conquer.
Invasions are commitments of the utmost gravity. You're not just committing materiel for the opening, kinetic phase of the war, but also a permanent force to occupy and govern. This was the key strategic error the United States made in the second Iraq war, they conquered Iraq easily but could not then govern or even effectively and swiftly prop up a new government. The costs for invading even a nation that offers no credible military threat to you are far in excess of what most observers understand.
Yes, a nuclear-armed Iran is a problem for U.S. interests, but as long as there are other possibilities to resolve the matter (diplomacy being the preferred), it's a very hard sell to say that an invasion would be less of a problem.