Frame challenge, mostly motivated by:
The U.S. was willing to use nuclear weapon against China to deter and not in response to an invasion, but this strikes me as weird because you would think there are international laws that would prevent such a rash action.
You are perhaps giving too much weight to "international law". Yes, there are legal agreements that nations agree to abide by, but in the last analysis, nations are fully sovereign until they are coerced by force of arms or by sanctions to change their behavior.
If this was not the case, little peccadilloes like the 2003 Iraq invasion (US), ongoing occupation of Palestine (Israel), Uyghur persecution (China), occupation of Diego Garcia (UK), annexation of Crimea (Russia) would never have happened.
Oh, but these are powerful nations, you say? I give you the Rohingya (Myanmar).
In the case of US, China, France, UK, Russia, international law, via the UN, is even less of a bother as these 5 have veto powers.
On to the actual circumstances.
After WW2, while it had a nuclear monopoly, the US was not averse to waving a nuclear bully stick to get others to stay in line. I am posting this 16 Nuclear Crises of the Cold War, because it seems to sum up a good deal of them, but it is quite possible the information is biased against the US (think Chomsky). Still, something to chew on.
The fact that the Soviet and Mao Communist system were evil and needed to be resisted? Very relevant, but that nuclear coercion still took place nevertheless.
Now, in 1958, the USA had just gotten out of the Korean War, a war of aggression freely started by North Korea in 1950. NK's efforts quickly collapsed and the war then was equally freely chosen to be continued by PRC on North Korean territory. This war was savage, often involved maltreatment of prisoners (by both sides, but largely a Communist activity). It was cease-fired for reasons of mutual exhaustion by both principals (i.e. USA and PRC) in 1953. The US learned that China had endless sources of soldiers and was not at all averse to wasting their lives (Chang and Halliday claim Mao used the war to conveniently get rid of undesirables).
This is the context you are looking at. At that point, the US still, as it would show during the Cuban crisis of 1962, very much in nuclear coercion mode whenever it felt it could get away with it.
Over time, nuclear war has become less and less acceptable, both for risk limitation reasons (i.e. not blowing up the whole planet), for reasons of military effectiveness (some studies made during the Vietnam War doubted the US could win it by nuking) and, hopefully ethics (or the desire not to look bad).
What "international law" had to say in 1958 probably was very secondary to the second and third consideration, military advantages and not looking too bad.
In 2021? "International law" would still probably be very secondary to the general revulsion first use would draw on the country doing it, unless, maybe, if they were at threat of annihilation.
Still, first use is a useful bit of ambiguity to leave in play, especially if you want to avoid nuclear wars happening. Take two nations, A and B, both being nuclear, but B has the edge in conventional forces. If A is coy about not renouncing first use, B has to factor in that attacking A may very well result in nuclear strikes. This can dissuade B from trying to defeat A conventionally and then ultimately causing A to launch. So both remain quiet.
For A and B, read NATO vs USSR during the Cold War in Western Europe. Or Pakistan and India nowadays. Best to stay away from "let's try a good old conventional war because first use is against international law".
Of course, given a long enough period of time, low-probability events will eventually happen and that's why nuclear weapons, even if they've so far managed to stop a conventional WW3 from happening, are a long term risk for mankind.
Finally, note that just because some in the US military presented the president with an option, the president didn't in fact take that choice, so your question does not, quite, have the moral significance you seem to imply.
(As far as what international law actually has to say, I'd be surprised if it didn't condemn first use under most conceivable scenarios and I defer to @meriton 's answer).