Does their hesitation mean that Ukraine forces attacking within Russian borders would change the nature of the current war from Ukrainian self-defense to foreign aggression?
It depends on what you mean by "the nature of the current war".
From the UN Charter point of view, Article 51 gives Ukraine the right to self-defence in response to Russia's invasion of Ukraine. There are no rules that prevent Ukraine from attacking military targets in Russia. Relevant is Article 52 of Protocol I of the Geneva Convention which covers international armed conflicts. Section 2 states:
Attacks shall be limited strictly to military objectives. In so far as objects are concerned, military objectives are limited to those objects which by their nature, location, purpose or use make an effective contribution to military action and whose total or partial destruction, capture or neutralization, in the circumstances ruling at the time, offers a definite military advantage.
Russia might not see it that way though. The 2014 Military Doctrine of the Russian Federation contains the following provision:
The Russian Federation shall reserve the right to use nuclear weapons in response to the use of nuclear and other types of weapons of mass destruction against it and/or its allies, as well as in the event of aggression against the Russian Federation with the use of conventional weapons when the very existence of the state is in jeopardy.
The decision to use nuclear weapons shall be taken by the President of the Russian Federation.
In his September 21st announcement, President Putin said:
In the event of a threat to the territorial integrity of our country and to defend Russia and our people, we will certainly make use of all weapon systems available to us. This is not a bluff.
The above statement was made in the context of supposed nuclear threats by "leading NATO countries (...) against Russia". Nevertheless, attacks on Russian territory may be seen by Russia as an escalation by Russia and they may escalate in kind. Whether they will, or whether they would consider their territorial integrity to be under threat can only be answered by speculation.
Let's go to your title question:
Will Ukraine jeopardize its position as a country waging a war of self-defense if it attacks deep inside Russian territory?
It depends on who you'd ask. From an international law perspective it is permissible to attack military targets on Russian soil.
Russia may see it as an escalation. On the other hand, Russia does not see Ukraine's fight as self-defense. If anything, President Putin's September 21st speech makes it seem as if Russia is engaged in a pre-emptive war of self defense. Quoting again from his recent speech:
After the Kiev regime publicly refused to settle the issue of Donbass peacefully and went as far as to announce its ambition to possess nuclear weapons, it became clear that a new offensive in Donbass – there were two of them before – was inevitable, and that it would be inevitably followed by an attack on Russia’s Crimea, that is, on Russia.
In this connection, the decision to start a pre-emptive military operation was necessary and the only option. The main goal of this operation, which is to liberate the whole of Donbass, remains unaltered.