The Institute for Government has published an explainer on this, the most relevant part of which is below, and sets out the two parties' current negotiating stance:
UK–EU trade disputes
If the UK government concludes a comprehensive free trade agreement
(FTA) with the EU within the transition period, that agreement will
set out a way to resolve trade disputes. The EU’s negotiating
directives propose a system similar to that under the Withdrawal
Agreement, with independent arbitration panels. As under the
Withdrawal Agreement, issues of EU law must be referred to the ECJ.
The UK’s mandate is less detailed on the subject, but proposes that
governance arrangements including, "if necessary, dispute resolution’
should be ‘appropriate to a relationship of sovereign equals, drawn
from existing Free Trade Agreements, such as those the EU has with
Japan and Canada". The EU’s trade agreement with Japan has a very
similar system of arbitration to that set up by the Withdrawal
Agreement. The UK mandate also suggests that this system should not
apply to the chapters of the agreement dealing with level playing
field issues such as labour and environment standards. It also
strongly rejects any role for the ECJ – but this may have as much to
do with its rejection of a substantive role for EU law in the future
relationship as with its views on procedural dispute settlement
provisions.
Although there is currently no clear answer, as negotiations are still ongoing, several possibilities are nevertheless evaluated. The ECJ is seen as unlikely to be agreed upon, as the UK government's clear negotiating position is that the dispute mechanism should be "appropriate to a relationship of sovereign equals".
The EFTA Court, created initially to enforce European Economic Area Agreement, is also suggested, but as the UK is not currently looking for a trade partnership with the EU as close as the EEA, this is also unrealistic.
The IfG seems to conclude that the most likely scenario, in general, is the creation of a new body:
The UK and EU could set up a new court or arbitration panel to settle
disputes. This could involve some combination of UK judges and
European judges. The system for dispute resolution for Canada and the
EU is along these lines, and both the UK and the EU have suggested
such a mechanism for the future UK–EU FTA.
Clearly, however, if there is no future trade deal agreed between the UK and the EU this is also immaterial. In this case, it seems most likely that the International Court of Justice (ICJ) would become the venue for any claims made by states, rather than EU institutions, as "in theory, the ICJ can enforce any treaty."