Is seems that one of the problems of finding a solution acceptable for both UK and EU was that no other non-EU country has the kind of relation with EU that UK seeks. But Japan has a reasonably symbiotic trading relation with EU without any form of immigration/freedom of movement policy.
The agreements cover fundamentally different things. The agreement rejected repeatedly by the House of Commons, the withdrawal agreement, covers:
- The rights of EU and UK citizens in the UK and EU.
- The transition period.
- The treatment of goods in transit on the last day of the transition (they can be exempted from new tariffs, etc).
- Legal processes begun in EU courts before Brexit/end of the transition (they're still valid).
- The transfer of social security entitlements (eg, transferring your state pension rights from one country to another)
- The legal acceptability of goods already in circulation which were approved by regulators in the EU and so previously OK in the UK but would not otherwise be in the future.
- Ongoing police and judicial cooperation processes at the time of exit (I'm thinking of the European Arrest Warrant, but there must be many others)
- The revocation of access to and destruction of data from EU-only databases.
- The rules applying to public procurement processes begun before Brexit.
- The payment by the UK of contributions to projects the UK committed itself to during its membership, and of things like the pensions of former EU commission staff members.
- The border with Gibraltar and Ireland and the bases in Cyprus.
Almost none of these are an issue between the EU and Japan, often because they come from the reduction in rights and cooperation that Brexit will bring about.
What the agreement does not cover, except during the transition period, is trade.
A Japan-style agreement would not resolve any of the above, and would require extensive negotiation on a whole bunch of new things related to trade.
From a trade standpoint: because of the Good Friday Agreement and its implications on the Irish border.
A Japan style deal would be fine, if it weren't for that peace agreement, which requires that there be no hard border between Ireland and Northern Ireland.
This has implications for the EU as a whole, because if the UK does its thing and opens up to, say, chlorinated chicken, then you could imagine a UK company smuggling that into Ireland and then to the rest of the EU.
For everything else, see Alex Hayward's excellent answer.
According to the EU, there are actually two sets of negotiations:
- The divorce agreement on leaving the EU.
- The future trade agreements on what happens after Brexit and the transition period (if any).
The EU has always insisted that the questions must be seen separately. There are good reasons for their position. Mixing one-off issues like the UK share in EU pension payments with ongoing issues like mutual recognition of consumer safety tests is a bad idea.
The UK has always tried to mix the two questions. There are good reasons for their position. Brexit gets even more complicated if the terms cannot reference the future relationship. Consider the Backstop -- it exists only to cover the potential failure of the future negotiations regarding Ireland.
What you propose would cover the second set of negotiations. The current problems are with the first set of negotiations. So Japan, Norway, Canada, or an unique UK-EU relationship are not yet on the table.
You asked "why doesn't the UK go for..." You should have asked "why didn't the UK go for...".
At this point, 2 days after the UK's EU membership was planned to end, and less than two weeks before the extension runs out, trying to go for any different deal is completely pointless. If the UK manages to get a two year extension (if that is what they want), then we might start from fresh with a completely different deal.
However, there is one huge difference between UK and Japan: The UK is next door to the EU, Japan is far away. "Free movement" between EU and Japan is not very important. There are not that many people who would wish to move from the EU to Japan or vice versa. There are only about 15,000 British citizens being residents in Japan.
With the UK on the other hand, there are many British and Irish citizens living on one side of the UK / Ireland border and going to work on the other side. Millions of UK citizens living in the EU and vice versa. For Japan, the EU accepts free trade without free movement because of the low demand for free movement. For the UK, no way.