What leaving means a minima is quite clear: Not being a party to the main treaties (TEU and TFEU) and therefore having no members to the European Parliament, no commissioner, no vote in the EU Council or in the European Council, no judge to the EUCJ, etc. With the weight of a referendum whose result was not even close, it's incredibly difficult to see how the UK could backtrack on this and not formally leave the EU in a few years time.
Beyond that, it's impossible to predict how the negotiation might go and that makes the now famous phrase “out means out” almost meaningless. Clearly, one option would be an EEA-like deal but it would involve very heavy constraints for the UK and it's difficult to reconcile this with the promises of the Leave campaign and the apparent motivations of the voters. At the same time, since the UK is also very keen on retaining preferential access to the single market, it might find itself forced to access something like that (and the many influential people in UK business or politics who have lost the referendum are probably not going to sit still and not defend their interests in this matter).
If that's what comes to pass, you could end up with situation much like that of Norway; pretty much all the downsides of membership with even less sovereignty than before and little influence on the direction of the EU. You could argue whether that counts as being somehow part of the Union but formally and legally, the UK would clearly be “out” of it. It's also difficult to see how such an arrangement could be viable in the long-term, with all the same rules people have been complaining about still in place and now literally being dictated from Brussels.
For, unlike what the other answer implies, that's exactly what's required of Norway: Applying most EU rules (including the stuff on light bulbs, vacuum cleaner, etc. but with one major exception: agriculture and fisheries) with no actual say on their contents. The single market just cannot work without making sure, e.g., that products authorized in each and every country of the EEA meet common agreed-upon standards regarding safety, environmental impact, labelling, etc. So Norway isn't by any stretch of the imagination free to decide alone or do anything else than automatically transcribing the rules and regulations coming out of the EU.
In practice, the EU decision-making process proceeds without any input from non-EU EEA member states and once it's done, experts from the non-EU EEA states have to evaluate whether the decision seems “EEA-relevant” and submit their view to the EEA Joint Committee (“joint” because it includes representatives of the non-EU EEA states and of the EU). If the JC decides something is EEA-relevant, non-EU EEA states have to implement it in a timely manner with no flexibility at the national level (see the description of the process on the EFTA website).
Interestingly, EEA members and even Switzerland contribute to the EU budget (in particular to structural funds and also to some other programmes they take part in) so, again contrary to what the other answer implies, you can't even say with any certainty that being out of the EU will allow the UK to reduce its financial contribution, at least not if it wants any sort of agreement with it. It's also important to realize that the UK already negotiated many special conditions, including a significant rebate on its financial contributions.
One thing the Leave campaign promised (implicitly or explicitly) is that the UK would be able to do better than this, basically pick and choose and negotiate a special deal and especially get rid of the free movement for workers and any financial contribution. To be sure, the UK is larger than Norway, Switzerland, and all the microstates that have special relationships with the EU but it's unclear whether that's an advantage that would allow it to negotiate something very different from the EEA or the bilateral agreements with the EU which Switzerland (still) has. There are many technical and political reasons why it would very difficult for the EU to allow this without seriously putting its very existence in jeopardy (that is: even if it wanted to help the UK out of this conundrum, there is no easy way to do it).
The last option is a clean break, which would actually allow the UK to free itself from all the rules you alluded to. I have heard many people in EU circles (bureaucrats or proponents of further EU integration) who would unofficially favor this solution and have no interest in nudging the UK to remain in the single market, precisely because they are afraid that anything else than a clean break would require the kind of deal I mentioned before and tempt other countries to renege on their commitments and make new demands.
Legally, a clean break would seem unproblematic and the UK would unambiguously be completely out of the EU. It would therefore clearly respect the wishes expressed by the British people in the referendum. The main question here is how much normal (free) trade relationships with the EU would hurt the UK economically (including but not limited to the financial sector) compared to the current unfettered access to the single market. It might also make life a bit more difficult for British citizens living on the continent and impact people in a few other ways but I suspect that this will not be the number one concern of the negotiators.