A useful breakdown of why there is a divorce bill, and the process for deciding how much it will be, appears here.
What is the reason a country has to pay?
To sum up, items the UK will be expected to pay for include:
Outstanding budget commitments: The EU Budget operates through a multi-annual spending structure, which means projects are paid for over a period of several years. As a result, EU Budget payments are back-loaded and many will be paid out post-Brexit.
EU officials’ pensions: Like the UK civil service pension scheme, the Pension Scheme of European Officials (PESO) is an unfunded scheme and operates on a ‘pay-as-you-go basis’, with costs being covered by the annual EU Budget as they arise.
Contingent liabilities: The EU incurred contingent liabilities while the UK was a member state. These liabilities effectively constitute payments that would be triggered in specific circumstances only, for example, Ukraine defaulting on its EU loan.
Other costs of withdrawal: This would cover the relocation of the two London-based EU agencies after Brexit; the European Banking Authority and the European Medicines Agency. Other costs include the decommissioning of the Joint Research Centre nuclear sites and funding British teachers seconded to European schools until 2021.
(Text of list items is from the above link)
What would happen if a country refused (or in an extreme case did not have the means) to pay?
Theoretically, the EU could impose trade sanctions on the UK or take action through the International Court of Justice. At a minimum, it would probably refuse to conclude any other post-Brexit agreements with the UK, which by itself would have severe consequences for the UK economy.