I'm not aware of any previous cases of this, at least not on such a massive scale as the Irish border would represent. As for remedies, the ECJ can issue fines to member states. Beyond that the EU would have to get involved and decide on sanctions, such as withdrawal of support (subsidies etc.) or even ejection from the union (although that is extremely unlikely).
In practice the EU is very unlikely to punish Ireland for what it considers to be a problem of the UK's making, and seems understanding of the desire to have no border infrastructure to prevent violence and division.
In fact there would more likely be measures taken to try to force the UK to accept the so-called "backstop" of staying in the customs union. That could include things like refusing to allow UK aircraft to cross EU airspace such as Ireland and Sweden, making it very difficult for UK airlines to operate.