Northern Ireland, like the rest of the UK uses the First Past The Post (FPTP) system of elections. A well-established result of this system is the formation of two-party systems. This shouldn't be taken literally - it means that two main parties will form, and compete for majorities, while other parties are much smaller.
In such two-party systems, typically one political dimension will divide the two parties. A common dimension is the "Left/Right" division, as seen in most of the UK.
The elections at stake here are parliamentary seats. This is different from presidential systems, as the two main parties that exists within one electoral district may not always align with the main national parties. This can be explained by the fact that the nationally separating dimension is locally less relevant.
In particular, it appears that the relevant dimension in NI is nationalist/unionist, and in Scotland it's similar. That's why NI has its own parties, and why the SNP has a majority in Scotland.
Why don't the main UK parties enter the NI elections? The problem is that their candidates would have to position themselves on the nationalist/unionist dimension, and they would not be as believable on that dimension as the existing NI parties. Locally, both Tories and Labour would be third parties in a two-party system.