In theory, a party split is possible. That's how both current parties were founded:
The Democratic-Republican Party (which often called itself a/the republican Party) split after the 1824 election was sent to the House of Representatives to decide - one of the two candidates (Andrew Jackson, the winner) formed the Democratic Party with his supporters, and the his opponents formed the National Republican Party, which absorbed the remnants of the Federalist Party and eventually merged into the new Whig party. (The modern Democratic Party traces to this split.)
The Whig Party tore itself apart over the question of slavery after their 1852 convention - many northern Whigs (including Lincoln) joined the new anti-slavery Republican Party, while many of the southern Whigs joined the American Party (aka the American Republican Party, aka the Native American Party (in the sense of anti-immigration, not that of Indians), aka the Know Nothings) or the Constitutional Union Party, and then eventually seceded. (The modern Republican Party traces to this split.)
However, in the modern highly-polarized political atmosphere, a split like that would likely result in the splitting party losing most or all of their political influence for the next several years: 49% vs 48% is relatively even, but 49% vs 26% vs 22% is a clear victory for the 49%. Even if it becomes 45% vs 30% vs 22%, the unsplit party has a significant advantage.1 This doesn't mean it won't happen, but it does mean that a large number of career politicians would have to choose principles over power - something that was much less of an issue in the past.
On a more immediate note, it is too late for a newly split-off party to nominate a candidate for the 2016 elections. Each state has ballot access rules which determine who will show up on that state's ballots in November. I believe all the deadlines have passed at this point, so a new candidate would have to be a write-in in every state, but I'm not sure all states even count write-ins.
It would be possible to organize a campaign to promote faithless electors to vote for the new party's candidate instead of the one they're bound to, but that would probably provoke a major backlash from the public (even worse than the one that was threatened around "stealing" the convention nomination from Trump) and may trigger a constitutional crisis ("Why are they allowed to do that? We need to fix it!").
So not only would the splitting party lose long-term influence, they'd also be all but throwing this election in the process. A split around the mid-term elections, with both halves having two years to organize themselves and put up a presidential candidate in 2020 would be their best option, but that just plays back into the long-term issues.
1: All numbers for demonstration purposes only - they don't correspond to any actual values.