Since the ratification of the Constitution, five incumbent U.S. presidents who were eligible for another term and who actively sought the presidential nomination1 have been denied it, all of them long before the primary era when being nominated for president only required a few hundred votes rather than millions:
- John Tyler (W/I2) in 1844 (the Whigs hated him by 1844, and it wasn't technically even his party anymore, since they'd expelled him from it back in September 1841 for voting against it).
- Millard Fillmore (W) in 1852 (the northern Whigs dropped him like a hot potato over the Fugitive Slave Act, causing him to narrowly lose the nomination to Winfield Scott).
- Franklin Pierce (D) in 1856 (two words: "Bleeding Kansas").
- Andrew Johnson (D) in 1868 (even the Democrats realised that the recent impeachee was a millstone around their collective neck by then).
- Chester A. Arthur (R) in 1884 (in poor health, likely wouldn't have been up to the task of being president another four years).
Four of those five (Tyler, Fillmore, Johnson, and Arthur), as noted by the source given in @Fizz's answer, were elected as vice presidents, and succeeded to the presidency upon a presidential death. Only one of the five, Pierce, was elected as president only to be denied the nomination the next time around.
1: Thus excluding incumbents who bowed out very early in the race, such as Truman in 1952 and LBJ in 1968.
2: Although Tyler was elected vice president (and succeeded to the presidency upon the death of William H. Harrison) on the Whig ticket, the party expelled him in late 1841 for voting against it.