Coup d'état attempt in 2016
For a long time, before Erdogan came to power, Turkey was silent partner in NATO. They mostly towed NATO line, without much complaint. Their territory and airbases were used for invasion of Iraq, Afghanistan etc .. As a reward, Turkey was allowed some leeway considering Cyprus, occasional military coups, corruption and not very democratic behavior etc ... For NATO, most important thing was that Turkey effectively controlled entrance to Black Sea and Turkish territory could be used against southern parts of Russia(USSR) and of course other countries in Middle East. NATO secretly supported Kemalist elite in Turkey which was mostly centered around army as a "guardian of the Republic".
Things begun to change with rise of Erdogan, mildly Islamist and neo-Ottoman. Erdogan's promise was to make Turkey great power again - this came at moment when it became clear that Turkey would never became full member of EU. Starting from let's say 2010, European public opinion definitely turned against Islamization of Europe for various reasons (terror attacks, population replacement, increase of sexual assaults etc ...). Turkey with its customs, religion, language and even genetic background is largely alien to Europe, and EU population increasingly doesn't want Turkey in EU. To compensate for that, Erdogan promised to Turkish population that Turkey would turn to Middle East, became leader of Muslim Ummah as Ottoman Empire once was, and be more independent in its foreign policy.
Now, Erdogan is shrewd, experienced politician, so he didn't immediately break ties with the West. Instead, he proceeded cautiously, wanting to show that Turkey could do some things independently from NATO and the West, especially if they do not get what they want from them. In case of air defense systems, at that time Turkey was moving into Syria (again wanting to show themselves as leaders of Muslims) and Erdogan wanted to quickly remove government of Assad and replace him with some of Islamist factions that Turkey supported and still supports (dubbed "moderate opposition") . West did support Turkey (and Saudi Arabia) but more cautiously - public opinion was not very keen for another war, and they questioned why should they root for someone who had undeniable ties with Al Qaeda and even with emerging Islamic State (IS, ISIS) . There was also question of Israel (West did and does support them, while Turkish relation increasingly soured as Turkey supports Palestinians) and of course, elephant in the room - Kurds. As Assad government gradually lost control, Kurds became more independent and created their own semi-state in Syria. It is well known fact that Turks and Kurds do not like each other (to say the least ), so Turkey was appalled by this prospect, while West saw Kurds as their best best in Syria and gave them reasonably good support, including arms and training - which could be latter used against Turkey itself.
Therefore Erdogan decided to demonstrate that some independence, and started collecting offers for aid defense system. Mind you, classical air defense systems (against aircraft, not so much against ballistic missiles) are not West's forte - they rely on air power for that. Russia and China are more advanced and experienced in this field. As Russia at that time (as well as now) supported Assad, China and their HQ-9 (FD-2000) became logical choice. Of course, NATO (especially US) was not very pleased with Chinese acquisition either, so there was lot of backroom talks, deals, promises and threats. Finally, Turkey cancelled the purchase, and that was that for a time.
But then came 2016 Turkish coup d'état attempt. What happened, why and who organized whole thing is still murky. What is important is that looks like Erdogan and most of the Turks believe coup was organized with US backing , possible Saudi involvement etc... Officially , cleric Fethullah Gülen is prime suspect, he lives in US, and US refused extradition to Turkey (at least for now). What is more important, it looks like Russian intelligence services warned Erdogan about coming coup. This seems like plausible explanation, helping hand from old foe could explain sudden 180 degrees turn in relations between Turkey and Russia - especially if we consider that year before Russia and Turkey were on the brink of the war after Russian Su-24 shootdown.
After that, it is quite possible that Erdogan does not see West as partners anymore (after all they just tried to oust him, possibly kill him) and acquisition of S-400 is not only financial reward for Russia but is also to show the West that if they play dirty he could do that too. Old ties and treaties are no longer taken as granted and unquestionably, and Turkey would mind their own perceived interests first and foremost.