Because life in North Korea is not that bad, and life in the South Korea is not that good
First of all, most of the stories we get about NK are from very suspicious South Korean sources, that have their own vested interest to show their neighbor as a land of Mordor. Then we have so called journalist who actually go to NK but then try to persuade themselves and audience that everything they see (including traffic on streets and even buildings) is actually fake or staged. Finally, we have regular people that go to NK, and their stories are closest to truth.
Reality is that North Korea is country under siege, under constant threat of attack by US and its allies. Therefore, they have to spend large amount of their GDP on defense. People in NK are poor, but they do stable society, low crime rate, sense of belonging, and relative equality. I imagine that people sent to Olympics belong to upper levels of society, so their patriotic feelings about their country are even higher. It is well known fact in psychology that happiness does not come from wealth, but from respect of others. People sent to Olympics are athletes, coaches, journalists and other functionaries. Those people do not have much reason to throw away their reasonable good life and defect.
What about South Korea ? Although undoubtedly richer then their Northern neighbors, SK has very high suicide rate, especially among females and senior citizens. Alienation, estrangement, breakup of traditional family ties, society that rewards egoism, these are the plagues of modern South Korea. They even had spikes of suicides and suicide attempts after death of certain celebrities, which shows that many South Koreans do not have anyone in their life close to them, so they create emotional connections with media personalities.
Comparing all of this, it is doubtful that someone from NK Olympic delegation would have vastly better life if he choose to defect to SK.