This question shows an – unfortunately widespread – misunderstanding of what GPT-3 is and what it isn't.
GPT-3 is a language model. It is being fed with vast amounts of text, and from this text it learns (note that "learns" here is a technical term and does not imply any form of understanding) one thing and one thing only: the rules of the language. It does not learn facts, it does not learn information, it does not learn data, it does not learn "the truth", and it most certainly does not learn intelligence.
All it can do is string together words into coherent English sentences that are grammatically correct, vaguely related to the question you put in, and plausible-sounding. In short: it can construct text that sounds like it could plausibly be a valid answer to the question, but it does not actually answer the question, and it doesn't even attempt to.
That's it.
The fact that it gave you a wrong answer it not surprising: since there are many more plausibly-sounding wrong answers than there are correct answers, it is statistically much more likely to generate a wrong answer than a correct one.
The fact that this particular answer has a particular bias is just a statistical fluke. Nothing more.
The way GPT-3 "learns" is by breaking down the training dataset into individual tokens and then looking at each individual pair of tokens and record how often which token follows which token. For example, the sentence
the rain makes the street wet
might lead to something like the following token pairs:
- start-of-sentence the
- the rain
- rain makes
- makes the
- the street
- street wet
- wet end-of-sentence
From this, GPT-3 learns that there is a 100% chance that a sentence starts with "the", a 50:50 chance that "the" is followed by "street" or "rain", etc.
When it is asked to write a sentence, it will thus start with "the", then continue with either "street" or "rain", etc. Of course, in reality, GPT-3's training set is not a single sentence but rather pretty much the content of the entire World Wide Web, Wikipedia, Project Gutenberg, etc.
GPT-3 is a predictive model, i.e. when it sees a series of tokens, it tries to predict what the next token will be, and then the next, and then the next, and so on.
You know this party game where you are supposed to write a message by repeatedly only taking the first suggestion from your phone's predictive keyboard? You know how those messages always come out sounding awkward and hilarious but still at least somewhat believable?
That's essentially how GPT-3 generates text, just that its algorithms are orders of magnitude more powerful and its database is orders of magnitudes larger: it has been trained with essentially the entire World Wide Web and from this has extracted a 800 GB big model. So, its text sounds a lot better than what you get from your phone's predictive keyboard, but the concept is still the same: randomly stringing together words that happened to follow each other in the training corpus.
In this particular case, GPT-3 has somewhere in its model that text that talks about sunken ships in WW 2 often talks about German u-boats and that texts that talk about sunken ships sometimes talk about salvage and display in a museum, and that's why it generated those two things even though neither of those two things are true. It doesn't matter whether they are true or not, what matters is that they can plausibly follow each other.
The fact that you asked this question shows that GPT-3 is working as intended: it generated a text that sounds plausible, if it had generated random gibberish, you wouldn't have asked the question.
