It simply depends on the status of the territory: if it's part of a country which belongs to the EU then by definition it is part of the EU.
Formally British Overseas Territories are not considered part of the UK:
The British Overseas Territories (BOTs) or United Kingdom Overseas Territories (UKOTs) are fourteen territories under the jurisdiction and sovereignty of the United Kingdom. They are remnants of the British Empire that have not been granted independence or have voted to remain British territories. These territories do not form part of the United Kingdom and, with the exception of Gibraltar, are not part of the European Union.
Whereas French Overseas department are full-fledged parts of France:
The overseas departments and regions of France (French: département et régions d’outre-mer or DROM) are departments of France which are outside metropolitan France, the European part of France. They have nearly the same political status as metropolitan departments, although special constitutional provisions allow them greater autonomy and they are excluded from certain domestic statistics, such as the unemployment rate.
As integral parts of France and the European Union, overseas departments are represented in the National Assembly, Senate, and Economic and Social Council, vote to elect members of the European Parliament (MEP), and also use the euro as their currency.
There are only 5 French Overseas departments: French Guiana, Guadeloupe, Martinique, Mayotte and La Réunion. These should not be confused with French overseas collectivities, which have a semi-autonomous status and as such are not part of the EU (with the exception of Saint Martin, otherwise it would be too simple!).