The only ways for a member of the Commission to lose their post before the end of the Commission’s five-year term are (according to TFEU, article 246) if they die, resign, or are compulsorily retired. The President of the Commission can force any member to resign, and the European Parliament can force the whole Commission to resign by a vote of no confidence. Compulsory retirement happens when the Court of Justice decides (on application by the Commission or the Council) that a member of the Commission is either unfit for office or is guilty of “serious misconduct”.
As a special case, the High Representative of the Union for Foreign Affairs and Security Policy – who, by virtue of their office, acts as a member of the European Commission, too – can be removed from office by the European Council. The European Council also appoints the High Representative.