At the point in which Caroline Lucas won the seat in 2010, Brighton Pavilion had already developed into a 4 way marginal between Labour, the Conservatives, the Green and the Liberal Democrats. In the previous election in 2005 the respective vote shares were 31.5%, 23.9%, 21.9%, and 16.5% (note boundary changes make some of these figures less usefult than is ideal). This isn't all that surprising. Brighton is a city with a young, progressive and relatively wealthy population, especially compared to others on the South coast. In many ways this makes it more comparable to London constituencies such as Islington North or Islington South, or other urban constituencies such as Bristol West and Norwich South than the rest of East Sussex.
The 2010 election occurred during a downswing in Labour support. However a lot of this was a movement of progressives away from Labour and towards the Liberal Democrats and (in the areas where they already had significant support) the Greens. In the specific case of Brighton Pavilion this swing was heavily directed to the 3rd place Greens (the Liberal democrat vote stayed constant in absolute terms, but fell as a percentage due to higher turn out.
In the subsequent election in 2015 Lucas benefited from both name recognition as the incumbent and the massive collapse of the Liberal Democrat vote following their entry into a coalition government with the Conservatives. In the snap elections of 2017 and 2019 the Liberal Democrats didn't run a candidate, which further benefited the Greens.