The Bosnian government and elections can reasonably be called "intricate".
I'll focus on the presidency, since it seems most clear. There are two regions: the Federation of Bosnia and Herzegovina ("Federation"), and Republika Srpska ("RS"). These regions are defined geographically. They follow the lines of the peace agreement in 1995. Bosniaks and Croats live mostly in the Federation, Serbs in the RS.
When a president is to be elected, the RS chooses one person by simple plurality (the person with largest number of votes wins). The Federation selects two people. Each candidate in the Federation must enter either the Croat or the Bosniak election (it is expected that an ethnic Bosniak would not enter the Croat election, and vice versa, though this seems to be enforced only by tradition). Each elector must choose which election to participate in. It is expected that electors will participate in the election that they identify most strongly with, but this also isn't enforced. Two people are chosen: one to represent Croats, and one for Bosniaks.
Jewish people or other nationalities may choose to vote in either election. The former foreign minister Sven Alkalaj is Jewish, and represents a Bosniak nationalist party. This may point to how other Jewish people chose to vote.
The three people elected share the Presidency, with the position rotating every 8 months.
This system has been ruled to violate the European Convention on Human Rights (ECHR) in the case Sejdić and Finci v. Bosnia and Herzegovina, because the presidency was guaranteed to the three major ethnicities, with Jews and Roma being excluded. At some point the country will need to change its constitution if it is to come into line with the ECHR (often seen as a prerequisite for EU membership).
The upper house of the legislature is elected on an ethnic basis, similar to the presidency; each voter in the Federation can either vote for Croat or for Bosniak candidates, not for both; the Federation elects 10 members. The RS chooses an additional 5 members.
The lower house is selected by an open list method of PR, with separate elections being held in the Federation and the RS. This ensures that 2/3 of the members represent the Federation and 1/3 represent the RS. Multi-ethnic parties exist and field candidates in both regions, however the ethnic parties remain dominant.
sources: Wikipedia and an info booklet from the EU.