Usually, by 'Gentlemen's Agreement'
The representatives of the Bundestag are legally free in the exercise of their mandate, yet there are niceties to be observed and failing to do so is a political slap in the face. This happened to the AfD in this (and other) instances.
Normally, the largest faction gets to nominate the President of the Bundestag, each faction gets to nominate a Vice President, and (as explained by Philipp in his answer) there is a kind of rota to determine committee chainpersons. Then there is an election (in the whole Bundestag or in the committee, respectively) and the representatives usually ratify the nomination of the faction which 'was supposed to get' the position.
It is customary that the largest opposition party gets the finance committee. (The exception is the state legislature of Berlin, where the Hauptausschuss would have acted as a rump parliament in case of another Berlin Crisis and was hence led by the largest government party). The slots are also picked by order of faction size, so it is customary that the largest party skips this important position unless it is in opposition.
The decision 'behind the scenes,' as you put it, depends on the dynamics within one faction. People want to become committee chairs, they want to become chairs of specific committees rather than any one, and so they are bargaining with their fellow faction members and the party leadership in general how their party should exercise the pick. Sometimes that depends on where a party wants to make a point, but also on who doesn't get a cabinet post. For instance, the Greens picked Europe for Hofreiter, both because Europe is important and probably also because Hofreiter represents an important block of party members. I don't know why the FDP picked Defense.
In many cases, the other parties acknowledge the right of the AfD to nominate a candidate for a postion in principle, yet refuse to elect the specific individual put forward by the AfD. And the AfD tries time and again to have their chosen candidate elected. (It would be an interesting question if there is any member in the AfD faction who would satisfy the other parties, but trying that would let the others effective pick how the AfD assigns their posts, so we will probably never see it.)