Wikipedia says that Republika Srpska is effectively exercising a veto over the recognition, partly through the threat of using Kosovo's independence as precedent to support its own potential secession from Bosnia and Herzegovina. Furthermore, as far as I understand it, the three members of the presidency of Bosnia and Herzegovina would have to agree unanimously to extend recognition to Kosovo, so as long as the Serb member of the presidency withholds consent, it will not be possible.
Presumably, no similar structure exists in the Montenegrin constitution whereby Serbs in government could veto foreign policy decisions.
User rs.29 also notes in a comment that public opinion in Montenegro shifts back and forth between greater and lesser affinity with Serb identity, and therefore greater or lesser support for Serbia's political goals. Montenegro's recognition of Kosovo came during a time when it was seen as preferable to align with "the West." Such a state of affairs is far less likely to occur in Bosnia and Herzegovina because of the constitution, particularly its effect of magnifying nationalist political sentiment along with the need for unanimity in the three-member presidency.