Meta-Theoretical Problems
As alluded to in the question, many of the criticisms of The Logic of Political Survival are meta-theoretical in nature: de Mesquita (et al)'s theory is built on rational-choice theory. Many criticisms stem from rational-choice theory and trickle down.
Rational choice theory assumes perfect information is available. Every voter must know every possible choice they could make and the expected outcomes from those choices. They should also know every move available to every other voter as well as the leader (as well as the expected outcomes of every choice the leader could make). In reality, nearly all of this information is unknown.
Second, it assumes that actions choose rationally. That is to say that each person (voter and leader) know exactly what they want and exactly which options to pick to procure the most of what they want at the lowest cost. In reality, voters often don't know what they want and are often incapable of comparing the various options they have.
Third, it assumes that actors have unlimited cognitive ability to spend on choices. Every voter, as well as the leader, must compare every possible outcome that can possibly happen to determine what choices are optimal. In reality, voters often spend very little time developing opinions and may not engage in any meaningful critical thinking about their choices.
James Morrow, one of the authors who wrote The Logic of Political Survival, is well-known in game theory circles and none of these criticisms would be a surprise to him.
Empirical Problems
There are also some empirical or methodological problems that have been noted.
de Mesquita's analysis suffers from omitted variable bias (Clark and Stone, 2008). In certain statistical tests and situations (when the omitted variable is correlated with both the dependent variable and one of the independent variables) this can make the results appear stronger than they are. Once this error is corrected, the results are no longer interesting.
There have also been problems applying the theory to case studies. For example, when the theory fails to predict different outcomes from different former U.S.S.R member states (Gallagher and Hanson, 2013).
Although selectorate theory was designed to explain politics in authoritarian regimes, its basic constructs (like the selectorate) are not clearly applicable to non-democratic regimes (Gallagher and Hanson, 2014). In these countries, formal political institutions don't really structure how politics happens, so a theory based on those institutions doesn't make sense.
Gallagher and Hanson also found that the Logic of Political Survival suffers from using crude measurements. For example, according to de Mesquita's coding scheme about 80% of all countries have an elected legislature. This includes both the United States as well as countries like Uzbekistan. These measurements are so broad that they are meaningless. Any conclusion drawn from these measurements is (at best) imprecise.
Finally, Gallagher and Hanson are also concerned that the theory assumes every member of the selectorate is the same. However, in reality they would be motivated by a variety of different concerns. This weakens the link between the benefits the leader can provide and the support of the selectorate.