I've gone and read the actual rules currently in effect (pdf, hosting webpage) and as best as I can tell, the answer is that they just won't be counted. However, it's really the subsequent votes where it gets interesting (although this is going beyond the scope of your question).
Rule 40 (d) and (e) are the relevant ones here:
(d) When at the close of a roll call any candidate for nomination for
President of the United States or Vice President of the United States
has received a majority of the votes entitled to be cast in the
convention, the chairman of the convention shall announce the votes
for each candidate whose name was presented in accordance with the
provisions of paragraph (b) of this rule. Before the convention
adjourns sine die, the chairman of the convention shall declare the
candidate nominated by the Republican Party for President of the
United States and Vice President of the United States.
(e) If no
candidate shall have received such majority, the chairman of the
convention shall direct the roll of the states be called again and
shall repeat the calling of the roll until a candidate shall have
received a majority of the votes entitled to be cast in the
convention.
Specifically, a vote for anyone who didn't meet the 40(b) threshold is not even reported in the final totals, as if they'd abstained. They would still count to prevent the front-runner from reaching a majority, though.
It's also worth looking at Rule 16(a)(2) (emphasis mine):
The Secretary of the Convention shall faithfully announce and record each delegate’s vote in accordance with the delegate’s obligation under these rules, state law or state party rule. If any delegate bound by these rules, state party rule or state law to vote for a presidential candidate at the national convention demonstrates support under Rule 40 for any person other than the candidate to whom he or she is bound, such support shall not be recognized. Except as provided for by state law or state party rule, no presidential candidate shall have the power to remove a delegate.
The first part (before my bolding) means that the first vote is entirely a formality, because anyone who doesn't vote in accordance with their bound vote is ignored. So the "faithless elector" equivalent cannot happen at the Republican convention. The bolded part is more interesting, however. It means that a candidate can't get around their lack of Rule 40(b) support by suborning enough delegates to get past the threshold - any such support is also ignored.
This then interacts very interestingly with 40(e) (which simply says that they keep voting) and the last part of 40(b) which says that supported names have to be submitted one hour prior to the nomination. Since 16(a)(2) prevents any delegates from offering their support to other candidates prior to voting, 40(b) prevents names from being added on the fly, and 40(e) requires repeated votes until someone wins, there are only two possible outcomes: Either the front-runner picks up enough delegates on the second vote (once they're unbound) to push them over the majority, or the convention is deadlocked. There's no way to pick anyone other than the front-runner who qualified, but they can't win either.
My guess is that at this point, someone would move to adjourn, but I can't tell whether or not new names can be added during a break, or if 40(b) is a one-time thing. My reading of it is that it's a one-shot, but I'm not an expert. Under that circumstance, there would be no options except to change the rules (or suspend them, as per Rule 32), after which anything goes.
