The BBC describes Navalny as Putin's main opposition rival (example article), despite the fact that he has rarely scored over 2% in opinion polls (at least two other opposition candidates score ≥6%). Navalny alleges that his candidacy rejection is politically motivated, but what political motivation would Putin have to bar a candidate with such an apparently low popularity from standing?
The spin on Navalny
For the question of why Navalny is labeled by western media as Putin's main opposition rival is an issue of framing. If you look at the two candidates who were ahead of Navalny last he was included in opinion polls on the link you provided - Zyuganov (now Grudinin) and Zhirinovsky, members of the Communist party and the populist-nationalist LDPR party respectively - both are older men from older established Russian parties (both have been around since the early 90s, contrasted with Navalny's party's inception in 2012) with ideologies the west doesn't care for so neither make for as good a story as Navalny
Navalny is not the next most favored politically to rival Putin given opinion polls, but he is the strongest contrast to Putin of any of the potential candidates, having labeled Putin's United Russia party a Party of Crooks and Thieves and having organized several protests of corruption and current leadership. He is ideologically further from and more opposed to the current regime than the other greatest contenders.
Reasons for barring Navalny
As for why Putin would bother to bar him from the election, it is important to consider that it is not only this election cycle and presidential term that Putin is taking these actions in consideration of. Putin has sidestepped and even rewritten the Russian constitution to give himself more time in power. To believe that Putin will completely relinquish power over the country he will have led for 24 years only because it is legally ordained would be, I believe, naive. And so, the reasons Putin has barred Navalny would be more forward-facing than just this election.
A strong reason would be to prevent this from turning into a slippery slope for Putin's power going forward. Navalny's party, the Progress Party, was created only in 2012 and has become quite prominent in a short time. Being very ideologically opposed to this party, Putin would not want to give any legitimacy to this party. Perhaps this election they only get 2% of the vote, but that means Navalny getting his message out for a whole election season, televised alongside the rest of the contenders. Navalny is young and charismatic, the next election would have the chance to be quite different.
Another reason would be that Putin has a goal of what he calls 70 at 70, or 70% voter turnout with 70% voting for Putin. Part of maintaining the strongman image he has cultivated is by dominating any challenge. Anything that could be construed as weakness needs to be quashed. 70% of the vote is a lot, and any chip at that is significant. Barring someone from entry is an extreme move towards this end, but this is not the only reason Navalny is barred (see above).
When reading about this, I cannot help but remember the party-state system within communist states. While currently there are six parties within the Russian Duma making Russia a multi-party system, United Russia clearly dominates the Russian politics.
According to this article, Russian authorities recognize the de facto one party system:
Dmitry Medvedev cautions opponents of Russia's de facto one party system Dmitry Medvedev, the Russian president, warned anti-Kremlin opposition politicians on Thursday against adopting radical tactics, promising he would head off any attempts to undermine the de facto one party system set up by Vladimir Putin.
This was happening 9 years ago, but things have not changed very much in the mean time:
Elections in Russia have established a de facto one-party system at the State Duma - "United Russia" will have about 340 seats, which is a constitutional majority
While 2% might not look like much, this actually means about 2 million persons and his actions are quite visible.
Alexei Navalny, a charismatic anti-corruption lawyer. Although Russia’s government-controlled election committee wants to keep Navalny off the ballot, the 41-year-old opposition leader has been campaigning across Russia for months. His campaign has attracted tens of thousands of young people as volunteers and raised over a million dollars in donations from ordinary Russians. Putin has called opposition figures like Navalny “national traitors.
”Navalny’s success has provoked a series of vicious attacks by pro-Kremlin activists (...)
From what I gather (mostly by observing Russian internet trolls, mind you, and by being a Russian internet troll myself), the main reasons are:
Putin supporters claim very high, nearly unanimous level of, well, support for Putin. Those "minus two percent" may look insignificant enough, but keep in mind that Navalny's supporters are probably the most discontent and politically active part of the population, so most of them would go vote, while many Putin's supporters wouldn't, because why bother, he wins every election anyway. Therefore the number of actual votes Navalny gets could be much higher that what you get from randomized opinion polls. Not enough to win, but maybe enough to shake the image of "unanimity". (IIRC, he ran for the mayor of Moscow and his numbers, although not winning, were surprisingly high.)
Lawfully or not, Navalny was accused of some crimes and sued before. Quite possibly for purely political reasons, e. g. to remove him from the political stage when it wasn't clear how much support he is going to get. Even though it could now work in Putin's favor to let him stand for election so that everybody could see his paltry 2%, that would require dropping the criminal charges, thus admitting they were politically motivated in the first place, thus admitting Putin was not so confident in his unanimous support and would use unlawful means of silencing his opponents, etc., etc. Not gonna happen.
Now, Navalny mostly gets support through internet-based media, he has no access to national TV, etc., which cuts off a big part of his potential audience. That would change if he is allowed to run for the election, because then he would have some mandatory TV time. Probably not gonna get many more supporters among the older generation of people who don't use internet, but who knows. (this is mentioned in the @Gramatik's answer as well)
If the [illusion of] unanimity is so important, why not bar the other contenders (Communists, LDPR etc.)? They are widely regarded as Kremlin's convenient "pocket opposition": they know their place; their opposing stances are quite moderate; they've been in politics since forever, always get their share of votes, but nothing significant ever comes out of it. Besides, giving people an [illusion of] choice is also important.
Navalny is not as unknown in Russia as those polls may suggest. When he started his political career, he was gradually gaining popularity to the point he scored 27% in Moscow mayoral elections, 2013. Presumably, around this time Russian political establishment realized they are seeing a real competition emerging: criminal charges against Navalny, largely believed to be a mean of political harassment, started in 2012. These cases got a broad coverage in the government-controlled media, and naturally had a negative impact on Navalny's popularity.
Two among the most popular candidates, Zhirinovsky and Zyuganov, have participated in every presidential election since at least 2000. There is a hearsay that they don't have the aim to actually challenge the party in power, rather, they are used to draw away protest votes from the real opposition. Their presence also makes Russia look more democratic, so there's no reason to bar them.
Navalny and his supporters are being used to undermine rule of law in Russia.
Both Navalny and his Western handlers are aware that he currently has low support in Russian population, so they are not trying to win elections or even become significant opposition at this time. Instead, they are deliberately breaking Russian law and trying to get away with it in the name of "democracy". For example, Navalny often calls for unsanctioned demonstrations where and when he pleases. When authorities sometimes ban these gatherings, he deliberately pushes on and tries to get arrested, and then Western media start their usual propaganda about Russia not being democratic etc.
This tactic has already being used in so called "colored revolutions" both in Russia and abroad. Some tiny minority of pro-Western extremists must be allowed to break law (for example, remember "Pussy Riot" incident), otherwise Western propaganda machine will declare country as undemocratic. Nevermind that Russian Duma has more dissenting options and more political parties then US Congress where Democrats and Republicans are often described as Uniparty. Western media in their role of "moral arbiter" would immediately declare country as dictatorship, and political class would follow with sanctions.
And if Russia backs down in name of "democracy" and allows Navalny to break law in one way, he and other Western backed groups would immediately seize opportunity to go even further. Gradually, they would become stronger as those above the law, and finally we would have Ukrainian scenario where certain groups of "demonstrators" change "undemocratic and corrupt" government violently in the street.
Therefore, it is quite clear that Russian authorities must not back down an inch, and must upheld Russian law no matter what Western media tells about them.
You may be over-relying on published polling numbers. It certainly seems Putin isn't relying on them as much as you are. I think you've correctly identified that he seems to see something you aren't seeing by looking at the polls.
Gary Kasparov, chess grandmaster and founder of opposition group United Civil Front*, when asked about polling in Russia, has argued that since Russia doesn't really have freedom of speech, any public polling done in that country that touches on the leadership is essentially meaningless.
Here he is making this point to NPR a decade ago:
MONTAGNE: And yet various polls have put President Vladimir Putin's popularity as high and sometimes above 70 percent.
Mr. KASPAROV: Oh, absolutely. This is true if you ignore one very important fact. You don't run normal polls asking personal questions in a police state. I'm sure that Saddam Hussein's popularity, a few days before American tanks rolled in Baghdad, was also 99 percent.
When you call to an average Russian asking about Putin, the president of the country and the KGB officer, you don't expect a true answer. If the same person asked about other issues - economy, crime, security, living standards, you receive very different answers.
* - ...and also barred from running against Putin in 2008