Trump talked about voter fraud and called it a big problem in the US which nobody has the guts to talk about it.
Vote fraud does occur in elections, though I personally doubt that it's a big problem. It's actually a rightwing conspiracy theory.
Trump occasionally talks about voter fraud and it might be one of the reasons he'll give if he loses the election in November. As The New York Times described it, it is an argument to "revive his embattled candidacy".
Politico does have an article about this which states that:
It’s also common for White House losers and their allies to respond with a declaration they’ve just been a part of a sham election.
So, personally I think that it's simply an excuse and a foreshadow for Trump if he loses in November.
What do US people think about the issue (poll results)?
There have been polls around about this issue (though they are not that common).
One such poll by Gallup shows that 36% of US citizens thought that it's a major issue and Republicans were more likely to view voter fraud as a major problem.
What are the (probable) count system malfunctions that let Trump claim so?
Count system malfunctions are very rare unless the systems somehow get hacked.
Trump mainly claim that people voted multiple times, but he didn't explicitly say what kind of fraud he is hinting at and of course he has no evidence to back up his claims.
“Watch your polling booths, because I hear too many stories about Pennsylvania, certain areas. We can’t lose an election because you know what I am talking about.”
~ Donald Trump at a rally in Lancaster, Pennsylvania
Did unfair counting of votes happen before?
Notably, many also claimed the 2000 presidential election was rigged, where Gore won the popular vote but lost electoral vote and Bush ultimately wins the presidency. However, there isn't hard evidence to prove this.
Unfair counting has happen before but due to cases of voting multiple times or impersonation. However, recently, with the implementation of Voter ID in many states, it has become very difficult to impersonate others.
This article by The Washington Times lists out some minor cases.
Personally, I do not think that voter fraud is a major issue and would really change the results of the election. His prediction that there would be voter fraud in Pennsylvania is absurd and baseless. It's just for him to attack Clinton's honesty which fits in his line of attack.
Most importantly, it's an excuse for Trump if he loses.
Finally, check out this commentary by The Guardian which addresses your question quite clearly:
Donald Trump believes he’s losing, and he doesn’t like that. Calling for ‘election observers’ to monitor the poll is the dangerous first episode in an ugly tantrum