Your understanding of it doesn't seem to be entirely correct. It was a general statement (and as far as they go, a poorly worded one) in response to a proposal by the Greens to increase refugee intake from 13,750 to 50,000 per year (as far as I can tell, effective immediately); and the part about refugees taking jobs and refugees being unemployable are two separate sentences.
The general statement made, according to an ABC report, was:
"For many people, they won't be numerate or literate in their own language let alone English," he said.
"These people would be taking Australian jobs, there's no question
"And for many of them that would be unemployed, they would languish in
unemployment queues and on Medicare, and the rest of it. So there
would be a huge cost.
"There's no sense in sugar-coating that, that's the scenario."
The (very badly worded) point he was trying to make was that such a dramatic increase would place significant stress on the system and incur significant expense with little to no benefit (at least initially).
From the same article:
Speaking earlier in response to the proposal, Mr Dutton warned any
significant increase would be costly and put pressure on Medicare and
He told Sky News it could lead to a large number of people struggling
to integrate into Australian society.
Other Liberal figures attempted to better explain the position said:
Senator Macdonald said Mr Dutton's comments were "matters of fact".
"Unskilled people coming in could well put pressure on the current
unemployment situation in Australia," he said.
Foreign Minister Julie Bishop earlier defended Mr Dutton's remarks,
saying the Minister was pointing out the costs of resettling refugees.
"These are all significant costs and we shouldn't run away from it,
that's a fact," she told Sky News.
"Peter Dutton was making a self-evident point that there are
significant costs involved in ensuring that people are able to
contribute to Australian society."
The outrage occurred mainly for two reasons:
1) It's a badly worded, generalized, blanket statement that shouldn't have been made in the manner it was (and politicians should know better than to make such broad, blanket statements)
2) It was said at the start of an election campaign (which meant that it would, of course, be jumped on with glee by every man and their dog).
People have been quick to demonise the statement, and point out examples of where immigrant workers have become highly successful, but with much of the confected outrage generated by the media, a lot have seemed to miss the, albeit extremely badly worded, point that he was trying to make:
That increasing the refugee from current levels to 50,000 per year (nearly a four-fold increase) is not sustainable or economically viable with the current resources available.
Please note: I do not endorse or condone his statement. I am not affiliated with any particular party and have a rather dim personal view on Australian politicians in general. I'm merely trying to be objective and my answer is based purely on my observations.