Contrary to popular belief, the rule in Canada is not that the party with the most seats in the House of Commons forms the government. The actual rule is that the leader who can get a majority of MPs (170) on their side will become Prime Minister.
In the event of a minority parliament, the incumbent Prime Minister (currently Justin Trudeau) will have the first chance to seek to form a government. Even if the Liberals finish in second behind the Conservatives, if Mr. Trudeau believes that he can get the support of enough smaller parties, then he can present a Throne Speech to Parliament.
If the House of Commons votes to approve of the contents of the Speech, then Mr. Trudeau can continue as Prime Minister. If the House of Commons votes against approving the contents of the Speech, or if it becomes clear ahead of time that such a speech cannot pass, then Mr. Trudeau must resign as Prime Minister and allow the Leader of the Opposition (Andrew Scheer) to seek to form a government by presenting his own throne speech.
If neither party is able to get a Throne Speech approved by the House of Commons, then the Governor General may convene the party leaders to see if an agreement can be made. If it cannot, then Parliament will be dissolved and a new general election will be called, likely early in the new year.