The Australian Electoral Commission has some material about this on its website: https://aec.gov.au/Voting/counting/ The following is based on that and my personal experience.
Regarding what ballot papers do and don't count, the best reference is the Ballot Paper Formality Guidelines: https://www.aec.gov.au/Elections/candidates/files/ballot-paper-formality-guidelines.pdf
On election night, there is a hand count conducted on election night at each polling place.
For the House of Representatives: ballots are sorted by first preference - during this process, ballot papers which appear to be non-compliant with the rules are caught and sorted into a separate pile. Following the first preference count, a Two Candidate Preferred count is done - as the count is conducted on a polling place level rather than across the whole electorate, polling officials can't do a proper distribution of preferences at this point, so the Divisional Returning Officer selects the candidates they believe are most likely to be the final two candidates (generally, one Liberal/National and one Labor) and polling officials sort ballots according to which of the two candidates has the higher preference.
For the Senate: ballots are sorted by first preference - during this process, ballot papers which appear to be non-compliant with the rules are caught and sorted into a separate pile. The first preference results are counted, and that's it.
(When I say "ballot papers which appear to be non-compliant": the Formality Guidelines are complex, and sometimes a ballot paper could go either way depending on how you interpret the rules. In that case, you put it in the "informal" pile and let senior officials deal with it later.)
The polling place closes at 6pm and results will start flowing in from 7pm for small rural polling places all the way through to 9pm, 10pm, 11pm or potentially later for large urban polling places.
The election night count doesn't ultimately determine the result of the election - it's only used for media reporting. However, it's typically enough for media outlets to call the result of most races and the overall outcome of the entire election by about 9pm/10pm Sydney time.
After election night, ballots are moved to an AEC counting centre. This process also involves reconciling a lot of paperwork from the polling places to ensure that ballot papers haven't been lost and that everything matches up. Absentee and provisional votes also need to be transported to the home electorate and the envelopes verified against the electoral roll so officials can make a decision whether or not to allow them to be counted. For federal elections, the law allows up to 13 days after election day for postal votes to arrive.
House of Representatives ballots will then be counted by local electorate staff - this process is done by hand, in the presence of party scrutineers (poll watchers, in American terminology) who will check ballot papers to ensure they are being correctly counted and that incorrectly completed ballots are excluded. They have the right to challenge decisions to count or exclude a particular ballot, which is used frequently in close contests.
Once it becomes clear that a candidate has a lead wide enough that it is impossible for any other candidate to win, the Divisional Returning Officer will declare them elected. This means that they don't usually wait the full 13 days for postal votes to arrive before officially determining the candidate, but in a close contest they will need to wait for every last postal vote.
Senate ballot papers will be sent to the state's Central Senate Scrutiny facility, where they are scanned and counted electronically, with a system that flags ballot papers for human review if they don't look like they are completed correctly. (This is also done in the presence of scrutineers, but their job is much harder post-2016 now that scanning is used instead of manual data entry.)
The Senate count cannot be finalised until all votes have been scanned, so it typically takes several weeks to officially declare. The media can usually get a reasonable guess at what the party makeup of the Senate will look like based on the preliminary figures, but it's much less predictable than the House of Representatives.
