Unfortunately you wont be able to use Qantas points to fly or upgrade with Air Canada as they are different airline alliances i.e Oneworld and Star Alliance respectively.
If its a necessity you have to stop in Vancouver and want to definitely go in September.
You may be hard pressed to find a ‘quick route’ and/or readily available award bookings through your Qantas points. Chances are you will have to stop in LAX, SFO or somewhere. with flights booked with Qantas or American Airlines. There is also difficulty in finding reward availability for a separate booking onward to Montreal as I doubt there is a booking that will get you all the way through to Montreal via Vancouver especially with Qantas points as no Oneworld affiliates do a direct flight between Vancouver and Montreal.
If I could make a suggestion, if you want to use points and save money, It would be best to try do economy reward bookings to Vancouver with Qantas, American Airlines or Cathay Pacific, (there is availability but cost is upwards of 500k+ points for 2 Adults & 2 Children return on Qantas site through various dates in September)
You would then have to pay for the flights onward to Montreal.
Depending on whats available you will most likely have to open up to the idea of be more flexible and having that further stop and longer journey time if you want go by a secured reward booking.
You could always simply just pay the fares and try your luck at getting the upgrades with the points but that is always, a gamble… and you can only upgrade on Qantas operated flights which are will only available as like you said in Winter
As for the number of points it will cost to upgrade I’m unable to calculate this.
On another note & suggestion if you were to pay you could go with Air NZ from SYD-AKL-YVR-YUL. This gives you the stop in Vancouver, All in one booking. It comes in handy if you had any Star Alliance memberships (Air Canada is one) as you can credit miles earn’t to a respective account. Also depending on the age of your kids & to try something new there may be opportunity for a ‘Skycouch’ which is on special with Air NZ at the moment until the end of the month.
Keep in mind this part of the response disregards the opportunity to use your Qantas points, although I’m just choosing a lateral alternative especially if you were to pay.
If you can hold out until Winter you can get some availability for all 4 travelers at a total of 360k qantas points direct from Sydney-Vancouver. That’s an economy reward booking, not a reward upgrade.
I hope this feedback helps.