The rest of the article seems to be talking about Canadians living abroad. So only a few tens of thousands have to leave annually to get to that total amount. I think that first sentence is just poorly written.
Thinking of the number folks I know who went to the US for retirement, work, or to get married, it seems possible.