It's also about the fact that social standards and laws change. What may be perfectly acceptable today, could cost you your job or freedom in a few years.
Look at how many people lost their jobs, careers, and social standings because of old tweets that were resurfaced years later. That's not even legal changes, just social propriety changes. The internet used to be the Wild West. People would go out with guns blazing, hootin ' and a hollerin'. Then several years later we found ourselves in this socially progressive identity focused environment and shit people said in passing, likely without any thought at all, suddenly became part of their public identity.
Now imagine that there's a drastic change in the law. Today you're celebrating pride month, and 5 years from now homosexuality is outlawed retroactively... Seems preposterous, but it's really not. We need our privacy, from both the government and corporations.