Yeah, it really depends on how much you trust the vendor.
Google? Say what you want about the company, but they'll never intentionally serve malware.
Random company with no track record where we don't even know who is maintaining the code? Much less trustworthy. The polyfill . io repo is currently owned by a Github user called "polyfillpolyfill" with no identifying information.
Third-party CDNs make less sense these days though. A lot of hosting services have a CDN of some sort. Most sites have some sort of build process, and you usually bundle all your JS and CSS (both your code and third-party code, often as separate bundles) as part of that.