First of all, there is the security aspect to half of the web relying on a single source of information, though these days it become common to include the "integrity" attribute when referencing the file from a CDN so you ensure that the hash of the file matches what you expect it to be. However, the importance of including this attribute is not always made clear when giving end users quick and easy JS snippets to paste into their site to use the library.
Then we have the idea of relying on a single source to return the file. Sure, it's a "CDN", so the content is distributed across many servers but ultimately, this is still somewhat of a single point of failure in a lot of cases. If a CDN starts to have issues for whatever reason, then your site stops working, even if all of your servers are functioning correctly, you can now start to fail based on this external variable that is out of your control. It's also an unknown to you as a server admin, you don't really know anything about this CDN, you just assume it works. If it doesn't work, then how do you know it's not working?
To make matters worse, thanks to the concept of a CDN they are often geolocated. This means that if something goes wrong with a CDN in a particular region, your site will only stop working for a small portion of your users. You may not ever realise that your site was not working at some point for some of your users or worse, waste a ton of time debugging complaints from a subset of users, without ever figuring out what went wrong.
But the performance gains!
Yeah, that's really the big meme that everyone likes to use to justify using a CDN for lots of their common JS libraries like jQuery. People will already have the exact version of jQuery we are including in their cache from other websites that use the same CDN, so it'll be an instant load! How common do you really think that is? Seems to me like we're pulling at straws to justify using a CDN here. Oh by the way, that whole concept is going away because it turns out a shared cache is a terrible idea as it leaks private information about your users.
So even without the instant return due to a shared cache of the resource, we're still saving all that time because of the geolocation aspect of a CDN! Are we really though? Is that actually worth it, does it really make a difference?
Let's say you have a website which requires jQuery and you want to figure out if you should load it from a CDN or from the same server that you run your website from. Let's say that your server is located in Europe and your test user is in Australia. Your Australian user will probably have a latency of around ~300ms to a European server. So let's go with that, 300ms.
Now your user makes a request to load /index.html, it takes 300ms to return index.html, along with its contents. In the contents, it references the public jQuery CDN in a script tag. So now the browser goes and loads that. Thanks to geolocation, getting the script from the CDN only takes 20ms as it uses a local Australian server. So our total time to first paint is around 320ms.
When we run through this exact same example but we self-host the jQuery library, the first paint will be 600ms. This is because it'll take ~300ms to return the JS library from our own server. That's a significant increase and we all know how important that first meaningful paint is. With subsequent loads however, the time it takes to load the JS will be 0ms because we'll leverage browser caching. So this extra 300ms load time is only on the very first load. Any webserver like nginx with some basic static asset cache configuration will do this.
So really, we've only acquired this benefit on the very first load of our website. Additionally, browsers make requests in parallel so there isn't really any scaling aspect to this benefit. That is to say, you can include several more JS/CSS libraries and it won't really increase the time it takes to load the files as it will run those requests in parallel. So really, all this CDN is giving you, in this almost worst case scenario, is about 300ms of time saved and only the first time they load the website, every subsequent load there is no time saved thanks to your webserver's caching setup.
Is it really worth it? I don't know about you but it doesn't sound worth it to me. Sounds like a whole lot of additional complexity without much gain.