You're assuming your app has to be served as one large chunk of stuff. There are many JS libraries for asynchronously loading additional content, and it is simple enough to use one of these (or implement your own) so that your app is served as one small bootstrap and a bunch of additional content that is loaded as needed.
You say "static sites can spit back pages in like 200ms (across the network!!)". This time is entirely dominated by network latency. There is no way you're getting data to the client in 200ms if they're based in Australia and you're in, say, east coast US. With the client-side model you can push your app's bootstrap to a CDN to get ~200ms latency just about anywhere. You can render the UI and show a "Loading data" spinner to hide the latency of the initial request to your application server. This gives the user the appearance of a snappy site even though network latency might be 1s or more. You can't do this if you go the server-side route.
I'm not making any assumptions, I'm telling a story about what I watched happen first hand. As you say, you can split up your assets, that does help significantly. You can also put stuff on CDNs, that helps significantly too. Believe me, Twitter engineers are super smart and they've done all of this. But at the end of the day if your business model depends on people looking and interacting with your content, or if you just want your site to be super fast, having a loading spinner while you wait and download the rest of your application in the background isn't going to cut it.
You say "static sites can spit back pages in like 200ms (across the network!!)". This time is entirely dominated by network latency. There is no way you're getting data to the client in 200ms if they're based in Australia and you're in, say, east coast US. With the client-side model you can push your app's bootstrap to a CDN to get ~200ms latency just about anywhere. You can render the UI and show a "Loading data" spinner to hide the latency of the initial request to your application server. This gives the user the appearance of a snappy site even though network latency might be 1s or more. You can't do this if you go the server-side route.