Yup, that's how it works. The point I was making is that that's the only point of vulnerability, there is no way to gain access to every private key from some central point.
For example, let's say you collected a large amount of https traffic by listening in on wifi hotspots or such like, then you attacked a certificate authority and managed to gain access to everything they have. This would be very bad, but you wouldn't be able to decrypt the traffic you'd recorded because you still wouldn't have each of the private keys used. That's very different from the problem of RSA tokens. Gaining access to RSA's servers resulted in compromising every token. Decentralization is often key to security.
For example, let's say you collected a large amount of https traffic by listening in on wifi hotspots or such like, then you attacked a certificate authority and managed to gain access to everything they have. This would be very bad, but you wouldn't be able to decrypt the traffic you'd recorded because you still wouldn't have each of the private keys used. That's very different from the problem of RSA tokens. Gaining access to RSA's servers resulted in compromising every token. Decentralization is often key to security.