I makes some sense. If you're using SSL and someone is trying to snoop on your connection. If you visit a non-SSL site from an SSL site and the referrer gets sent then the snooper knows the URL of the page you were on last. Whereas if the target site is SSL then no information is leaked to listeners.

The point of SSL is to protect from snooping, not as a general privacy-protection mechanism that should have everything privacy-related shoehorned into it.