Renego: Popular, unpatched and vulnerable sites
By Yngve Nysæter Pettersenyngve. Thursday, May 19, 2011 7:22:23 PM
In the year or so we have been running the TLS Prober we have steadily expanded which tests are performed1, in addition to the checks for support of the TLS Renego patch. Thus, we have added tests for various standard compliance problems, for example.
Most recently, we added tests for TLS connection renegotiation, a test that has been delayed due to the significant changes that were needed in the TLS client implementation source. Renegotiation is used to change the encryption keys for the connection in midflight. This is the functionality that had the error that caused the Rengo Indication Extension patch to be necessary. Renegotiation can be used to add more confidentiality to the handshake, and many use this functionality when requesting Client Certificate authentication, in part to protect the user's identity.
Last week, we also added a lot of new hostnames to the list of servers (and more are being planned), by testing about 200 different hostnames for each domain in the Alexa top million list. The result was a 10-fold increase in number of names (from 2 million to 23 million, though it only resulted in a 50% increase in the number of unique servers (~380000 to ~590000), due to the number of IP addresses having multiple names assigned to them. Given the size of the list, I am considering running it only every month or so.
The current numbers indicate the following:
On their own, these numbers might not look that bad, but it gets worse once you look at these numbers in association with other information about these servers.
That 27.9% of servers accept renegotiation indicate a significant problem, since most of these sites do not need renegotiation, particularly not the client-initiated variant (no major browser client actually supports it). Part of the reason for the number of servers is that accepting renegotiation has been enabled in OpenSSL, and probably other SSL/TLS implementations, by default until the Renego problem was discovered, and Apache servers make up about 70% of the affected servers, but Microsoft IIS is registered for about 10% of the sites (but for both a front-end server may be handling the actual SSL/TLS connections).
Looking specifically at which of these servers are Renego patched, and which aren't, a really problematic pattern emerges: Only 0.5% of the sites allowing client-initiated renegotiation are Renego patched, 99.5% are NOT patched (for server-initiated renegotiation 71% of the servers are patched).
This is bad news, because it means that these 27.9% of all the tested servers are vulnerable to the full HTTP injection Renego attack, meaning that somebody could be manipulating what requests are sent to the unpatched "secure" website server you are visiting, and there is no direct way to tell if it is happening.
When you combine this with the fact that 45.7% of all servers are unpatched, you get an even more problematic number: Of the unpatched servers, 61% are vulnerable to the Renego attack. That is, roughly, only 2 of 5 (a bit more than 1 of every 3) unpatched sites are *apparently* safe from the attack. However, since the site could still perform renegotiation under very specific preconditions we are not able to use, we have no way to know for certain.
I do not know whether anyone has actually used this attack mode, short of the first Proof of Concept announcements, I have noticed no reports about it being used. While it is relatively easy to execute, it does require a bit of preparations, particularly being able to intercept the client's connection. My guess is that, if such attacks have been performed, they have been mistakenly been thought to be the result of phishing, rather than an injection attack. If the attack isn't used, it is probably because standard phishing and malware tactics work equally well, or better.
However, that an attack vector is (apparently) not used, does not mean that it is safe to leave it in place, particularly when the problem is so easy to detect, and to fix.
Then, there is what Ivan Ristic noticed about a month ago when he was running a new survey: there were fewer patched websites when their Alexa rank was higher, than when it was further down on the list. A check of our own numbers confirmed this: in parts of the Alexa top 1000 as few as 20% of the servers are Renego patched, and, in fact, even including the whole list of Alexa ranked sites we are testing, the coverage never exceeds the overall patch coverage in the set. This means that less popular sites must have a significantly higher patch coverage than popular sites, since the non-Alexa sites in the list are about 25% of the test set. If there should have been a difference between these groups, the percentages should, by rights, be the other way around; popular sites should have a much higher patch rate than less popular sites.
When checking this, and correlating to the servers accepting renegotiation, in parts of the Alexa list as many as 80% of the unpatched, high ranking servers accepts renegotiation and are vulnerable to the injection attack!
Lastly, there are the sites with Extended Validation certificates. Those ought to have their server properly patched, right? Wrong! Unfortunately. While 54% of all tested servers have been patched, only 50% of tested EV sites have been patched; that is, the patch coverage is about 90% of the general coverage. For the most part we see the same pattern for high-ranking sites as mentioned above, except that among the top 100 sites, the patch coverage is significantly higher than the general coverage in that segment, although far below the overall patch coverage.

While some high-ranking sites, most notably Google, have patched their servers, a large number have not, Facebook, Yahoo!, and Amazon are foremost among those that haven't patched.
Among the unpatched sites, Facebook and Amazon are playing it (relatively) safe, but we can unfortunately currently count Yahoo and many other popular sites as vulnerable. Although at least one major site has protected its main entry sites against the vulnerability, they haven't patched the problem, and other servers on the site were vulnerable.

These numbers definitely make me consider whether browsers should remove the EV indication, disable the padlock for Renego unpatched sites, or turn on the security warning, despite the fact that doing so would cause trouble for almost half the most popular websites. Unfortunately, at present, this is something that probably require coordinating with the other browsers.
A problem with this is that I believe a number of SSL accelerator companies still haven't released fixes; the latest news from one is that the patch will be released in June, which is not a moment too soon, if you ask me.
However, if you as a user want to get more visible indications about unpatched sites than the one in the Security Information dialog available from the security indicator, in Opera you can get this by updating the opera:config#SecurityPrefs|CryptoMethodOverrides preference with a sum of your selection of the following:
Keep in mind that this preference will get overwritten once the browsers start taking broader action against theses sites.
At present, I am not planning to publish the list of servers, but we have shared the current list of 193000 unpatched servers (of 267000) with the CAs that are members of the CA/Browser forum, including whether the site in question is vulnerable to the injection attack.
As for the websites: Facebook, Yahoo!, Amazon, and the rest, it is time to get down from the patch fence!
Most recently, we added tests for TLS connection renegotiation, a test that has been delayed due to the significant changes that were needed in the TLS client implementation source. Renegotiation is used to change the encryption keys for the connection in midflight. This is the functionality that had the error that caused the Rengo Indication Extension patch to be necessary. Renegotiation can be used to add more confidentiality to the handshake, and many use this functionality when requesting Client Certificate authentication, in part to protect the user's identity.
Last week, we also added a lot of new hostnames to the list of servers (and more are being planned), by testing about 200 different hostnames for each domain in the Alexa top million list. The result was a 10-fold increase in number of names (from 2 million to 23 million, though it only resulted in a 50% increase in the number of unique servers (~380000 to ~590000), due to the number of IP addresses having multiple names assigned to them. Given the size of the list, I am considering running it only every month or so.
The current numbers indicate the following:
- 28.1% of probed servers perform renegotiation.
- 0.2% of servers requested renegotiation during the simple tests we perform.
- This means that 27.9% of servers accept client-initiated renegotiation.
On their own, these numbers might not look that bad, but it gets worse once you look at these numbers in association with other information about these servers.
That 27.9% of servers accept renegotiation indicate a significant problem, since most of these sites do not need renegotiation, particularly not the client-initiated variant (no major browser client actually supports it). Part of the reason for the number of servers is that accepting renegotiation has been enabled in OpenSSL, and probably other SSL/TLS implementations, by default until the Renego problem was discovered, and Apache servers make up about 70% of the affected servers, but Microsoft IIS is registered for about 10% of the sites (but for both a front-end server may be handling the actual SSL/TLS connections).
Looking specifically at which of these servers are Renego patched, and which aren't, a really problematic pattern emerges: Only 0.5% of the sites allowing client-initiated renegotiation are Renego patched, 99.5% are NOT patched (for server-initiated renegotiation 71% of the servers are patched).
This is bad news, because it means that these 27.9% of all the tested servers are vulnerable to the full HTTP injection Renego attack, meaning that somebody could be manipulating what requests are sent to the unpatched "secure" website server you are visiting, and there is no direct way to tell if it is happening.
When you combine this with the fact that 45.7% of all servers are unpatched, you get an even more problematic number: Of the unpatched servers, 61% are vulnerable to the Renego attack. That is, roughly, only 2 of 5 (a bit more than 1 of every 3) unpatched sites are *apparently* safe from the attack. However, since the site could still perform renegotiation under very specific preconditions we are not able to use, we have no way to know for certain.
I do not know whether anyone has actually used this attack mode, short of the first Proof of Concept announcements, I have noticed no reports about it being used. While it is relatively easy to execute, it does require a bit of preparations, particularly being able to intercept the client's connection. My guess is that, if such attacks have been performed, they have been mistakenly been thought to be the result of phishing, rather than an injection attack. If the attack isn't used, it is probably because standard phishing and malware tactics work equally well, or better.
However, that an attack vector is (apparently) not used, does not mean that it is safe to leave it in place, particularly when the problem is so easy to detect, and to fix.
Then, there is what Ivan Ristic noticed about a month ago when he was running a new survey: there were fewer patched websites when their Alexa rank was higher, than when it was further down on the list. A check of our own numbers confirmed this: in parts of the Alexa top 1000 as few as 20% of the servers are Renego patched, and, in fact, even including the whole list of Alexa ranked sites we are testing, the coverage never exceeds the overall patch coverage in the set. This means that less popular sites must have a significantly higher patch coverage than popular sites, since the non-Alexa sites in the list are about 25% of the test set. If there should have been a difference between these groups, the percentages should, by rights, be the other way around; popular sites should have a much higher patch rate than less popular sites.
When checking this, and correlating to the servers accepting renegotiation, in parts of the Alexa list as many as 80% of the unpatched, high ranking servers accepts renegotiation and are vulnerable to the injection attack!
Lastly, there are the sites with Extended Validation certificates. Those ought to have their server properly patched, right? Wrong! Unfortunately. While 54% of all tested servers have been patched, only 50% of tested EV sites have been patched; that is, the patch coverage is about 90% of the general coverage. For the most part we see the same pattern for high-ranking sites as mentioned above, except that among the top 100 sites, the patch coverage is significantly higher than the general coverage in that segment, although far below the overall patch coverage.

While some high-ranking sites, most notably Google, have patched their servers, a large number have not, Facebook, Yahoo!, and Amazon are foremost among those that haven't patched.
Among the unpatched sites, Facebook and Amazon are playing it (relatively) safe, but we can unfortunately currently count Yahoo and many other popular sites as vulnerable. Although at least one major site has protected its main entry sites against the vulnerability, they haven't patched the problem, and other servers on the site were vulnerable.

These numbers definitely make me consider whether browsers should remove the EV indication, disable the padlock for Renego unpatched sites, or turn on the security warning, despite the fact that doing so would cause trouble for almost half the most popular websites. Unfortunately, at present, this is something that probably require coordinating with the other browsers.
A problem with this is that I believe a number of SSL accelerator companies still haven't released fixes; the latest news from one is that the patch will be released in June, which is not a moment too soon, if you ask me.
However, if you as a user want to get more visible indications about unpatched sites than the one in the Security Information dialog available from the security indicator, in Opera you can get this by updating the opera:config#SecurityPrefs|CryptoMethodOverrides preference with a sum of your selection of the following:
- 128: Disable EV for unpatched sites
- 1024: Disable padlock/secure indication for unpatched sites (implies Disable EV)
- 256: Warn about unpatched sites (implies disable EV)
- 2048: Disable support for server initiated renegotiation if the server is not renego patched. This protect against attacks that target the client (which is not that different from other MITM attacks), but may cause connections to hang or end with an error on unpatched sites that apparently require this functionality (about 0.1% of probed sites).
Keep in mind that this preference will get overwritten once the browsers start taking broader action against theses sites.
At present, I am not planning to publish the list of servers, but we have shared the current list of 193000 unpatched servers (of 267000) with the CAs that are members of the CA/Browser forum, including whether the site in question is vulnerable to the injection attack.
As for the websites: Facebook, Yahoo!, Amazon, and the rest, it is time to get down from the patch fence!








Mağruf ÇolakoğluZAHEK # Thursday, May 19, 2011 8:49:54 PM
Samed Ayanagresiff # Thursday, May 19, 2011 11:26:22 PM
Charles SchlossChas4 # Friday, May 20, 2011 12:11:19 AM
So what does the non patched servers mean for the every day person browsing he web?
Yngve Nysæter Pettersenyngve # Friday, May 20, 2011 2:05:01 AM
An unfortunately, there is no way for the user, or the browser, to reliably tell which is which, and currently 3 out of 5 unpatched servers is in the latter category
My advice: Check the renegotiation indicator (in the security information dialog) of secure sites you frequent, and then contact every site that is not supporting secure renegotiation and ask them to fix the CVE-2009-3555 issue.
Charles SchlossChas4 # Friday, May 20, 2011 3:56:25 AM
Krio LythKriolyth # Friday, May 20, 2011 7:15:31 AM
With the secret CryptoMethodOverrides flags set I was surprised to see the following:
http://files.myopera.com/Kriolyth/files/opera_renego.png -
Yngve Nysæter Pettersenyngve # Friday, May 20, 2011 10:10:32 AM
huxr # Friday, May 20, 2011 5:43:47 PM
Why there has been no coordinated action by browser vendors, or CA's, already?
After all, browsers should be the user's agent.
Sites are the customers of CA's but end-users are the customers of browser vendors.
I also wonder how this responsibility would stand legally. Of course, it's mostly the responsibility of Microsoft and Mozilla, as the two majority vendors. But also of Opera where it's a major market force.
Dan Alexandrudantesoft # Saturday, May 21, 2011 12:46:33 PM
Yngve Nysæter Pettersenyngve # Saturday, May 21, 2011 1:46:40 PM
The 45% (overall) percentage of unpatched servers is why using the warning level is not really an option today, except for testing ones own servers without too much hassle.
huxr # Saturday, May 21, 2011 4:15:25 PM
Originally posted by yngve:
Can you get the documentation guys to add the available flags to the settings file explanation page?
The different flags are getting too much scattered across different blog posts.
tomato42 # Saturday, August 6, 2011 2:21:17 PM