Security @ Opera

In the year or so we have been running the TLS Prober we have steadily expanded which tests are performed¹, in addition to the checks for support of the TLS Renego patch. Thus, we have added tests for various standard compliance problems, for example.

Most recently, we added tests for TLS connection renegotiation, a test that has been delayed due to the significant changes that were needed in the TLS client implementation source. Renegotiation is used to change the encryption keys for the connection in midflight. This is the functionality that had the error that caused the Rengo Indication Extension patch to be necessary. Renegotiation can be used to add more confidentiality to the handshake, and many use this functionality when requesting Client Certificate authentication, in part to protect the user's identity.

Last week, we also added a lot of new hostnames to the list of servers (and more are being planned), by testing about 200 different hostnames for each domain in the Alexa top million list. The result was a 10-fold increase in number of names (from 2 million to 23 million, though it only resulted in a 50% increase in the number of unique servers (~380000 to ~590000), due to the number of IP addresses having multiple names assigned to them. Given the size of the list, I am considering running it only every month or so.

The current numbers indicate the following:

• 28.1% of probed servers perform renegotiation.
• 0.2% of servers requested renegotiation during the simple tests we perform.
• This means that 27.9% of servers accept client-initiated renegotiation.

On their own, these numbers might not look that bad, but it gets worse once you look at these numbers in association with other information about these servers.

That 27.9% of servers accept renegotiation indicate a significant problem, since most of these sites do not need renegotiation, particularly not the client-initiated variant (no major browser client actually supports it). Part of the reason for the number of servers is that accepting renegotiation has been enabled in OpenSSL, and probably other SSL/TLS implementations, by default until the Renego problem was discovered, and Apache servers make up about 70% of the affected servers, but Microsoft IIS is registered for about 10% of the sites (but for both a front-end server may be handling the actual SSL/TLS connections).

Looking specifically at which of these servers are Renego patched, and which aren't, a really problematic pattern emerges: Only 0.5% of the sites allowing client-initiated renegotiation are Renego patched, 99.5% are NOT patched (for server-initiated renegotiation 71% of the servers are patched).

This is bad news, because it means that these 27.9% of all the tested servers are vulnerable to the full HTTP injection Renego attack, meaning that somebody could be manipulating what requests are sent to the unpatched "secure" website server you are visiting, and there is no direct way to tell if it is happening.

When you combine this with the fact that 45.7% of all servers are unpatched, you get an even more problematic number: Of the unpatched servers, 61% are vulnerable to the Renego attack. That is, roughly, only 2 of 5 (a bit more than 1 of every 3) unpatched sites are *apparently* safe from the attack. However, since the site could still perform renegotiation under very specific preconditions we are not able to use, we have no way to know for certain.

I do not know whether anyone has actually used this attack mode, short of the first Proof of Concept announcements, I have noticed no reports about it being used. While it is relatively easy to execute, it does require a bit of preparations, particularly being able to intercept the client's connection. My guess is that, if such attacks have been performed, they have been mistakenly been thought to be the result of phishing, rather than an injection attack. If the attack isn't used, it is probably because standard phishing and malware tactics work equally well, or better.

However, that an attack vector is (apparently) not used, does not mean that it is safe to leave it in place, particularly when the problem is so easy to detect, and to fix.

Then, there is what Ivan Ristic noticed about a month ago when he was running a new survey: there were fewer patched websites when their Alexa rank was higher, than when it was further down on the list. A check of our own numbers confirmed this: in parts of the Alexa top 1000 as few as 20% of the servers are Renego patched, and, in fact, even including the whole list of Alexa ranked sites we are testing, the coverage never exceeds the overall patch coverage in the set. This means that less popular sites must have a significantly higher patch coverage than popular sites, since the non-Alexa sites in the list are about 25% of the test set. If there should have been a difference between these groups, the percentages should, by rights, be the other way around; popular sites should have a much higher patch rate than less popular sites.

When checking this, and correlating to the servers accepting renegotiation, in parts of the Alexa list as many as 80% of the unpatched, high ranking servers accepts renegotiation and are vulnerable to the injection attack!

Lastly, there are the sites with Extended Validation certificates. Those ought to have their server properly patched, right? Wrong! Unfortunately. While 54% of all tested servers have been patched, only 50% of tested EV sites have been patched; that is, the patch coverage is about 90% of the general coverage. For the most part we see the same pattern for high-ranking sites as mentioned above, except that among the top 100 sites, the patch coverage is significantly higher than the general coverage in that segment, although far below the overall patch coverage.



While some high-ranking sites, most notably Google, have patched their servers, a large number have not, Facebook, Yahoo!, and Amazon are foremost among those that haven't patched.

Among the unpatched sites, Facebook and Amazon are playing it (relatively) safe, but we can unfortunately currently count Yahoo and many other popular sites as vulnerable. Although at least one major site has protected its main entry sites against the vulnerability, they haven't patched the problem, and other servers on the site were vulnerable.



These numbers definitely make me consider whether browsers should remove the EV indication, disable the padlock for Renego unpatched sites, or turn on the security warning, despite the fact that doing so would cause trouble for almost half the most popular websites. Unfortunately, at present, this is something that probably require coordinating with the other browsers.

A problem with this is that I believe a number of SSL accelerator companies still haven't released fixes; the latest news from one is that the patch will be released in June, which is not a moment too soon, if you ask me.

However, if you as a user want to get more visible indications about unpatched sites than the one in the Security Information dialog available from the security indicator, in Opera you can get this by updating the opera:config#SecurityPrefs|CryptoMethodOverrides preference with a sum of your selection of the following:

• 128: Disable EV for unpatched sites
• 1024: Disable padlock/secure indication for unpatched sites (implies Disable EV)
• 256: Warn about unpatched sites (implies disable EV)
• 2048: Disable support for server initiated renegotiation if the server is not renego patched. This protect against attacks that target the client (which is not that different from other MITM attacks), but may cause connections to hang or end with an error on unpatched sites that apparently require this functionality (about 0.1% of probed sites).

Keep in mind that this preference will get overwritten once the browsers start taking broader action against theses sites.

At present, I am not planning to publish the list of servers, but we have shared the current list of 193000 unpatched servers (of 267000) with the CAs that are members of the CA/Browser forum, including whether the site in question is vulnerable to the injection attack.

As for the websites: Facebook, Yahoo!, Amazon, and the rest, it is time to get down from the patch fence!

[Note: This is a updated version of the Opera Labs article from 2006]

A large number of people are looking for security problems and vulnerabilities in Opera and other applications. Some people do not like this, but we think it is fine. When done responsibly, this can increase security for the application and is thus of great benefit to users. However, when done irresponsibly, the activity can cause needless alarm, wasting the time of application vendors and end users.

An example of irresponsibility is how they report and classify the issues they find, and how the issues are communicated to the application vendor.

The primary problems we see with some of the reporting is that:

• Many reports are about problems that may be bugs, even significant stability issues, but they are not really security issues.
• Some reports exaggerate the severity of the problem.
• Some reporters do not give us an opportunity to check the problem, verify that it is indeed a security issue, and release a patched version before they publish their findings.

The terms "security issue" and "vulnerability" cover a large number of serious problems within an application.

Security real issues list:

• A real issue makes the application reveal the user's protected information, such as file system information, user name, passwords, etc., without authorization.
• A real issue makes the application mislead the user about what is going on.
• A real issue makes the application execute arbitrary code.
• A real issue uses weak protocols when it is possible to use stronger protocols.

Those are just some of the possibilities. As important is what should NOT be called a security issue.

Security non-issues list:

• A non-issue involves crashes that do not prevent the restart of the application, and do not lead to an execution of arbitrary code.
• A non-issue involves attacks that permanently disable the application.
• A non-issue states that users are not able to access or use their online banking sites, shopping sites, etc. due to site specific problems
• A non-issue states that Web sites claim cookies do not work.
• A non-issue states that the application uses stricter security protections than another competing application.

Sometimes, reporters incorrectly classify instances from the non-issues list as issues from the real issues list.

The two main severity concerns prone to be exaggerated are allegations that a problem "can be used to execute arbitrary code" or is a "Denial of Service" (DOS) attack.

It seems that the combination of a very large input and a crash leads some people to automatically think the problem may be used to "execute arbitrary code", that is, take over the user's computer. That is not always the case, and is usually only possible for what is called a stack buffer overflow, even though it is also sometimes possible for heap (allocated) buffer overflows. However, we have seen this phrase used even for ordinary "NULL pointer" crashes that had no possibility whatsoever to be
abused.

Another exaggeration is calling any crash that can be triggered by a Web site a "Denial of Service" attack. In our opinion, crashing a server which could potentially be running an important or critical service can rightly be called a Denial of Service, as long as the server is not able to automatically recover. This is because bringing down the server will deny service to all users.

Crashing a client, while it is an undesirable event and one we want to fix quickly, can only be achieved by enticing the user into accessing a document that crashes the client. Even then it will usually only inconvenience the user long enough to restart the application. That is not sufficient cause to call it a Denial of Service; in most of these cases it is a stability issue.

The only way, in our opinion, that a crash in a client can be called a Denial of Service is if the attack causes the client to crash again when it is restarted, and when fixing the problem requires more than changing the immediate startup configuration (previous session or blank page). Consequently, this requires editing of the configuration files or a complete re-installation.

In our opinion, calling a normal crash problem in a client a "Denial of Service" vulnerability is a devaluation of the meaning of the term "Denial of Service", which eventually may rob it completely of its correct meaning. Much in the same way, the word "hacker" is now considered to mean "computer criminal", not "highly skilled computer specialist", which was its original meaning. Please see http://en.wikipedia.org/wiki/Hacker_(computing). One may ask if an attack that disables the client really is a
security problem. After all, it does not compromise the user's information security, privacy, or the system's integrity. Generally, we do not consider ordinary crashes, freezes, and other so-called "Denial of Service" attacks security problems; they are stability issues.

Other browser developers have also seen this kind of exaggeration, and Gerv of Mozilla apparently shares our opinion about calling crashes Denial of Service. While we think Microsoft is stretching the definition of Denial of Service, their situation is a bit more complex, which may justify using the definition in their case.

We would like to use this opportunity to ask the security research community to revisit and update/clarify the meaning of their classifications for the various contexts where they are used. In particular, those who maintain vulnerability databases should consider the actual severity of a reported problem before posting the information.


"I think I have found a security issue in Opera. What should I do?"

We recommend: Report it to us before you publish it, so we can fix the problem and release an updated version as soon as possible. We take security seriously, and will investigate all reports.

You should file the report at https://bugs.opera.com/wizard/, with the severity "Security Issue" and you should fill in all information you have, or suspect, about the problem:

* Give a clear and concise explanation of the problem, as complete as possible, including why it is a security issue if that is not obvious.
* Identify which version(s) of Opera you have tested, including which operating system versions. At least one of the tested versions should be the most recent release for the affected platforms.
* Document what is needed to replicate the problem with a step-by-step procedure which includes the source code or command line operations.
* Provide an e-mail address where we can contact you in case we need more information, and to keep you updated about our progress.
* Provide actual test cases. The test cases should contain only the minimum complexity required to reproduce the problem.
* Provide references to previous discussions about the problem area, and if it has occurred in other software.

You should always file a bug report about the issue, even if you are going to send information to the Security Group via e-mail (see below). The report will be filed in our system, and it will remain in the system even if your e-mail should go astray.

Unless they are very short, the test cases should be attached to the bug report by using the e-mail address you get after you submit the bug report.

Server-side test cases should, if there are no special reasons to use other formats, be written in Perl, PHP, or Python. Aside from using standard libraries, they must be self contained.

To protect the data integrity of the test cases, you should PKZip/WinZip (.zip) or Tar/Gzip (.tar.gz) the test cases, even if they are single files.

If you further wish to protect the test cases you may PGP encrypt them to security (at) opera dot com using the public key below, before attaching them to the bug report.

We recommend that if you are planning to publish your report, the information you submit to us must at least include all the information you are going to publish, preferably more. We have occasionally received reports that contained very limited information as compared to what was eventually published, and therefore it was only after publication we were able to understand the true severity of the problem.

Allowing the application vendor to fix the issue before publishing the advisory is good for the users of the application. There is no point leaving end users vulnerable to attack. We at Opera will cooperate and verify the advisory; we can also point out if the issue is not exploitable, helping to increase the quality of the advisory. As a matter of principle we always try to give due credit to responsible reporters in our changelogs or press releases.

Happy, and responsible, hunting!


To contact the Opera Security group, send an e-mail to security (at) opera dot com , whose PGP key is below. As noted above, we prefer that Security Issues are reported via our Bug tracking system, even if you do send an e-mail about the issue.

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: PGP 8.1
 
mQGiBELOLzkRBADVw74D8uj9bPNzgmfgf9tC72ckgqnxGHrRTMMK9y42XiLXiIYp
B66OjYpPiwgWzNqivcuyhc0HUGHZeezyE1wK4qGZ0cvxYuWrod2B0xyIXcDg6jTt
Vr/KgbfZy1RhLb9CPW3ngH28wbKAGud7dOsCjqvC+RWXAqPzN9ZXT6S3AwCg/+q6
pzhHdFMjvWrmRuWnypwO3mcD/iDbfiKe7c4nJs+yUveVjEkP+2f/zNg/3lSNUjBO
sg9CaXQlC7QCsTLEmDWAmjlebTI/DVlOfnGYa22myj1KLZdx5/uUG0aZUWDMOkKn
MjqLSyFRqb4VwSjetoojGcHob5zJQUeZ3+q4fSt66kWRwxARmnP3fyeHtdt+ddWa
KkMlBACkj3ZjnORHkyFQDru88IcZKgzTuQzhMcYmCp9vd+gMAy4+r3vUEq5EnXeg
ydBVBzReI4bDizjvpwUwbCEOn+ei8n95x2n7DHEjpn2THasb1ENkVfZ1iHi4OnO3
DaqvnXwNhnobEfEEcLjQDvMEjfWx/T1fZ6I1FwXJZr7TzwvJIrQpT3BlcmEgU2Vj
dXJpdHkgR3JvdXAgPHNlY3VyaXR5QG9wZXJhLmNvbT6JAF0EEBECAB0FAkLOLzkH
CwkIBwMCCgIZAQUbAwAAAAUeAQAAAAAKCRB2h/HMa2j6eQKoAKCv6+UWqdumoTjy
bYL5fzdVlUm8BgCdFX2FFnsyd1O9tSszh7JQ20is1065Ag0EQs4vORAIAPZCV7cI
fwgXcqK61qlC8wXo+VMROU+28W65Szgg2gGnVqMU6Y9AVfPQB8bLQ6mUrfdMZIZJ
+AyDvWXpF9Sh01D49Vlf3HZSTz09jdvOmeFXklnN/biudE/F/Ha8g8VHMGHOfMlm
/xX5u/2RXscBqtNbno2gpXI61Brwv0YAWCvl9Ij9WE5J280gtJ3kkQc2azNsOA1F
HQ98iLMcfFstjvbzySPAQ/ClWxiNjrtVjLhdONM0/XwXV0OjHRhs3jMhLLUq/zzh
sSlAGBGNfISnCnLWhsQDGcgHKXrKlQzZlp+r0ApQmwJG0wg9ZqRdQZ+cfL2JSyIZ
Jrqrol7DVekyCzsAAgIH/1nn7DeOo79hX8V3cgGRbHDyw/CC251salCt5gislPdV
RaUAyAC24m2MprPUr6JV6QfmmXgt/FYYbdscREPkbGplXkZFSDlcH8PKfzDDaewG
dApQhOCg4bsFNUX46l/kcooAjlihLZbb9XUJ0IqIvbxJze0NtHD6o7ABTQeEagK+
dL/CBlYxMWG4SvCCol88IjsdkwpCrpIMcnh1lGPMEfvCt86hk96Y1jUouve1sLGx
zx9QbCqo8TkJyi5bJhCGEHS90nEQsdHmaS+Grvm3eep9dTLxhn+Aj/WTpdUy8VD0
rwPAEKWOo4V4HdNEgGjSNOV8LSp8/2wZR0z7zZc3vYeJAEwEGBECAAwFAkLOLzkF
GwwAAAAACgkQdofxzGto+nnZ1ACg9xn8tVb99DfZsC6wRGzr7dtfMU0An1aY3FZ7
k3eRlyzdgO2F+WOHux4J
=w+QX
-----END PGP PUBLIC KEY BLOCK-----

View the PGP key on our secure server