Skip navigation.

Implementer's notes

What might get caught in the gears under the hood?

Posts tagged with "Security"

American Express used revoked site certificate for weeks

, , , ...

In the past couple of weeks I have seen a few reports that the American Express site https://online.americanexpress.com/ (Testurl) was using a revoked certificate. This site redirects to various American Express services.

As of this morning (October 1st) the problem has been fixed.

As I have mentioned before, a revoked certificate is no longer valid, and therefore no browser should give access to the site. In the worst-case scenario, a certificate is revoked because the key has been compromised, and the site is being used by criminals for fraudulent purposes. Revocation of a certificate is analogous to blacklisting of a credit card because the card can be abused.

The available (human readable) information indicates that the current situation was not a worst-case scenario. What seems to have happened is that American Express requested the current certificate in May, then shortly thereafter requested an updated certificate, possibly to correct a minor mistake. The old certificate is revoked automatically after a reissue, and the Web site is expected to use the newer certificate, but it seems that American Express for some reason did not update their server. Since there is no way for a browser to distinguish between a benign mistake and a worst-case scenario; we have to assume the worst.

I am sure somebody is going to mention that the site worked without problems in many other browsers. Yes, it did, but that was because the certificate for this site is issued from a special Verisign certificate hierarchy (which is being retired) that does not specify the revocation list (CRL) download location, so browsers cannot go looking for the information, based on the certificate. The reason why the CRL is not specified may, as I understand it, have to do with the fact that the hierarchy was never intended to be used for Web site certificates, but for authentication of dedicated secure server-to-server connections. Opera may be one of just a few browsers that actually check the CRL for this hierarchy, but we can only do this because we have added the certificate to a list of overrides our repository that specifies where to fetch the CRL, otherwise we would not show a padlock on these sites (and as we discovered last year, Opera users really do check the padlock!).

A partially unanswered question is why reports appeared only so recently. Part of the reason is that a bug in our rootstore repository server caused the CRL override Opera uses to be dropped for 6-8 weeks (we fixed that two weeks ago), but that does not explain why there are no reports before the end of July. My guess, based on information from Netcraft, is that online.americanexpress.com is a new server that was brought online just a few months ago, which turns out to be well into the period when most of the CRL overrides were disabled by the repository bug, which means we did not check the CRL either for this site during that period :o: .

Once we learned of the problem, beginning of last week, we informed Verisign about it, and I assume they informed the American Express contact for this certificate about the problem.

Yet, more than a week later the revoked certificate was still being used by the Web site, until it finally got fixed last night. Frankly, I expected better from American Express.

Announcing the Security @ Opera home page

, ,

The Opera Security group just got its own home page.

"Security @ Opera" is where we will post information about security advisories, as well as occasional articles about security related topics.

The first article is about the recent developments concerning MD5 in SSL/TLS certificates.

W3C Web Security Context: User Interface Guidelines

, , , ...

The W3C's Web Security Context Working Group have just released the Last Call version of its "User Interface Guidelines" document, which is a set of recommendations for the security related UI in Web User Agents.

This specification deals with the trust decisions that users must make online, and with ways to support them in making safe and informed decisions where possible.



This document specifies user interactions with a goal toward making security usable, based on known best practice in this area. Subsequent testing of this specification will include conformance, interoperability, and usability testing.



If you want to comment on the document you are welcome to do so:

The W3C Membership and other interested parties are invited to review the document and send comments to public-usable-authentication@w3.org (with public archive) through 15 September 2008. We appreciate if comments follow these guidelines for writing good issues.




"Nobody checks the padlock" debunked by Opera users

, , , ...

There's been a number of criticisms directed at the padlock over the years, some of which may be correct, to some extent, at least.

One objection has been that users misunderstand the padlock's meaning, thinking it is an indicator of trustworthiness, rather than a protection rating for the connection.

Another objection (which flies in the face of the first), is that "nobody checks the padlock".

Well, I can't say much about the correctness of the first objection, but the past couple of weeks a growing number of Opera users have been working hard at debunking the second one.

In Opera 9.50 we added some new security features, and changed a some others. Two of these are the following:



Both of these turned out to encounter unexpected problems, not due to bugs in our implemenation, but at the CA side. In all cases the problem cause full validation of the certificates to fail, and as a result Opera reduces the security level of the connection so that a padlock will not be displayed. Quite a lot of people noticed.

For OCSP we encountered problems with at least three different brands of OCSP responders. These responders did not respond correctly when we sent a request for infromation about a certificate. It wasn't until a couple of weeks ago that the vendor for one of them discovered the reason for the problem, their expectation of input format was wrong. According to my information they have developed a patch, and I assume they are doing QA on it now, before sending it to the two (or more) CAs that are affected by this. The others have shown up recently and I do not have good information about the causes.

With CRLs we encountered two problems. Just before 9.50 was released we got the first reports about the first case, but it was not until a few days after the release that we got an overview of the situation.

It turns out that one specific CA created an intermediate certificate a few years ago for one of their certificate hierarchies. This certificate included an incorrect URL for where we should go to fetch the CRL, so when Opera fetches the CRL it gets a CRL created for a different hierarchy than the one being verified. As a result Opera's ceritificate validation code can't find the right CRL, and this step fails. This certificate became installed on thousands of servers before the mistake was discovered, and despite a campaign to replace the certificates, several hundred sites (many of them banks) still use the old certificate.

In another case, which came to light last week, another CA have issued an intermediate certificate directly below their root that does not contain a URL for a CRL, but the sub-ordinate certificate issued from that intermediate does specify a CRL. As Opera's certificate validation code expects that if one certificate have a CRL they all use CRLs, validation of the topmost intermediate certificate fails because it can't find the CRL for that certificate.

What can be done about this?

  • About the OCSP issue, we wait for the vendor patch, but in Opera 9.51 we have added an override that lets us use the certificate update system to specify which OCSP URLs that must use the old POST method. In addition one CA also deployed a workaround.
  • About the first CRL issue, the websites need to update their installed certificates, in the second case the CA must also issue a new updated certificate. While waiting for that, in 9.51 we are adding a similar override mechanism as for OCSP, by specifying extra CRL download locations for specific CAs. In the second case we might also be able to fix the issue if the CA has a root for that particular CA name by distributing it to all installations that access the affected sites (that is not a realistic option for the first case).


These changes will be active in the upcoming Opera 9.51 RC2 (Update: Now available), and is already active in the online certificate repository. You will get the updates the next time Opera checks for updates, or if you use Help->Check for updates.

Update July 9: There are some sites with certificates that does not provide any CRL or OCSP information in the site certificate, while the rest of the certificate in the chain have CRLs. Due to a minor bug, and restrictions in the override functionality, the override for that will not become active before 9.52 has been released. At present we are only aware of 10-20 sites affected by this.

More testing: Updated EV information and new Roots

, , , ...

A few hours ago the new online certificate repository that the most recent weeklies are using was updated with several new roots, and an additional CA, Comodo, was also provisionally EV-enabled

The new and the updated CAs are

  • America OnLine
  • Cisco
  • Comodo
  • Digicert


There is no need to download an updated Weekly (if you have one of the two recent ones). When you next restart one of the Opera 9.50 Weeklies with support for online certificate updates, it will immediately download the indexes, and download the new certificates when necessary. Please give it a minute to finish the update.

Here are a couple of testsites:


Known Issues: The complex certificate chain system used by Comodo encounters some, mostly hidden, problems with our OpenSSL certificate verification support, and that will cause some EV sites to not be recognized. We will try to fix it, but it may not be advisable to include a fix in 9.50.
[*] DigiCert:


I do not currently have testcases for Cisco, as they have not yet started issuing certificates from the new root.

More about known issues:

  • We know there are some problems with OCSP and CRL responses (the two kinds of revocation information) from some Certificate Authorities. These problem may lead to the website getting a lower security level. We are looking into these problems together with the CAs. In last week's build some of the CRL problems will cause a "Fatal Error 50", in the most recent build that has been fixed. We may decide to work around some of these, but they should preferably be fixed by the CA.
  • At least one CA (who is not in our repository) is using CRLs with a critical extension, which will cause the secure connection to fail with error code 554. In this case we are following the standard, although one might wonder why the specification says "Although the extension is critical, conforming implementations are not required to support this extension". The problem have been "fixed" internally by recognzing the extension, then ignore it, as we do not need it.
.

New in Kestrel: Faster Root Certificate updates

, , , ...

Extended Validation (EV) is not the only new feature in the most recent Weekly; we've also improved the certificate database by making it able to download new Roots automatically. This means that we can add new Roots to the Certificate Authority database without requiring users to update their installation.

Root Certificates are one the fundamental pillars of Internet security. They are used to confirm the identity of secure webservers by acting as a trusted third party. The introduction of a third party is what makes it possible for two parties that have had no previous contact to determine who the other is, and, based on that, establish an encrypted connection. That is the true "magic" of Public Key Cryptography.

The problem with Root Certificates has been that before they can be used, each relying party (in particular, TLS clients, as in our case) must have a copy of it. If a party doesn't have a trusted copy of the Root it cannot verify the certificate that the other party sent, and can therefore not make any statement about the identity, nor establish a secure connection to that other party.

In SSL/TLS clients like Opera, much of this initial problem has been handled by the vendor shipping a list of trusted Roots with each installation, as well as some update mechanism to add new Roots when necessary. In Opera, this update mechanism has until now been rather crude, as a new version of Opera would have to be installed by each user.

In this Weekly, we are adding an automatic certificate download capability that works like this:

  • A number of frequently used Roots are still shipped with Opera.
  • If a Web site presents a certificate issued from a Root that is not in the local Certificate store, but is available in the online repository, it will be downloaded and installed in the repository. (Please note that this means that if you delete a certificate rather than marking it as untrusted, it will be downloaded again if necessary.)
  • If a Root is added to the repository, and is closely associated with another Root, we can instruct all Opera instances to download that root if they have the other Root. This is particularly important in relation to how EV certificate chains have to be organized.


The certificates are downloaded from a repository hosted at https://certs.opera.com/, which is also the server hosting the EV information, and the information is refreshed every 6 hours. The certificate files hosted on this server, whose names are constructed from a SHA-256 digest of the CA name and other information that uniquely identifies them, are all digitally signed to prevent forgery.

A separate list in the repository identifies all the certificates that are included in the repository. This list is used to stop Opera from checking the repository for unknown issuers. The list is currently retrieved every time Opera starts, but will later be checked when Opera checks for other updates, that is, once a week when Opera starts.

We are particularly interested in your experiences with this new functionality and would like you to test it in various environments, such as:
  • performing an upgrade of a used test installation of 9.2x
  • clean installs,
  • whether or not secure sites that worked in 9.2x and previous Kestrel Weeklies work OK in this build.


Also, somebody may ask, "Does this mean Opera now have the ability to automatically remove a Root certificate?". Yes, in extreme cases, such as the unlikely event (I hope) of a Root Key compromise, we now have the ability to do what previously would have required an emergency security update.
We also have the ability to add certificates to the new "Untrusted" certificate store, which might be necessary in cases where an important certificate has been issued in error.

New in Kestrel: End of the Extended (Validation) wait

, , , ...

Today we're releasing the first Kestrel Weekly with Extended Validation (EV) support.

As I've written before, Extended Validation is a new way to indicate, in a Web site certificate, that the identity of the company behind the Web site has been verified according to a rigorous standard. The guidelines for this process is defined by the CA/Browser forum, which is a group consisting of many certificate issuers (CAs), browser vendors, and others.

We are now ready to start public alpha testing.

So, how does it work?

When a Web site is recognized as an EV site, Opera will change the background color of the security toolbar to green, instead of yellow as it is for normal secure Web sites. There will be a few further adjustments done in this area, the most major being that we have stopped displaying the Organization Name field and country in the security toolbar for non-EV sites, because this name might not have been properly verified for non-EV certificates.


What is required to be permitted to issue EV certificates?

Root CAs that want to issue EV certificates must first pass a rigorous audit to prove that they have the proper procedures in place to identify accurately the company requesting a certificate, and that this company is in control of the given server. Further, an agreement between the Root CA and the browser vendor is required before the browser can recognize the CA's certificates as EV certificates.

How does Opera know it is an EV certificate?

All EV certificates contain an identifier (called an EV-OID, actually a Certificate Policy identifier). The CA will insert this identifier when it has verified all the information. Only certificates issued from specific Roots are allowed to use these identifiers.

How does Opera know that a Root is allowed to issue EV certificates?

Each Opera instance will regularly download a digitally-signed list identifying the CA, its certificate, and which EV-OID(s) it is permitted to use (different CAs can use different EV-OIDs). When Opera verifies a certificate issued from this Root, it will "sift" through the data where the EV-OIDs are stored to see if it contains one of the EV-OIDs recognized for this Root. If such an EV-OID is present, Opera will proceed to check if the other requirements (see below) for saying the Web site is an EV site are fulfilled.

What is required by a Web site to be considered an EV site?

The primary requirement is, of course, that it is eligible to get an EV certificate, and that it has purchased and installed one. The certificate, and all the intermediate certificates must (of course) still be valid, and cannot have been revoked by the issuer (we now check both OCSP and CRLs). Further, there must be no problems verifying the certificate; that is, there must be no unknown issuers, no mismatch of server name, and no weak encryption.

There are also a couple of specific encryption key requirements: The Web site's key should (for RSA) be at least 2048 bits long, but non-root certificates which expire before 23:59:59 UTC/GMT December 31, 2010 MAY use RSA keys of at least 1024 bits. Except for the Root key, which (for RSA) must be at least 2048 bit long, all signing keys must be at least as long as the key of the certificate it is used to sign.

And lastly, if the Web site includes content from other Web servers, those servers must *also* be hosting EV-sites. This is a point at which we are much stricter than the other EV-capable browsers currently. As I said earlier: "It ain't EV 'til it is EV, all EV".

The reason for this requirement is that these other servers may provide content that directly controls the appearance of the entire site, either through frames or external Ecmascript embedded in the page (and the latter has full control of the site's content).

Considering how many click-wrap licensed Web services (such as for Web statistics) there are, it is not likely that the Web designers signing their sites up for these services will have done anything close to good-enough legal identity check of the service. Also, don't forget all the liability disclaimers such contracts include. It is impossible to check the contracts, but we can check that the sites have been able to get an EV certificate.

EV is intended to give you better information about who provided the content with which you are viewing and interacting. If not all the servers providing the content are providing this kind of information, do you really know who provided the important part of the content? Our answer is that you don't, therefore such sites, which at the moment include Paypal.com, will not get the EV indication unless they either remove all references to the non-EV content, or those providers upgrade to provide EV content, as well.

Which Root CAs are currently recognized as issuing EV certificates?

During the test period starting with this release we have provisonally configured three Roots as EV Roots (in alphabetic order)

  • Entrust's EV Root
  • GlobalSign's EV Root
  • VeriSign's G5 Class 3 (EV) Root

Other CAs and Roots may be added later.

About the online repository

As mentioned above, Opera retrieves information about which CAs are recognized as EV-issuers from an online repository. This repository, hosted at https://certs.opera.com/, contains a digitally-signed file listing all the EV issuers and a (separately) signed list of the EV-OIDs they are using.

This repository is refreshed every 6 hours, so we can start updating clients very quickly. At present Opera will check this repository immediately when it starts, but the final version will only check once a week at the same time as we check for other updates.

Examples of EV sites

New^W NOT in Kestrel #2: "Export"-grade encryption is junk(ed)

, , , ...

SSL v2 is not the only old encryption capability that is no longer supported by Opera 9.5, also known as Kestrel. We have also removed support for 40 and 56-bit encryption. Like SSL v2 these methods have been disabled by default since v9.0

There are several reasons for this move:

  • 40 and 56-bit encryption methods are "soooooooo" last century. These methods were defined as a way for US based browser vendors to be able to distribute their SSL enabled client outside the US. The reason this was necessary is that the US government and several allies defined (and I believe still define) encryption as a weapon, but it was possible to get permission to export software that only supported 40 bit encryption (and later 56 bit) and that included a couple of other restrictions (there was an exception for financial services). The reason for the limitations was, presumably, that the intelligence communities of these countries had the technology needed to break these keys. In 1999/2000 the restrictions were partially lifted, at least for mass market products like browsers and email clients.

  • 40 and 56-bit methods are dismally weak given today's technology. In mid-1998 56-bit DES was broken in less than a 56 hours, this was reduced to 24 hours less than half a year later. Given today's technology I expect that the same can be done in less than 30 minutes with enough hardware. And 40-bit can probably be broken in less than a second. What this means is that these methods no longer provide any protection at all.

  • Any server that only supports these methods is more than 8 years old, which means the actual security of the server, even ignoring the lack of encryption strength, is .... questionable. To top it off, a number of Certificate Authorities sell "SGC" certificates that will permit most US produced and exported servers (and clients) to enable 128 bit encryption. These certificates were originally reserved financial institutions, but after the crypto export restrictions were lifted they have become available to all.


While I believe servers that only support 40/56 bit encryption are a bit more common, in absolute numbers (perhaps a few thousand), than SSL v2 servers, I can't remember hearing about any such sites for over a year, despite the fact the the methods have been disabled by default about that long in several browsers. That indicates that the servers are not visited by a lot of people, if any. I think it is time to signal quite clearly to the sites that may be left that the technology they are using is obsolete.

If you do encounter a "secure" site that require 40 or 56 bit encryption, what can you do? Well, I don't recommend it, but you can go back to Opera 9.2x and enable the weak ciphers. But before you do, perhaps you should ask the system administrator this question: "Why are you running the site with 8 year old software?"

New^W NOT in Kestrel: The death of SSL v2

, , , ...

As I've written earlier, in Opera 9.0 we disabled SSL v2 by default, but if necessary a user could enable it.

In Opera 9.5 (Kestrel) we've taken this one step further, and completly disabled the support for SSL v2. That is, as of Opera 9.5 Opera is no longer able to connect to servers that only supports SSL v2.

There are several reasons for this:

  • SSL v2 is OLD. It was added in Netscape 2, back in 1994! SSL v3 replaced it 1996, which means that any service that only wants to use SSLv2 was designed in 1996, or earlier. And it hasn't received a significant upgrade since! Think about what that means about the technology used, and the security of the site...

  • SSL v2 is binary incompatible with SSL v3 and TLS; you cannot send a modern TLS handshake to a SSL v2-only server, it won't understand it. In fact, one of the very few things SSL v2 and SSL v3 have in common is the name! Given new TLS functionality Opera 9.x had already put SSL v2 as the last thing it will try before giving up.

  • SSL v2 itself is known to have at least one major security vulnerability. This particular vulnerability is not present in SSL v3

  • There are few, if any, publicly accessible SSL v2 servers left. A major reason for this is Gerv from Mozilla's campaign two years ago; he managed to convince the hosting company with almost 90% of the servers to upgrade. According to my information Netcraft stopped counting them last year, because there were so few left.

    If you do encounter a "secure" site that requires SSL v2, what can you do? Well, I don't recommend it, but you can go back to Opera 9.2x and enable SSL v2. But before you do, perhaps you should ask the system administrator this question: "Why are you running the site with 12 year old software?"

It ain't EV 'til it's EV, all EV

, , , ...

In the past couple of weeks I have been sewing up most of the loose threads in Opera's support for Extended Validation (EV) certificates, and have actually started testing something that resembles the final system.

The work to add Extended Validation has been going on for a while, the basic functionality was implemented last year, but there hasn't been time to build the surrounding infrastructure until this year.

Now, we are finally starting to see the end of the road.

However, I have started to notice something that could become a problem: Not all EV sites, such as Verisign's and PayPal's sites, are coded the way I think they should be coded, or the way Opera's EV support need to be able to recognize them as EV sites.

The sites themselves and the content they serve are not the problem. The thirdparty content they have chosen to include on their site is; Content that is NOT served from EV servers. Apparently, MSIE 7 does not take this external content into consideration when deciding to show the EV indication.

One of the most prominent examples of this third-party content is Google Analytics, which is added to a web page by an external Javascript from a non-EV secure server hosted by Google. Since Google's servers are not using EV certificates, Opera's EV support does not consider the site to be an EV site.

This is due to how Opera's security evaluation system works. Each URL loaded is evaluated based on which protocol was used, what kind of encryption, keylengths, and whether or not the certificate validated correctly. All of this is summed up as a Level in the range of 0 to 3 as follows

  • 0: No encryption (example, plain HTTP)
  • 1: Weak encryption (40 and 56 bit symmetric methods), weak keys (less than 900 bits RSA/DH/DSA), certificate errors, authentiction only. Additionally a combination of problems that would result in Level 2 security being displayed if only one of them were encountered.
  • 2: One, and only one, of these: medium strength keys (900-999 bit RSA/DH/DSA) or failure to get a valid OCSP response. A combination of both results in Level 1 being chosen.
  • 3: Only when using 128 bit encryption, 1000 bit or more RSA/DH/DSA keys, no certificate validation problems.


EV is implemented as Level 4 in this ranking, and a Level 3 resource is only escalated to Level 4 if it contains the necessary EV identificator.

For a group of URLs, such as a web page with redirects, images, CSS, scripts, frames, and so on, the URL with the weakest security level determines the security level for the entire document. This means that a web page with a single unencrypted 1-by-1 image gets Level 0, and that an EV website which includes a 1-by-1 image from a Level 3 non-EV site does not get the EV status. The only exception to this rule is that, on certain conditions, the first request may be a HTTP to HTTPS redirect.

The reason for this design is simple:

  • It is difficult, and in most cases impossible, for a web browser to determine how sensitive a particular element of a page is.
  • Some elements can affect the meaning of the site (for example, a big image, a script, or a frame).
  • In some cases, such as unencrypted requests, analyzing that traffic may reveal what the user is working on (for example, evaluating a stock transaction).


One can make the argument that since the page is secure we can assume that the owners of the site know what they are doing, and that any use of less secure resources is intentional and does not represent a security problem. That assumption is wrong. They may know what they are doing, and have no intention of lowering the security level, but they can still make a mistake somewhere. In at least two cases I have seen, misconfiguration of proxies or redirects caused content that was not secure to be included in otherwise secure banking/payment pages. In one of the cases this content was an external JavaScript used in a payment page, and an attacker could have gotten full access to the secure page if he had changed the JavaScript in transit.

With respect to EV one can make the same argument, but it is flawed in this case, too:

  • The primary point of using an EV certificate on a site is to know the identity of the site's owner.
  • The EV site's owner may or may not know who owns a non-EV third-party site, the client has no way of determining that. We do not know, and cannot know, if the agreement is just a click-wrap or the result of months long lawyer negotiations.
  • In several of the observed cases the third-party content was external JavaScript, which means the third-party is essentially granted full access to manipulate the site's content, and all of it.
  • By allowing non-EV third-party content in a EV site it becomes possible to set up EV "redirector" sites that let non-EV sites look like they are using EV by embedding their content in a frame.


Part of the reason for creating Extended Validation certificates was raising the bar for identification on the Internet, so that is what we will do: Raise the bar.

When Opera ships a version with EV, probably in Kestrel, a site will not be considered as having EV unless all resources used in the rendered page are using Extended Validation certificates.

"It ain't EV 'til it's EV, all EV"
November 2009
S M T W T F S
October 2009December 2009
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30