Skip navigation.

Implementer's notes

What might get caught in the gears under the hood?

Posts tagged with "SSL"

American Express used revoked site certificate for weeks

, , , ...

In the past couple of weeks I have seen a few reports that the American Express site https://online.americanexpress.com/ (Testurl) was using a revoked certificate. This site redirects to various American Express services.

As of this morning (October 1st) the problem has been fixed.

As I have mentioned before, a revoked certificate is no longer valid, and therefore no browser should give access to the site. In the worst-case scenario, a certificate is revoked because the key has been compromised, and the site is being used by criminals for fraudulent purposes. Revocation of a certificate is analogous to blacklisting of a credit card because the card can be abused.

The available (human readable) information indicates that the current situation was not a worst-case scenario. What seems to have happened is that American Express requested the current certificate in May, then shortly thereafter requested an updated certificate, possibly to correct a minor mistake. The old certificate is revoked automatically after a reissue, and the Web site is expected to use the newer certificate, but it seems that American Express for some reason did not update their server. Since there is no way for a browser to distinguish between a benign mistake and a worst-case scenario; we have to assume the worst.

I am sure somebody is going to mention that the site worked without problems in many other browsers. Yes, it did, but that was because the certificate for this site is issued from a special Verisign certificate hierarchy (which is being retired) that does not specify the revocation list (CRL) download location, so browsers cannot go looking for the information, based on the certificate. The reason why the CRL is not specified may, as I understand it, have to do with the fact that the hierarchy was never intended to be used for Web site certificates, but for authentication of dedicated secure server-to-server connections. Opera may be one of just a few browsers that actually check the CRL for this hierarchy, but we can only do this because we have added the certificate to a list of overrides our repository that specifies where to fetch the CRL, otherwise we would not show a padlock on these sites (and as we discovered last year, Opera users really do check the padlock!).

A partially unanswered question is why reports appeared only so recently. Part of the reason is that a bug in our rootstore repository server caused the CRL override Opera uses to be dropped for 6-8 weeks (we fixed that two weeks ago), but that does not explain why there are no reports before the end of July. My guess, based on information from Netcraft, is that online.americanexpress.com is a new server that was brought online just a few months ago, which turns out to be well into the period when most of the CRL overrides were disabled by the repository bug, which means we did not check the CRL either for this site during that period :o: .

Once we learned of the problem, beginning of last week, we informed Verisign about it, and I assume they informed the American Express contact for this certificate about the problem.

Yet, more than a week later the revoked certificate was still being used by the Web site, until it finally got fixed last night. Frankly, I expected better from American Express.

Extended Validation Update

, , , ...

Today we have formally EV-enabled the first two CAs.

For more information, please see The Rootstore's announcement

W3C Web Security Context: User Interface Guidelines

, , , ...

The W3C's Web Security Context Working Group have just released the Last Call version of its "User Interface Guidelines" document, which is a set of recommendations for the security related UI in Web User Agents.

This specification deals with the trust decisions that users must make online, and with ways to support them in making safe and informed decisions where possible.



This document specifies user interactions with a goal toward making security usable, based on known best practice in this area. Subsequent testing of this specification will include conformance, interoperability, and usability testing.



If you want to comment on the document you are welcome to do so:

The W3C Membership and other interested parties are invited to review the document and send comments to public-usable-authentication@w3.org (with public archive) through 15 September 2008. We appreciate if comments follow these guidelines for writing good issues.




Announcing "The Rootstore"

, , , ...

We have now created a homepage for the Opera Root Certificate repository: The Rootstore.

The Rootstore will be where we announce updates and general information about the root repository.

If you are a Certificate Authority, or generally interested in Opera's certificate store, The Rootstore will be the place to watch.

"Nobody checks the padlock" debunked by Opera users

, , , ...

There's been a number of criticisms directed at the padlock over the years, some of which may be correct, to some extent, at least.

One objection has been that users misunderstand the padlock's meaning, thinking it is an indicator of trustworthiness, rather than a protection rating for the connection.

Another objection (which flies in the face of the first), is that "nobody checks the padlock".

Well, I can't say much about the correctness of the first objection, but the past couple of weeks a growing number of Opera users have been working hard at debunking the second one.

In Opera 9.50 we added some new security features, and changed a some others. Two of these are the following:



Both of these turned out to encounter unexpected problems, not due to bugs in our implemenation, but at the CA side. In all cases the problem cause full validation of the certificates to fail, and as a result Opera reduces the security level of the connection so that a padlock will not be displayed. Quite a lot of people noticed.

For OCSP we encountered problems with at least three different brands of OCSP responders. These responders did not respond correctly when we sent a request for infromation about a certificate. It wasn't until a couple of weeks ago that the vendor for one of them discovered the reason for the problem, their expectation of input format was wrong. According to my information they have developed a patch, and I assume they are doing QA on it now, before sending it to the two (or more) CAs that are affected by this. The others have shown up recently and I do not have good information about the causes.

With CRLs we encountered two problems. Just before 9.50 was released we got the first reports about the first case, but it was not until a few days after the release that we got an overview of the situation.

It turns out that one specific CA created an intermediate certificate a few years ago for one of their certificate hierarchies. This certificate included an incorrect URL for where we should go to fetch the CRL, so when Opera fetches the CRL it gets a CRL created for a different hierarchy than the one being verified. As a result Opera's ceritificate validation code can't find the right CRL, and this step fails. This certificate became installed on thousands of servers before the mistake was discovered, and despite a campaign to replace the certificates, several hundred sites (many of them banks) still use the old certificate.

In another case, which came to light last week, another CA have issued an intermediate certificate directly below their root that does not contain a URL for a CRL, but the sub-ordinate certificate issued from that intermediate does specify a CRL. As Opera's certificate validation code expects that if one certificate have a CRL they all use CRLs, validation of the topmost intermediate certificate fails because it can't find the CRL for that certificate.

What can be done about this?

  • About the OCSP issue, we wait for the vendor patch, but in Opera 9.51 we have added an override that lets us use the certificate update system to specify which OCSP URLs that must use the old POST method. In addition one CA also deployed a workaround.
  • About the first CRL issue, the websites need to update their installed certificates, in the second case the CA must also issue a new updated certificate. While waiting for that, in 9.51 we are adding a similar override mechanism as for OCSP, by specifying extra CRL download locations for specific CAs. In the second case we might also be able to fix the issue if the CA has a root for that particular CA name by distributing it to all installations that access the affected sites (that is not a realistic option for the first case).


These changes will be active in the upcoming Opera 9.51 RC2 (Update: Now available), and is already active in the online certificate repository. You will get the updates the next time Opera checks for updates, or if you use Help->Check for updates.

Update July 9: There are some sites with certificates that does not provide any CRL or OCSP information in the site certificate, while the rest of the certificate in the chain have CRLs. Due to a minor bug, and restrictions in the override functionality, the override for that will not become active before 9.52 has been released. At present we are only aware of 10-20 sites affected by this.

Lowering the EV bar

, , , ...

Last week at the W3C's Web Security Context Working Group's meeting at Opera HQ here in Oslo we discussed what should be the criteria for displaying the Extended Validation (EV) indicator, or the Augmented Assurance (AA) indicator, as the WG has decided to call this technology.

As I have said earlier, we are of the opinion that the EV indicator should not be displayed unless all content is loaded from EV servers.

The opposing view is that, so long as all content is loaded over secure connections, the displayed document is what the author of the main document intended (bugs, vulnerabilities, and all), and that it is therefore only necessary to verify that the main document is served from an EV server, as this will provide the information necessary to identify the author.

The decision of the WG was in favour of the less restrictive position: if an AA/EV document loads all resources over strongly TLS-protected connections, then the document can be displayed with an AA/EV indicator.

In the interest of providing a common user experience with respect to EV we have decided to follow this recommendation, and today's Kestrel snapshot include this policy change.

We have, however, left the old logic in place, controlled by a preference that can be updated remotely. This permits us to quickly change to a stricter mode if the consensus about what constitutes an AA/EV site changes in the future.

As a consequence of this policy change a large number of sites, such as Paypal, with mixed EV and non-EV content will now get the "Green Bar" in Opera:

More testing: Updated EV information and new Roots

, , , ...

A few hours ago the new online certificate repository that the most recent weeklies are using was updated with several new roots, and an additional CA, Comodo, was also provisionally EV-enabled

The new and the updated CAs are

  • America OnLine
  • Cisco
  • Comodo
  • Digicert


There is no need to download an updated Weekly (if you have one of the two recent ones). When you next restart one of the Opera 9.50 Weeklies with support for online certificate updates, it will immediately download the indexes, and download the new certificates when necessary. Please give it a minute to finish the update.

Here are a couple of testsites:


Known Issues: The complex certificate chain system used by Comodo encounters some, mostly hidden, problems with our OpenSSL certificate verification support, and that will cause some EV sites to not be recognized. We will try to fix it, but it may not be advisable to include a fix in 9.50.
[*] DigiCert:


I do not currently have testcases for Cisco, as they have not yet started issuing certificates from the new root.

More about known issues:

  • We know there are some problems with OCSP and CRL responses (the two kinds of revocation information) from some Certificate Authorities. These problem may lead to the website getting a lower security level. We are looking into these problems together with the CAs. In last week's build some of the CRL problems will cause a "Fatal Error 50", in the most recent build that has been fixed. We may decide to work around some of these, but they should preferably be fixed by the CA.
  • At least one CA (who is not in our repository) is using CRLs with a critical extension, which will cause the secure connection to fail with error code 554. In this case we are following the standard, although one might wonder why the specification says "Although the extension is critical, conforming implementations are not required to support this extension". The problem have been "fixed" internally by recognzing the extension, then ignore it, as we do not need it.
.

The trouble with the (b)leading edge: It's sharp!

, , , ...

One of the new SSL/TLS features we added in 9.0 was the TLS Certificate Status Extension, which is essentially asking the TLS server to do the revocation check for its own certificate (OCSP), thus telling the client whether or not the certificate it presents is valid.

At first this can look pretty much like putting the fox to guard the henhouse, but there is no real danger as even a malicious server can't get away with sending the wrong data for long. This is because the status information is generated and signed by the certificate issuer (CA), meaning it can't be faked, and is only valid for a few days at most, reducing the window of opportunity to a minimum.

The primary reason for using this method is reducing the traffic load on the OCSP servers. Once all clients support OCSP the load on those servers will be tremendous, so it is prudent to look at solutions to reduce not just the network load on the CA's servers, but also the time used to check the validity of the certificate. It is a fact that making a direct OCSP check takes time, and that it increases the time to establish a connection, slowing down the user's surfing.

Therefore, using the TLS server as a proxy makes sense, and we introduced it in 9.0.

Unfortunately, it turned out that the implementation had a problem. Due to a mistake in the way the Extension was read, a method introduced in 9.0, Opera 9.0 to 9.25 will crash when the server tells it that it will send the status message.

This showed up only when Microsoft started testing its Windows 2008 server (IIS 7), due to be released this month.

Unfortunately, it was not until less than two weeks ago I learned of the problem, when Microsoft contacted me directly with a testcase (Thanks). It didn't take me long to find and fix the problem, but it still requires releasing a new Opera version.

In this case I will strongly recommend that users of Opera 9.0 to 9.25, inclusive, upgrades to the upcoming Opera 9.26 (Update: It is now available), due to be released later this week (but an RC is available). The next 9.50 weekly will also contain a fix for the problem. In particular, you should upgrade if your installation crashes when it tries to connect to an SSL/TLS/HTTPS site.

We apologize for the inconvenience, in this case we were a little bit too far out on the (b)leading edge.

New^W NOT in Kestrel #5: As many certificate warnings (if you don't want them)

, , , ...

As I explained in "NOT in Kestrel #4", the certificate is the passport of the website. What was not significatly mentioned in that article is that if there is any non-fatal problem with the verification of the certificate, or with other related information, Opera (and other clients) will display
a certificate warning.

Problems that cause certificate warnings to be displayed can for example be:

  • The certificate issuer is unknown (and it is not possible to discover a link to a known Root).
  • The certificate is expired
  • The name of the server does not match any of the servers named in the
    certificate.
  • Weak encryption keys (now only public keys)


These warnings are displayed the first time in a session that you connect to a given server, and would not be displayed again for the rest of the session if the user accepted the certificate.

This has (understandably) caused some irritation when a user is frequently visiting a site causing such warnings to be displayed, and there has been frequent requests to be able to accept such certificates more permanently.

I have been, and still am, skeptical to such an ability, because I think a serious and secure website should not trigger security warnings.

I have, however, decided to meet the requests halfway. In Opera 9.50 it is now possible from the security panel of the certificate warning to "permanently" accept a certificate for the given SSL/TLS server (and port). Although accepted by the user, Opera will (same as before) not display a padlock for these sites because Opera has not been able to properly establish the server's credentials.

The acceptance isn't, however, quite "permanent" (therefore the quotes). A certificate that has not expired will be accepted until it expires (at which time the webmaster SHOULD replace it), and for periods of 90 days at a time after expiration.

Enjoy, but use with caution.

Seasons Greetings! See you in the new year.

New^W NOT in Kestrel #4: As many warnings about unknown certificate issuers

, , , ...

One of the most important parts of SSL/TLS, as it is used in browsers, is the use of certificates to identify the website you are visiting, and to identify the keys used to agree on the encryption keys used for the connection.

The certificate is the website's passport, and like a passport it is issued by an authority trusted by the people checking them. For passports this is the government of a country, for certificates it is a trusted Certificate Authority (CA).

As with passports, there are a lot of things that have to be checked to make sure the certificate is not forged.

Each certificate contain, among other things the name of who the certificate is issued to (the website); the public key of the website; who issued it (the CA); and a digital signature of the data in the certificate, which is signed by the CA.

To verify the signature the browser must use the public key of the CA to decrypt the signature (which could only be created by using the associated private key), and compare the result with the calculated "checksum" of the certificate. The public key is stored in another certificate issued to the CA - the CA certificate.

There are two types of CA certificates, the ones that are signed by yet another CA, called Intermediate CA certificates, and certificates signed by the CA itself, which are known as the Root Certificates or Self Signed Certificates.

The Intermediates CA certificates are verified using the public key of the signing CA, while the Root CA certificates are verified using the public key in the certificate itself.

The Root CA certificate is a special case. By itself it only verifies that somebody claiming to be the named issuer issued the Root Certificate, and by extension vouches for the named identity in the certificates issued from it. But that does not carry any real assurances.

To be sure that the Root CA is valid the signature on the certificate have to be verified by a trusted copy of the Root Certificate, stored in the client's list of such certificates. These certificates are usually shipped with the client, and are accepted and trusted by the vendor to be reasonably careful about who they issue certificates to and how they store the private key. The user can elect to not trust some of these roots, and may add his own list of certificates.

Only a certificate and its associated certificate chain that can be verified and traced back to a Root in the Trusted Root store on the client is accepted automatically. If a certificate cannot be verified because the signature does not validate this is a fatal error; if we can't trace the certificate to a known root we display a certificate warning.

A problem often encountered is that a web site gets a certificate from a CA that is issued from an Intermediate CA (Most CAs do this now, for convenience and security reasons, as well as for selling CA certificates to CAs without their own root or to cross-sign roots) but the web site administrator forgets to install the Intermediate CA certificate on the server. This means that the client will not be able to trace the certificate to a known Root, even if it has the Root in its repository, because clients usually do not, and are not expected to, know specific Intermediate CA certificates; the result (for the user) is an unexpected Unknown Issuer certificate warning.

SSL/TLS servers that do not send intermediate certificates are actually not operating in compliance with the SSL/TLS standard. The standard requires the server to send any CA certificates it cannot reasonably expect the client to have already, and the only thing it can expect the client to have is the root certificate, and not any intermediates.

There is however a mechanism (called Authority Information Access, AIA) defined in the specification of the certificate format that let the CA specify a "CA Issuer" download location for the Intermediate CA certificates. This mechanism is already used by at least one other browser (which neatly explains why so many badly configured sites get past testing), and now Opera 9.50 also use this mechansism.

When Opera 9.50 encounters a site with a certificate chain that is missing an Intermediate CA certificate linking it to the Root, but the certificate specifies an AIA issuer download location, Opera will download the specified certificate. If the verification then succeeds (without other incidents) to link the certificate with a known root, the certificate will be accepted as normal, and the intermediate CA certificate will be "cached" in the new Intermediate CA certificate store in Opera's certificate manager for future reference, to avoid downloading it again later.

What this means is that you will no longer get any certificate warnings about unknown CAs when the certificate is issued from a root and a CA that also includes an AIA issuer download location for the intermediate certificate(s) for the Intermediate CA(s) that issued the certificate.

Special thanks to Gary and Les in Verisign for helping me debug this new functionality.
November 2009
S M T W T F S
October 2009December 2009
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30