Skip navigation.

Implementer's notes

What might get caught in the gears under the hood?

Posts tagged with "CRL"

Extending Certificate Status in TLS Extensions

, , , ...

For some time now Opera has supported the Certificate Status Request Extension to TLS, although we did have a false start which was fixed in Opera 9.26.

What this extention does is to provide a way for a client to ask the server to do the OCSP revocation check for its own certificate, rather than the client doing a separate connection to the issuer's OCSP responder. The benefit of this policy is that the client saves time completing the connection since it does not have to wait for the OCSP responder. Also, if the server stores the OCSP response for a while, then the traffic to the OCSP responder becomes much lower (and much less expensive). Mostly servers will request updates and not all the clients visiting the site. This is called TLS OCSP stapling.

This mechanism only works for the server's own certificate. It does not work for any of the other certificates in the chain, and these days most Certificate Authorities (CAs) use at least one intermediate certificate, and some use four, or more. Today all the revocation information about these are retrieved using CRLs, not OCSP. This means that the information is not as up-to-date as is possible for OCSP, as CRLs (particularly for intermediates) are valid for much longer periods than OCSP. This may not be an issue today because most intermediates are controlled by the CA or other relatively big CAs, but it could become a problem if CAs start issuing large numbers of intermediate CA certificates that they do not control, for example to corporate customers. This might become a possibility if/when better domain limitations are widely implementated in browsers. If one of those corporate customers or an independent sub-CA starts issuing bad certificates, it is imperative to be able to revoke those CA certifiates quickly, which would be difficult if the CRL was updated every 12 months. On the other hand, OCSP responses are usually valid for less than a week.

Some intermediate CA certificates are now issued with OCSP URLs specified, but no browsers are currently using them. It is my recommendation that they do not start using them. The reason is that for all clients to use OCSP to check intermediate CA certificates would increase traffic to those
servers multifold, perhaps dozens of times when TLS OCSP Stapling becomes widespread, meaning the bandwidth cost for the CA will increase significantly. A number of CAs have already been concerned about the cost of supporting OCSP just for server certificates while waiting for stapling to become widespread; they would not like the cost of supporting OCSP for one or more intermediate certificates.

The solution, of course, is to expand the TLS Extension to support multiple OCSP responses, which should have been a fairly straightforward task, which it was for the handling of the responses. It turned out, however, that it was not practical to use the existing Certificate Status Request extension in TLS, since it does not allow multiple methods to be specified (but only a single method), which would be necessary to support servers that do not support the new response format. The limitation is due to both a hard restriction in TLS, as only one entry for a given extension can be sent in any extension list, and the request extension only permits a single format to be specified, not multiple.

The solution in the end was to create a new extension that permits multiple formats to be specified, not just a single one as before.

I have just submitted an Internet Draft to the IETF TLS Working Group defining such an extension and new response format. The draft is based on the existing definition with enhancements for the new requirements.

I hope the TLS WG will take on the work to help me complete the draft so that we can get this new functionality into all new clients and servers as soon as possible.

Comments intended as contributions to the draft should also be posted at the TLS WG mailing list.

draft-pettersen-tls-ext-multiple-ocsp-00.txt (archive)

American Express used revoked site certificate for weeks

, , , ...

In the past couple of weeks I have seen a few reports that the American Express site https://online.americanexpress.com/ (Testurl) was using a revoked certificate. This site redirects to various American Express services.

As of this morning (October 1st) the problem has been fixed.

As I have mentioned before, a revoked certificate is no longer valid, and therefore no browser should give access to the site. In the worst-case scenario, a certificate is revoked because the key has been compromised, and the site is being used by criminals for fraudulent purposes. Revocation of a certificate is analogous to blacklisting of a credit card because the card can be abused.

The available (human readable) information indicates that the current situation was not a worst-case scenario. What seems to have happened is that American Express requested the current certificate in May, then shortly thereafter requested an updated certificate, possibly to correct a minor mistake. The old certificate is revoked automatically after a reissue, and the Web site is expected to use the newer certificate, but it seems that American Express for some reason did not update their server. Since there is no way for a browser to distinguish between a benign mistake and a worst-case scenario; we have to assume the worst.

I am sure somebody is going to mention that the site worked without problems in many other browsers. Yes, it did, but that was because the certificate for this site is issued from a special Verisign certificate hierarchy (which is being retired) that does not specify the revocation list (CRL) download location, so browsers cannot go looking for the information, based on the certificate. The reason why the CRL is not specified may, as I understand it, have to do with the fact that the hierarchy was never intended to be used for Web site certificates, but for authentication of dedicated secure server-to-server connections. Opera may be one of just a few browsers that actually check the CRL for this hierarchy, but we can only do this because we have added the certificate to a list of overrides our repository that specifies where to fetch the CRL, otherwise we would not show a padlock on these sites (and as we discovered last year, Opera users really do check the padlock!).

A partially unanswered question is why reports appeared only so recently. Part of the reason is that a bug in our rootstore repository server caused the CRL override Opera uses to be dropped for 6-8 weeks (we fixed that two weeks ago), but that does not explain why there are no reports before the end of July. My guess, based on information from Netcraft, is that online.americanexpress.com is a new server that was brought online just a few months ago, which turns out to be well into the period when most of the CRL overrides were disabled by the repository bug, which means we did not check the CRL either for this site during that period :o: .

Once we learned of the problem, beginning of last week, we informed Verisign about it, and I assume they informed the American Express contact for this certificate about the problem.

Yet, more than a week later the revoked certificate was still being used by the Web site, until it finally got fixed last night. Frankly, I expected better from American Express.

Announcing "The Rootstore"

, , , ...

We have now created a homepage for the Opera Root Certificate repository: The Rootstore.

The Rootstore will be where we announce updates and general information about the root repository.

If you are a Certificate Authority, or generally interested in Opera's certificate store, The Rootstore will be the place to watch.

"Nobody checks the padlock" debunked by Opera users

, , , ...

There's been a number of criticisms directed at the padlock over the years, some of which may be correct, to some extent, at least.

One objection has been that users misunderstand the padlock's meaning, thinking it is an indicator of trustworthiness, rather than a protection rating for the connection.

Another objection (which flies in the face of the first), is that "nobody checks the padlock".

Well, I can't say much about the correctness of the first objection, but the past couple of weeks a growing number of Opera users have been working hard at debunking the second one.

In Opera 9.50 we added some new security features, and changed a some others. Two of these are the following:



Both of these turned out to encounter unexpected problems, not due to bugs in our implemenation, but at the CA side. In all cases the problem cause full validation of the certificates to fail, and as a result Opera reduces the security level of the connection so that a padlock will not be displayed. Quite a lot of people noticed.

For OCSP we encountered problems with at least three different brands of OCSP responders. These responders did not respond correctly when we sent a request for infromation about a certificate. It wasn't until a couple of weeks ago that the vendor for one of them discovered the reason for the problem, their expectation of input format was wrong. According to my information they have developed a patch, and I assume they are doing QA on it now, before sending it to the two (or more) CAs that are affected by this. The others have shown up recently and I do not have good information about the causes.

With CRLs we encountered two problems. Just before 9.50 was released we got the first reports about the first case, but it was not until a few days after the release that we got an overview of the situation.

It turns out that one specific CA created an intermediate certificate a few years ago for one of their certificate hierarchies. This certificate included an incorrect URL for where we should go to fetch the CRL, so when Opera fetches the CRL it gets a CRL created for a different hierarchy than the one being verified. As a result Opera's ceritificate validation code can't find the right CRL, and this step fails. This certificate became installed on thousands of servers before the mistake was discovered, and despite a campaign to replace the certificates, several hundred sites (many of them banks) still use the old certificate.

In another case, which came to light last week, another CA have issued an intermediate certificate directly below their root that does not contain a URL for a CRL, but the sub-ordinate certificate issued from that intermediate does specify a CRL. As Opera's certificate validation code expects that if one certificate have a CRL they all use CRLs, validation of the topmost intermediate certificate fails because it can't find the CRL for that certificate.

What can be done about this?

  • About the OCSP issue, we wait for the vendor patch, but in Opera 9.51 we have added an override that lets us use the certificate update system to specify which OCSP URLs that must use the old POST method. In addition one CA also deployed a workaround.
  • About the first CRL issue, the websites need to update their installed certificates, in the second case the CA must also issue a new updated certificate. While waiting for that, in 9.51 we are adding a similar override mechanism as for OCSP, by specifying extra CRL download locations for specific CAs. In the second case we might also be able to fix the issue if the CA has a root for that particular CA name by distributing it to all installations that access the affected sites (that is not a realistic option for the first case).


These changes will be active in the upcoming Opera 9.51 RC2 (Update: Now available), and is already active in the online certificate repository. You will get the updates the next time Opera checks for updates, or if you use Help->Check for updates.

Update July 9: There are some sites with certificates that does not provide any CRL or OCSP information in the site certificate, while the rest of the certificate in the chain have CRLs. Due to a minor bug, and restrictions in the override functionality, the override for that will not become active before 9.52 has been released. At present we are only aware of 10-20 sites affected by this.

More testing: Updated EV information and new Roots

, , , ...

A few hours ago the new online certificate repository that the most recent weeklies are using was updated with several new roots, and an additional CA, Comodo, was also provisionally EV-enabled

The new and the updated CAs are

  • America OnLine
  • Cisco
  • Comodo
  • Digicert


There is no need to download an updated Weekly (if you have one of the two recent ones). When you next restart one of the Opera 9.50 Weeklies with support for online certificate updates, it will immediately download the indexes, and download the new certificates when necessary. Please give it a minute to finish the update.

Here are a couple of testsites:


Known Issues: The complex certificate chain system used by Comodo encounters some, mostly hidden, problems with our OpenSSL certificate verification support, and that will cause some EV sites to not be recognized. We will try to fix it, but it may not be advisable to include a fix in 9.50.
[*] DigiCert:


I do not currently have testcases for Cisco, as they have not yet started issuing certificates from the new root.

More about known issues:

  • We know there are some problems with OCSP and CRL responses (the two kinds of revocation information) from some Certificate Authorities. These problem may lead to the website getting a lower security level. We are looking into these problems together with the CAs. In last week's build some of the CRL problems will cause a "Fatal Error 50", in the most recent build that has been fixed. We may decide to work around some of these, but they should preferably be fixed by the CA.
  • At least one CA (who is not in our repository) is using CRLs with a critical extension, which will cause the secure connection to fail with error code 554. In this case we are following the standard, although one might wonder why the specification says "Although the extension is critical, conforming implementations are not required to support this extension". The problem have been "fixed" internally by recognzing the extension, then ignore it, as we do not need it.
.

Extended Validation v1.0 approved

, , , ...

As I posted earlier on Opera Labs, work has been under way to create an improved process, Extended Validation (EV), for issuing web site certificates that can give a higher degree of assurance to the user that the SSL/TLS website in question really is who it claims it is, and how to tell the browsers that this process has been used.

Last week, after two years of work, the members of the CA/Browser Forum, a group consisting of many Certificate issuers (for example, Verisign, Comodo, and Entrust) and browser vendors (KDE, Microsoft, Mozilla, Opera), voted to approve Version 1.0 of the Extended Validation Guidelines.

These guidelines describe which steps a CA issuer must (at least) take in order to validate that the information given is correct, such as confirming the legal existence of a business or government agency, ownership of a domain, authorization to request a certificate, etc. Compliance with the guidelines is verified by regular independent audits.

This version of the guidelines also address certain concerns about what kind of businesses are eligible to get EV certificates.

When the certificate is issued, and installed on the server, a browser supporting EV will not just verify the signature on the certificate, it will also:

  • Verify that the certificate is still valid, and has not been revoked because of some problem [link to revocation article],
  • Check for the presence of one of the CA's EV policy indicators (EV-OIDs) in the
    certificate.


If all of this is OK, then the browser will display a visible indicator to the user that the certificate for the site has been issued in accordance with the guidelines. The indicator agreed upon by the browser vendors is a green security toolbar beside the address field, perhaps with a couple of other embellishments.

EV certificates have been issued for a few months based on a preliminary version of the guidelines, and have been recognized by IE7.

No public version of Opera currently supports EV, although we built a demo version with rudimentary EV support last year. Work is going on to produce a full version that supports EV, and we are planning to include support in "Kestrel".

Work in the CA/B Forum is by no means at an end, there are a number of other areas that need similar functionality as that provided by EV to SSL/TLS, as well as possible improvements of the current guidelines.

Stay tuned.

Is that website still in business?

, , , ...

A few months ago I investigated a bugreport about a secure travel agency website that was causing Opera to display the omnious message "The certificate has been revoked by its issuer. The certificate was revoked because this site is no longer in operation". (Well, actually, due to a bug Opera 9.0 only displayed a less informative message and not that particular message, but 9.01 would have.)

What is recovation, and why is it important to you?

About revocation

Revocation of a certificate means that the Certificate Authority (CA) that issued the certificate for a website have decided that the certificate is no longer valid, even if it has not expired.

The information about revocation can be distributed in two ways: Certificate Revocation Lists (CRLs), or by using the Online Certificate Status Protocol (OCSP).

CRLs are (usually) large files that contain a list with information about all the currentely active (unexpired) certificates that are no longer valid. This file has to be downloaded from the CA by the client at regular intervals (usually at least a week apart), and may be quite large.

OCSP, on the other hand, means that the client asks the CA "Is this particular certificate still valid?", and the server responds "Yes" or "No". This method can usually be fairly well up to date, meaning the information is at most a few days old, as opposed to at least a week for CRLs.

All the major browsers support OCSP, but some (like Opera) does not currently support CRLs.

At present Opera is the only non-beta browser that has enabled revocation checking by default, but IE7 for Vista will also check revocation by default. I am unsure about the status of the other browsers, but if they have not already done so, it is only a question of time before they all enable at least OCSP by default.

Possible reasons for revocation

Why is revocation important enough that we actually stop the user from visiting a website?

A CA can revoke a certificate due to a number of reasons:

  • A new certificate has been issued to the website, meaning the old one is not going to be used anymore.
  • The website with the certificate is being used for purposes that are not accepted by the CA.
  • The certificate was issued based on incorrect information.
  • The owner is no longer able to use the private key associated with the certificate, for example the password is lost, the key storage was destroyed somehow, etc.
  • The private key has been compromised or stolen, which means traffic to the site is no longer secure.
  • The certificate and key have been stolen and is actually being used for fraud while posing as a legitimate website.


In the case I was investigating the revocation reason given by the CA was "The certificate was revoked because this site is no longer in operation". To all appearances this looked incorrect, after all the website was there, other browsers accessed it, and it was possible to order from it.

However, as mentioned above, the reason other browsers accessed the site was that, unlike Opera, they did not check the certificate status. Users of other browsers that have manually changed the OCSP and/or CRL settings would also have been barred from accessing the site.

Given this situation there were several possibilities:

  • Somebody at the website development team had made a mistake, either revoking the certificate by accident or forgetting to update the certificate on the server, after all, it still worked in every browser they used. (According to my information, the revocation was requested by the website owner.)
  • The company had indeed gone out of business, but somebody forgot to turn off the lights (and the servers) when everybody left. In such a case people ordering their vacation might have gotten a rather "exciting" start of their vacation when they found out payment had not been made to their hotel or airline. I'm pretty sure some lawyers would have found the aftermath interesting, if this had been the case.
  • The website's private key had been stolen, and the perpetrator was able to update the DNS entries or the network so that they could host their own completely "valid" secure server and use it to commit fraud and to steal credit card information from unsuspecting customers.


Especially the latter possibility sounds scary, doesn't it?

What was really going on?

What all this means is that Opera 8 and 9 refused access to the site, because we checked the current status of the certificate, while most other clients were quite oblivious to the situation because they did not, by default, check the certificate's status.

As I usually do in such cases I informed the CA about the problem. Naturally, they cannot share much information with me, but telling them gives them an idea about how many active websites are actually using revoked certificates (There aren't that many; we encounter at most one or two each month).

Then I tried to get in contact with the website owners so that I could ask them to update the certificate, but that proved to be a bit more difficult than I expected. There was little or no contact information on the site for contacting the webmaster, the only contact method was a comment form, which I filled out and submitted.

Then I started investigating a bit more, and found the parent company and tried sending them an email too, and then the grandparent company. Unfortunately, there were no results to show for the effort.

A couple of days later I recruited a colleague to help get in contact with the grandparent company, because he speaks the language. After several phone calls, more emails, faxes, and being redirected from one person to another, a couple of days later we finally established contact with the travel agency, and within hours a new certificate was in place.

It turned out that the company owning the website had reorganized and changed its name or established a new company to handle the website. During this process all the old certificates had been canceled, but new ones had not been installed.

In other words: Nothing more sinister than a human mistake. Fortunately.

This incident does however highlight what is becoming more and more of a problem, in particular for major websites (just ask David Storey, our Web Opener): Not being able to report serious website problems in a timely manner to somebody that can do something about it. There may be a contact form on the site, but at least in this case it did not have any choice for "website problem", meaning I had to file it in another category which could mean it could be lost somewhere in the system.

Forunately, as more clients enable revocation checking, this problem will diminish, at least for legitimate sites, as they will find out very quickly that they have a problem.

So what should you do if you encounter a revoked site?

First: Don't open a different browser to continue your transaction, provided it will let you. Revocation means you cannot trust the site.

Second: Tell the website operator that they have a problem.

Third: While you wait, shop somewhere else. :devil:
November 2009
S M T W T F S
October 2009December 2009
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30