karlcow

Opening The Web one bug at a time

Subscribe to RSS feed

Web Standards Links - 17 January 2011 to 23 January 2011

, ,

Promoting interoperability is at the heart of Open The Web.

Web standards

Javascript

I18N

CSS

HTML5

Mobile

Semantic Web

Web Accessibility

Auth

Opera

Events in Tunisia and Opera statistics

, , ,

Looking at Tunisia Mobile browsers statistics, there was a surprising spike in the curve. I wondered what could be the reason of this sudden increase. As you may know, Tunisia went recently through a lot of political events. Citizens revolted against the president and a new temporary government has been put in place. There are details on the English wikipedia page about the events but if you can read French, I would recommend the French version. I was wondering if the spike was due to more people in the street and then using a mobile browser. In fact, it is difficult to know. The spike starts on January 13 when the now former President announced he will not run again for the next elections and promise the Press and Internet freedom. This didn't put an end to the uproar of citizens and the next day, fights between the army and the citizen were happening in the streets.



I still wonder why this spike has happened, maybe the people, maybe something else. It is difficult to know. Opera Mini using big proxies through the world has been used sometimes to bypass the local censorship. (Update Opera Mini servers were used to be blocked in Tunisia. That might explain the spike if suddenly the users had access to more resources on Internet.)

The mix of social and technology changes has always fascinated me.

Update 2011-01-26: Wikipedia entry on Internet censorship in Tunisia

Magic Strings and User Agent Sniffing

, , , ...

User Agent sniffing is bad most of the time. It creates a lot of issues. It relies on the idea that a Web site should be working only for a few browser vendors. But User Agent sniffing becomes really unacceptable when the site definitely exclude specific browsers based on their user agent string. Even more so when we realize that once the user agent string has been spoofed, it is possible to access and use the content of the Web site.

Let’s take an example from last week, the exact domain name is not important, so let’s call it: http://bad.example.com/. It always starts with one or more bug reports of Opera users saying. I’m a customer of the company Bad Inc. and I’m not able to access the Web site with my browser. Then, we check if it’s a bug in Opera or an issue with the Web site. curl is a wonderful tool to quickly test what’s goint in between the browser and the server.

So let’s start. We check with Firefox, Safari and Opera and look what is working and not working. The combination is not always the same. In this case it was working with Safari, Firefox and not working in Opera. Let’s switch to the command line. The option I in curl creates a HEAD HTTP request.

% curl -sI http://bad.example.com/
HTTP/1.1 404 Not Found

That means that the server is clearly doing “whitelist” user agent sniffing. It allows only what it knows and blocks the rest. Let’s try with a Webkit user agent string.

% curl -sI -A "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; de-de) AppleWebKit/534.15+ (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4" http://bad.example.com/
HTTP/1.1 200 OK

It is working and with Opera?

% curl -sI -A "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.6; U; fr) Presto/2.7.62 Version/11.00" http://bad.example.com/
HTTP/1.1 404 Not Found

Not working. At this point the issue is clear the next step will be to contact the site and ask them to modify their server side user agent sniffing to include Opera or even better to include everyone else. But I was wondering what exactly triggered the user agent sniffing. For example, I tried to reduce the user agent string to Mozilla only.

% curl -sI -A "Mozilla" http://bad.example.com/
HTTP/1.1 404 Not Found

Not working that’s not it. What about Gecko?

% curl -sI -A "Gecko" http://bad.example.com/
HTTP/1.1 404 Not Found

Not working either… hmmm… ok one more try.

% curl -sI -A "Mozilla Gecko" http://bad.example.com/
HTTP/1.1 200 OK

Bingo! But what about IE it doesn’t have Gecko in its user agent string. after trial and errors I got

% curl -sI -A "Mozilla MSIE 6" http://bad.example.com/
HTTP/1.1 200 OK

with MSIE n, where the n >= 6. Put a 5 in there and it stops working. I thought ok that’s interesting what about adding these strings to Opera.

% curl -sI -A "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.6; U; fr) Presto/2.7.62 Version/11.00 Mozilla Gecko" http://bad.example.com/
HTTP/1.1 404 Not Found

Hmmm. oh! One more try

% curl -sI -A "Mozilla Opera/9.80 (Macintosh; Intel Mac OS X 10.6.6; U; fr) Presto/2.7.62 Version/11.00 Gecko" http://bad.example.com/
HTTP/1.1 200 OK

Bingo! The site is working. Big smile and then head banging on the table on realizing how it is dumb. What did I say? ah yes, do not use user agent sniffing if you do not know what you are doing.

HTML5. Feels. Good. OpenWeb

, , ,

In case you are living under a rock, you know by now that there is a new logo for HTML5 released by the amazing W3C Team. The logo is under a Creative Commons Attribution 3.0 Unported. It means you can play with it and makes your own version, make fun of it or with it, etc. That. is. cool.

Some vapid comments were made about collapsing all technologies under the umbrella of HTML5. Like it or not, the HTML5 word itself became the flag of an idea for Open Web Technologies for most people. W3C has been reproached in the past to not embrace the crowd, I'm pretty sure they will now be reproached to embrace it. So far what I have read on different venues on Internet is a lot of positive comments.

Opera has a lot less employees, around 750 according to Wikipedia which makes it 5% of its workforce participating to W3C. Not bad.

This is a good day for the Open Web when the community is having pleasure, when people are pushing technologies and fixing bugs and when we learn about the diversity of mobile markets.

Web Standards Links: 10 January 2011 to 16 January 2011

, , , ...

Web Standards Links: 3 January 2011 to 9 January 2011

, , ,

2DVN - The Two Digits Version Number Bug

, ,

Yesterday, I was writing about collecting the mistakes that create interoperability issues in Web browsers. So let’s start with a first one. You thought the Y2K bug was the biggest nightmare on Earth, that every servers will stop on the planet with buffer overflows. Well… You have not seen anything.

The Bug

Let me introduce you the 2DVN (Two Digits Version Number) bug. Browser identifies themselves on the Web with a User Agent string. For example, the user agent string for the old version of Opera 9.70 on Linux was:

Opera/9.70 (Linux i686 ; U; en-us) Presto/2.2.0

So far so good. Web developers included javascript scripts in their Web sites to match the name, the character “/”, and finally the version number. Because Web developers were not interested by the minor version number (.70 in our example), they made script matching only the major number (9 here in our example). They add things such as

browserName = Opera;
browserVersion = 9;

and with these were applying different rules. Opera browser proudly, before every other browsers, reached the version 10 beta on June 3, 2009 and then sites started to break everywhere. The issue is that they were doing it the wrong way by taking a substring with a length of 1 character. The value for the two variables were becoming

browserName = Opera;
browserVersion = 1;

The Patch

Sites started to block Opera thinking that it was a bad bot spoofing Opera, or applying the bad rules to the browser thinking that this and that is not supported. You can try to contact all the sites on earth to fix it and we try to do that when we detect this kind of mistakes, but the easiest solution was to do something such as:

Opera/9.80 (Macintosh; Intel Mac OS X 10.6.6; U; fr) Presto/2.7.62 Version/11.00

We kept the first number with 1 digit, so javascript will not break with a two digits number and we added a new string called Version/11.0 displaying the real version number of Opera. This creates issues with stats and analytics software which will not detect the new version of Opera for the same reason explained above, but at least most Web sites were not blocking anymore Opera for no reasons.

The Return Of The Vengeance Of Doom

New libraries have been developed for matching the Version/xx.xx number at the end of the User Agent string such as jQuery browser.version. Guess what… some Web developers are doing again the same mistake by using Javascript code such as:

jQuery.browser.version.substr(0, 1)

I see already some people smiling in the room thinking “Who cares? That is only Opera”. We then can wait with a big smile… Chrome has reached version 7 in two years, upgrading its number every 3 months. IE has reached version 9, the next one is 10. Here an example of code which is currently available on the Web on a cool site:

if ($.browser.msie && jQuery.browser.version.substr(0, 1) == "9") {}

Working to open the Web, solving interoperability issues, helps maintaining an open choice for users. So they can use any browsers to interact, communicate and share on the Web.

Be Strict To Be Cool

, ,

Be Strict To Be Cool was my mail signature when I was working at W3C. It was a "clin d'œil" to the Robustness principle (Postel Law), which is basically when you send a message to someone (or a software), be very strict in the syntax and grammar of your message, when you receive a message be flexible enough to deal with bad messages as long as you can interpret it. It is what I called once the Hear-Write Web.

Opera's Open The Web is the initiative to help Web developers to fix their Web sites. It is also a way for customers to identify when Opera browsers specifically, but sometimes others such as Firefox, Safari, Chrome, and IE, do not work with a specific Web site. The ODIN team is then looking at what could be the origin of the issue. Armed with curl, View Source, and the wonderful opensource DragonFly, we explore, analyze and figure our next actions.

Sometimes, it is really a browser bug, which is being taken care of by implementers at Opera. But most of the time, the issue is related to bad server settings, bad user agent sniffing, javascript mistakes, rogue server side libraries, etc. Some cases are mind boggling. Proxy and routers introduce their own share of issues too. The most difficult task is not being to analyze the issue but to find the right contact for the Web site. A person who will understand what you are talking about. The communications and marketing departments of big companies are more responsive and very helpful. Something going wrong on their Web site is less customers.


Looking at this daily pile of bugs and issues, I thought today that it might be interesting to collect them little by little and explain them. The goal is not to create a wall of shame, but to help Open The Web by giving hints to Web developers on what has been done wrong in their settings, developments, etc. Often, they just do not realize because it is a case that they never experienced. Everyone is making mistakes. What I'm interested in is how collectively, we leverage our knowledge for creating a better Open Web.

Production of Cookies for Web Developers

, ,

Some cookies are really bad for your health and you have to be careful when you cook them. A very simple cookie looks like that when the server sends it to the client:

Set-Cookie: cookieName=cookieValue

but more often it will looks like this

Set-Cookie: cookieName=cookieValue; Path=/; Domain=example.org

Sometimes they will contain an Expire date.

Set-Cookie: cookieName=cookieValue; Expires=Wed, 09 Jun 2021 10:18:14 GMT

Cookies are defined in the specification 2695 and now in the specification HTTP State Management Mechanism currently written by Adam Barth. The production rules for the servers are strict and defined in the section 4.1. Set-Cookie.

These are a set of rules you have to check when you are coding either javascript or your Web framework to produce cookies.

  • Set-Cookie:SP The space is important. US-ASCII SP (octet 32)
  • cookieName any US-ASCII characters except control characters (octets 0-31) and DEL (octet 127) and, the following characters “(“, “)”, “<”, “>”, “@”, “,”, “;”, “:”, “", “/”, “[“, “]”, “?”, “=”, “{“, “}”, the double quote character itself, US-ASCII SP (octet 32) or the tabulation (octet 9)
  • = no space before and after.
  • cookieValue same than cookieName

Then optionally you can add

  • ;SP The space is important
Just a little reminder because this morning I stumbled across a cookie which was badly defined on a Website:
Set-Cookie: {$aaa|xxx:"zzz"}=foo
the characters {, }, " and : are forbidden here.

RSS Readers in browsers and Opera

, , ,

There has been quite a few discussions around RSS readers these last few days. RSS is dying got a bit of coverage.

The controversy started because of Firefox which has an open issue on removing the feed icon in the toolbar. Mozilla made a very interesting usability survey about what is used and not used in the toolbar of Firefox, and they found out that "only 3%–7% of users use the RSS button on the toolbar".

It may mean a few things.

  • People do not care about feeds.
  • People do not know what is it.
  • People do not understand what is it.


The issue is not knowing why it is not used. Maybe the browser vendors should all do the same kind of survey with a more refined analysis to really understand what is happening. The tools inside the browsers for handling RSS feeds are often poorly designed and lack a bit of help for new users. They assume most of the time that people will understand what 1) the icon means 2) the principles behind this icon. There are plenty of software out there to manager your feeds, but none are really easy to use if you are a new user. It is mostly disconnected from the daily user experience. People are mostly used to the email because they have to work with it. People never really had to work with feeds, except those tracking information.

Opera browser has an integrated mail application with an RSS reader and there is a short screencast to explain how to add feeds to your reading flow. Still in my suboptimal, we (browser vendors) really need serious usability studies to better integrate this feature (if necessary) in the flow of people's information consumption.

Update 10:34am EST: It seems that some people misunderstood what I was saying here. So let's get a bit deeper.

  1. We need better usability studies for understanding what is really happenning
  2. RSS buttons might be a wrong metaphor for subscribing to feed readers (see above)
  3. Subscription and reading are too different things, but saying RSS is dying when one button disappears is inflammatory
  4. The issue is wider than feeds itself, it is about interchange flow of information. For example, you can't comment about a blog post in your feed reader. *That* is an issue.
  5. a few more issues wink