Slightly ajar

I took some time out recently to be interviewed by the leading Norwegian technical news site, Digi.no. You can find the interview here (Norwegian only I'm afraid, but it does have pictures). The title of the article is Web standards? Download IE, and the topics cover my work with Opera opening the web, along with some of our future plans ofr OtW and what issues IE8 will cause in this regard.

Over the next two months or so I'll be saying goodbye to the progressively colder Oslo, and hello to a number of cities around the globe. With the conference season in full swing, I'll be off meeting web developers, (hopefully) fixing web sites, and spreading the Opera word.

First stop will be Dallas, Texas for Web Jam Session. I'm missing Texas BBQ since my first and last visit to Texas for SXSW in March. I've got a lot of catching up to do. Austin is a great city, but I've heard Dallas is very different, so I'm not sure what to expect. I've had a number of friends from Dallas down the years, but I've never made it out there yet. Following Web jam, I'm hopping over the Pacific to Sydney Australia for Web Directions South. It'll be my first time in Australia, but not my first Web Directions. I went to the fabulous Web Directions North in Vancouver earlier this year and had a great time. It'll be nice to catch up with all those Aussies (of which there are far too many to mention) involved in making the web a better place. I had hoped to take some vaction and see a bit of what Australia has to offer, but instead I'll be heading straight off to Orlando, Florida for Mobile Web Americas. Our own Jon VonT will be speaking, and we have a great mobile message to spread to the world. No rest for Disney world or the beach however as I'm straight on the plane to Mountain View, California to visit our new Silicon Valley office. I'll actually be able to have some break from the ever changing time zones and jet lag for a while. I'll hopefully be visiting a number of major web companies while there, to discuss how to support Opera better. If you work for a web company in the area and want help in supporting Opera then drop me a line and I'll see if we can organise a meeting. I should get a chance to see some of San Francisco's sites before I leave. After three weeks or so I'll be off to NYC for Future of Web Design. This will be the first city on my tour that I've been before, and It'll be great to see the place after the last time I visited in 2001. There will possibly also be a visit to Boston before NYC, but it depends how things pan out.

If anyone is in any of those locations, then give me a shout. I might not be able to keep this blog up to date for a while, depending on how conference and hotel wifi keeps up, but I'll report back every so often with how things are progressing. When I get back in mid November it is bound to be freezing here, but it'll be nice to be home and see friends again.

There are often cases when web designers wish that browsers support more web standards or fix that pressing bug they always come across. Just how difficult can it be to support all of CSS2.1 or XHTML 1.1 or such? It can't be that difficult, right? Well, unfortunately it often is. I've just been thinking about the many issues we face as an alternative browser, and I though I'd share a little look into what we deal with when producing our rendering engine. This excludes everything else, such as the interface and features of the browser.

Types of issues

There are three broad categories of bugs that we can face when developing a rendering engine:

1. Browser issues
2. Web site issues (known as web or otw bugs at Opera)
3. Specification bugs

Browser issues are bugs and issues that are caused by our browser working incorrectly according to the specification or the de-facto-standard. These are the issues that Opera has to fix in order to be a fully standards compliant browser or work in major web sites. Web site issues are where a web site has bugs due to not following the spec correctly, or deliberately targeting our browser. Specification bugs are where the official spec has issues, such as specifying something that doesn't work in the real world or using vague wording that can be interpreted differently by different browsers. As can be seen, there is more than just following the spec and expecting all sites to work. It is not just this simple however, as each broad type of issue has a number of sub issues types.

Browser issues

The first type of browser issue is where we lack support for a specific part of a standard (such as border-radius in CSS3 for example). This is the part that many developers clamour for, and a very important part of developing a browser. Then there are browser specific extensions. These are generally something we don't want to add, such as ActiveX, but there are cases when it is important due to many sites using that feature or there being no standards based equivalent. Designmode, ContentEditable and xmlHttpRequest are examples of these, even if they are becoming official standards post implementation. These are often not trivial to add as there needs to be a lot of reverse engineering to work out how they should work. As they are not official specs they often don't have a spec to base the behaviour on or public test cases that can be used in the QA process. There is also the question of if we want to be bug compatible with the browser that created the feature in question, even if the quirk doesn't make sense. Added to this, there is the matter of standards and quirks mode, where a feature can act differently depending on what doctype the developer uses.

After missing features there are bugs in already implemented features. Browsers are complex pieces of software, so all have many bugs. The bugs that most people complain about is when a browser doesn't behave like the spec (or other browsers) behave. These are obviously also very important to get fixed as they cause issues that need to be worked around by developers, or leave the site working incorrectly. The bugs people often forget about are the bugs in error handling, where the browser would work correctly if the code was following the standard, but most people don't read the spec and can write invalid code. It is very common that sites don't validate and don't even use a doctype, so resort to the less specified and more murky world of quirks mode, where it can be a nightmare to work out what the browser should handle the content. This is one of the areas that HTML5 looks to specify better. If the error handling is different to what the spec states (if it is defined there) or Opera behaves differently to other browsers then this could be seen in the long term as a Opera bug that needs to be fixed. In the short term Open the Web can be used to contact the site and suggest the standards compliant way to do what was intended (such as include missing quote or whatever). This benefits the site in question as it is more likely to work on more browsers than relying on error handling, which is inconsistent at best. These two bug types can be broken down into two more sections; those that affect a lot of sites or very popular sites, and those that affect just a few sites or very minor sites. It is obvious that we have to concentrate on fixing issues that affect the most users and developers, so these get higher priority. These can often take a higher priority than adding new standards support, and take just as much work. If we have a bug in something like Google.com, even if they don't write to spec, it will affect many millions of users many times a day.

Web Site issues

Then there are web site issues. This is my area of work. No matter how well we follow standards, fix bugs and have have perfect error fallback these issues will cause problems that we can't fix without external help. There are a number of different types of issues in this category, some of which overlap with browser bugs hi-lighted previously. Web site bugs can be where the code breaks the specification. This is where we do what is expected, but the author of the site didn't specify exactly what they meant correctly. This maybe because they just tested in one or two browsers and assumed that if they work there then they wrote the code correctly, when instead they are taking advantage of a browser bug in another browser accidentally. This can often even when testing in two browsers, as one browser may get a different code path (a IE codepath and a standards codepath) or more than one browser implemented the same bugs (such as the event listener bug that both Safari and Firefox had with capturing events). I'd advise to test in at least two out of Opera, Safari and Firefox, at least after major changes or implementing complex JavaScript. If the results differ then a bug has been hit. The next bug type is where Opera's error fallbakc is incorrect or inconsistent. This overlaps with the same issue in Opera bugs, but can be handled by Open the Web if there is a feasible correct solution that still works in the other browsers. Then there are two types of bugs relating to browser sniffing.object detection. The first is where Opera gets sent different code to other browsers and thus breaks when the same code as sent to another browser would work correctly. This issue is very common, especially in old JavaScript files where a very old Opera bugs is being worked around, or the author is just being malicious (believe it or not this seems to happen). Object detection can be a problem as a script can detect one function such as document.all and assume the browser is a different browser (IE in this case) or uses other functions that were not tested, such as using VBScript, ActiveX or CSS filters. The next type of browser sniffing issues are when a site just blocks Opera, or give a browser warning before entering the site. This can be commonly found on Google properties for example.

The next type of web bug is when a site uses either a vendor specific extension to a standard (such as IE or Mozilla DOM extensions) or vendor specific technology (such as ActiveX or VBScript). This sometimes uses browser sniffing and can be included above, but often just is used for all browsers. For example, many video/music sites don't work as they need ActiveX to include the Media Player plug-in that uses DRM, and doesn't exist as a Netscape API plug-in. As mentioned above, for vedor specific extension they can be added into the browser engine if there is a need for this, but that is long term and not always desirable. If there is a standards based solution then open the Web will contact the site and suggest the alternative way to do the same thing. Sometimes there is other solution and the feature has to be added or the site has to stay broken. For vendor specific technology such as ActiveX there is mot often no way to add that technology, without unreasonable amount of work, or restrictions. These are often a lost hope until the site in question redesigns. These issues can also be broken down into issues with top sites and with the long tail. Again top sites will always get most priority as they affect the most users.

It isn't strictly true that Opera can't fix these issues without outside help. We have user agent spoofing for browser sniffing issues, and browser JavaScript to patch site bugs and object detection bugs. These will only be used for popular sites or 3rd party scripts, as there can't be too many patches included, and are quite fragile to site changes, so can only be a stop gap solution. There are two major issues with these techniques. The first is that it causes problems when a developer is testing the site and it isn't behaving as expected, or the patch/spoof isn't applied due to using a different domain for the test server. The latter is that with spoofing, Opera will disappear from site statistics in most cases, making it look like Opera isn't worth supporting.

Specification bugs

The final type of bugs are specification bugs. The w3C, or whoever writes the spec in question, have to be contacted about these issues, and everyone has to agree what is the correct way that the spec should be written and the feature implemented. If Opera ends up on the losing side of a spec change then it has to fix the bug to make it work like the new spec and other browsers. This will render older versions incompatible with the new spec. New versions of a spec can take a long time to be accepted, so these issues can be long term, and with no clear right way until the spec is changed, it is difficult for web designers to make something work cross borwser when using the feature in question.

As can be seen there are many types of different bugs and issues and all this has to be taken into consideration when developing the next version of the rendering engine and what to focus on. Real world compatibility or standards compatibility, adding xyz or fixing that nagging table error fallback bug? Then this can be all multiplied by the number of technologies there are to support in a modern browser, from CSS1,2.1,3, HTML4 and 5, XHTML 1 and 2, XForms, XPath, Web Forms 2, SVG 1.1, client side XSLT, JavaScript, EV Certificates etc. Then there is the rest of the browser wrapped around the rendering engine…

One of the things we've been focused on is increasing our real world compatibility with Kestrel, so I'm very much looking forward to its release. That doesn't mean we've stopped adding new standards though, as I've hi-lighted previously and we'll hopefully be able to add more before release time permitting.

Things are hotting up in the rich media arms race. In the red corner we have Adobe, with Apollo (web technologies, Flash and Flex), while in the blue corner we have the new entrant from the Redmond Giant, Silverlight (web technologies, XAML, .Net et al) — wow, these names really do sound like American Wrestlers. The W3C is also somewhere there in the mix with SVG and widgets 1.0. it could potentially be a bloody battle, and it is quite worrying for the web, as each of the two main players use their own propitiatory technologies, runtimes, codecs and developer tools. One has to use Visual Studio, or Expression studio on Windows to develop for Silverlight for example. Is this another example of the big guys trying to take control of the web again. After Mix, Ex-Microsoft Employee, Robert Scoble, posted about Silverlight, claiming Microsoft rebooted the web.

As Silverlight runs in a browser, it is very important that Opera supports it. We must be compatible with the web sites out there that will use this technology. I also think that it is important that Adobe have strong competition, so that it doesn't run away with a monopoly on rich media interfaces on the web, tying everyone to their own technologies and products. Silverlight has also already been demoed on Windows Mobile. Maybe this is the kick that Adobe needs to start pushing its development of Flash on mobiles and embedded devices. If Flash isn't there (or gets too outdated) maybe Microsoft will get there first.

I don't want to announce too much just yet, but as can be seen from this picture, there are plans in the works for Opera support. We've been discussing this with Microsoft and Tim Sneath since it was still called WFP/e. I hoped to be at Mix to announce Opera support, but unfortunately it came a bit too soon. It was good to see that CNet picked up Forest Key's announcement though. Progress and bug fixes are being made, and the Silverlight team have been very helpful. Hopefully we'll be able to make announcement on Opera compatibility soon.

It is not just the Silverlight team that are being helpful. Microsoft's Ajax library, ASP.net Ajax, has issues in our browser. We've started to look into how we can fix these problems, and have just started discussing with them on including their automated test suite into our build system. We can then start looking into fixing the problems on both their and our end. Once ASP.net Ajax is fully compatible, we should see a vast reduction in the problems in their services that make use of the library, such as the Live.com series.

I'm not sure if anyone here is familiar with the state of the Korean web industry, but I've just read a very interesting blog post on Engadget, related to an issue Opera and the Open the Web team know far too well:

South Korea's Ministry of Information and Communication, Ministry of Government Administration and Home Affairs, and Financial Supervisory Service have all come out against widespread Vista upgrades, advising Joe Consumer -- er, Kim Consumer -- to hold off on upgrading until ActiveX compatibility issues can be worked out. Apparently banks, portals, online games and online shops have relied a bit too heavily on the sometimes insecure ActiveX controls, and are scrambling to make their sites compatible with Windows Vista's new approach to ActiveX. Microsoft has been working with banking services and others to promote compatibility, but the changes are taking longer than it expected, and its not delaying the OS further to appease the stragglers. So the best the South Korean institutions can do is issue said warnings and hope for the best when the 31st rolls around.

Korea has a huge issue with web compatibility. Most sites use Active-X, and even if they don't the majority rely on IE bugs and quirks, instead of following the standards. Looking at almost all of the top sites in any none-IE browser is like looking at a page after it has went through a blender, with mis-placed elements everywhere, and missing content. If IE7 made all of its changes in quirks mode as well as standards mode the Korean web industry would be in even bigger trouble. We've been trying to contact Korean web sites for a long time to fix their sites, but they were either not interested (market share) or didn't reply. There are also grass roots level organisations like OpenWeb Korea that have been trying to combat this problem. Maybe the Korean government and the industry as a whole will realise what we've been trying to tell them all a long -- that relying on one browser, not following standards, and using vendor specific extensions is folly and it will cost you in the end.

Maybe the government should start promoting these open standards now to avoid issues such as this in the future. What will IE8 bring for example? How are you going to get sites to run on mobile, where IE is not king and Active-X or VBScript doesn't exist? How many more Won will it cost the country each time these issues surface? I'm not sure who from Microsoft they are working with to fix these issues, or who Microsoft is talking to, but we have a lot of experience with the problems are facing. We have many issues already found, analysed and suggested fixes ready. We'd love to work with Korea to get their web sites into the 21^st century and away from vendor lock-in, so the offer still stands. If any site want to work with us on fixing their issues then get i touch with me and we'll only be too happy to help