Opera Core Concerns

You know the complexity of web technology is getting out of hand when every website needs some special treatment. Imagine driving a car if you had to change the engine slightly or replace wheels every time you turned a corner and entered a new street? Browser development is a lot like that these days.

Nearly every modern browser has site patching features, to grease the wheels where the nuts and bolts of standards are exposed to the unpredictable elements of the web:

• IE8 shows sites on the compatibility list in IE7-mode
• Google Chrome spoofs on Hotmail
• Safari has obscure, un-documented site-specific hacks
• Firefox apparently only has the User Agent switcher extension - but it is extremely popular in spite of its niche audience.
• Opera has user agent spoofing and browser.js

Treating every site differently? That sounds insane. It's clearly impossible. For starters, there are billions of websites, all different. And they keep changing with millions of lines of code being added or changed every day. Who can possibly keep up with that? Besides, aren't standards supposed to be the answer?

Standards

I commend the standardisation efforts and the evangelists at The Web Standards Project and elsewhere. Without you we would be far, far worse off. The future might even be on your side - more and more sites validating and cleaning up their standards story. Power to you!

Various Opera teams take part in the standards work too, including doing education and outreach. Our Open The Web team contacts web sites with advice on cross-browser compatibility and standards compliance.

But the browser is the user's agent. Across all sites breaking the rules and standards in different ways, we owe the user to do anything we can to make them all work whenever possible. Often a broken site is not our fault, but it's most definitely our problem. And we must make it work today. Not in 2022..

Hence, we patch.

Since we see various more or less ad-hoc approaches to site patching develop among our fellow browser vendors, it seems like a good time to share some of our experiences - what I think are the strengths and constraints of our particular implementation.

Strengths

Experience

Opera has more experience with site-specific patches than other browser vendors, since we've been patching the web since Opera 8.01. I believe our solution is also by far the most advanced one. Special commands available to User JavaScripts make up a simple yet powerful API for modifying and correcting a website. With getters and setters and DOM interfaces we can change Opera's behaviour to match most standard deviations the script expects. Sometimes we add a specific missing style or even remove some object or policy that gets in the way. And when a web site uses browser sniffing we can often dig in and repair the exact line of code that is broken! Some random examples:

• We can add the single missing class name when AOL Japan abuses browser detection and doesn't apply correct styles, and correct AOL webmail's spelling mistake when they spell dojoType as dojotype.
• We can fall back to a graceful postMessage() call when E*Trade tries to violate the security policy against navigating another site's frame when it loads encrypted content

We can also override settings like what browser we identify as and what rendering mode (quirks or standards) we should use for a specific site through downloadable preferences.

Bugfixes

Given the power to change websites and change Opera, we can sometimes even work around our own bugs.

• When the menu on weather.com disappears because mouse events happen in the wrong order, I can re-order them
• When search results on Google Codesearch appear as white-on-white I can re-style them

Now there's a thought. Opera - the browser that works around its bugs so you don't have to? This is like a time machine - we can fix our past sins for already released versions.

In the long term, this is a very important and subtle effect of browser.js: there will be less Opera-specific hacks to worry about. Both IE8's upgrade woes and our own experience tells us that workarounds against our own old bugs are among our worst problems. We did the wrong thing, sites did the wrong thing in response. We fixed it on our side, but the web never forgets your old bugs.

The difference with browser.js? When we release a new version with bug fixes, corresponding browser.js patches are secretly dropped as obsolete. What used to work because of browser.js hacks keeps working because the new version supports it, making the upgrade experience smoother for users and web developers alike, and a lot simpler for us. I can almost feel the envy of the IE8 team across the Atlantic.

Another interesting effect is that we can implement new standards with less worries. When HTML5's <input required> feature breaks Barnes & Noble because the user name field in the login form isn't really required input after all when you click the "create account" button - we can first patch it, and later tell the spec editor about our problems and discuss if the spec can be made more web-compatible.

Transparency

From the very beginning, one of the main concerns was that our fixes would prevent webmasters from finding and correcting problems. We might even end up creating problems if our patches linger on while the site changes.

To make things as transparent as possible, the patches are documented, we try to keep the file itself readable and reasonably well commented (though we also try to avoid unnecessary download bloat). And every patch applied will output a message in Opera's error console trying to make it obvious what is going on.

Scope

Site patching is cross-platform. Actually, that's an understatement. Did you know we have site patches for Nintendo Wii? And for the DSi? For Opera on Windows Mobile, Symbian, BREW and Archos's portable media players? Heck, we even have patches for Opera Mini!

• When Hotmail and Yahoo mail require rich text editing to compose E-mail we can fake rich text support even on devices with input and CPU limitations where rich text support is impossible
• If Facebook's fancy animations take too long to run for the Opera Mini server - it needs to pass content on to the actual client after all - we can cut them short and yet perform the important actions Facebook saved for the end of the animation.

It's an ambitious scope, probably much more ambitious than it might seem from the outside. The devices and platforms have very different requirements and problems. For example:

• On the Wii we have a minimum font size to keep things readable - but this often causes unexpected widths, wrapping and repositioning content in ways that don't happen on a Desktop browser. We patch several sites that have too rigid positioning to accomodate a slightly larger font size.
• Using an on-screen keyboard on the Wii and many mobile phones means the window resizes itself every time you focus a form element - to make room for the keyboard. When British Airways re-loads the booking page every time you resize it, booking a ticket with a virtual keyboard becomes pretty hard...
• ..and when eBay in France tries to count your keydown events to make sure you don't cheat and paste your e-mail address into the registration form's "re-type E-mail" field it won't work with on-screen keyboards either. Some of them just don't support key events. Come on eBay, I really did type it in. I mean, REALLY! On hardware where cut-and-paste isn't even AVAILABLE!...

When dealing with such a range of problems the true and magic potential of browser.js becomes apparent.

Infrastructure

To support all this, we need some nifty download servers picking the correct file for any client that asks for one. We have a database of patches and meta data like what versions and platforms a patch applies to. But did you know we also have a script spidering sites to check if broken parts are still there? When a patched site changes a relevant piece of code, I want to know about it. I love the sound patches make when they hit the virtual "obsolete" bin.

Constraints

Time

So much power, so little time.. Analysing sites and developing patches takes a lot of time. We are nowhere near IE8's overwhelming list of 2400+ sites. (Actually, browsing the IE8 list is a Déjà vu experience. Many familiar names, many sites we need to patch or contact for various reasons too.)

Nevertheless, our list is shorter - fortunately! Technical analysis with the level of detail required to support our site patching can be very time consuming, so I hope we'll remain smaller and more focused than the IE list. We will however scale the efforts up - we're at 600+ entries in the database and there is room for plenty of more - and we welcome contributions.

Performance

I just said there is room for plenty of more patches. But how far can we scale site patching before performance suffers too much?

Apparently we can keep going for quite a while. Some profiling work we did recently indicates that we can patch upwards of 10 000 specific sites without slowing Opera down by more than a couple of percentage points. And frankly, by the time we've written patches for 10 000 sites you've probably bought a more powerful computer..

Security

Isn't it unsafe? Can't a virus or malicious server fake your browser.js file and gain instant control of your Opera browsing?

The short answer is: no.

The longer answer is that this would be possible without our security precautions, but it is in fact all taken care of. Every file is signed, so Opera can check that it is a genuine file and that it has not been tampered with. Such a signature is very, very hard to fake. If a malicious server or virus alters the file, Opera will stop using it and try to download a fresh version from the server.

Conclusions

One of the things that surprised me most about browser.js work is how quickly the web changes. By paying such close attention to specific sites we see the web as a whole evolve, and it's iterating faster than I would have dared to expect. And contrary to what you might expect, we do see sites change to become more Opera-friendly after we've patched them!

That means site patcing works. It works because it improves compatibility, thereby giving users a real choice in browsers and - as weird as it may sound - as a direct effect of that, it gives webmasters incentives to make sure their sites are Opera- and cross-browser-compatible.

Welcome to the future - made of and with standards, openness and site patching.

About the author

Hallvord R. M. Steen works for Opera's core team on quality assurance, testing and web compatibility. He maintains the browser.js file and can break all websites worldwide with a minor typo.