miscoded

Since users reported that seeing saved addresses and getting directions on Google Maps was broken in Opera I had to do some digging. Actually a lot of digging. Eventually, I ended up here:

xO.Tia=function(a){if(!a||a==window||!a.hasOwnProperty||xO.uSa(a))return xO.Type.NATIVE;if(xO.jTa(a))return xO.Type.PROTO;if(a.constructor===Function)return xO.Type.FUNCTION;if(a.constructor===Array)return xO.Type.ARRAY;if(a.constructor===Object)return xO.Type.OBJECT;return xO.Type.NATIVE};

..which in its unwrapped and scrambled glory is a method to figure out what sort of object "a" is. The bit we didn't handle like Maps expected was:

if(a.constructor===Object)return xO.Type.OBJECT;

If you pass a DOM node to this function that if clause is true and it will conclude that it is of type "Object". Further along there was some logic detecting that this node was an Object and thus not able to have event listeners.

So, no event listener is added and nothing at all happens when clicking the triangle to view your search history:


In Firefox, a node's constructor property points to the corresponding DOM2 HTML interface. To explain that in real English, the DOM 2 HTML specification says that for each element in HTML (BODY, A, DIV, P and so on) there should be a JavaScript object named window.HTML+name of element+Element (HTMLBodyElement, HTMLAnchorElement, HTMLDivElement, HTMLParagraphElement and so on).

What's the point of that? These are "prototypes" of the DOM. Through JavaScript's prototype inheritance feature you can add properties and methods to for example HTMLDivElement.prototype and all DIVs in your document will appear to have those properties and methods. For example

HTMLDivElement.prototype.getExternalLinks = function(){
    for(var list=[],links=this.getElementsByTagName('a'),l,i=0;l=links[i];i++)
        if((new RegExp( "https?:\/\/(?!"+location.hostname+")" )).test(l.getAttribute('href')))list.push(l);
    return list;
}

lets you call div.getExternalLinks() for any DIV in your DOM to list the anchors inside that DIV that point to a an external URL. Or maybe you want an isExternalLink() method on A nodes instead? No problem:

HTMLAnchorElement.prototype.isExternalLink = function(){
    return (new RegExp( "https?:\/\/(?!"+location.hostname+")" )).test(this.getAttribute('href'));
}

As you see, the DOM2 HTML standard gives you quite a lot of power to extend the DOM.

Back to our problem. When Google Maps in Firefox looks at div.constructor it points to the HTMLDivElement object.

In Opera, it points to window.Object.

The obvious question is: is Opera wrong?

The answer: No. We are perfectly standards-compliant. We follow ECMAScript 2.62 4.3.4 Constructor property spec, DOM2 Core, and DOM 2 HTML.

The problems is: none of these specifications really cover what the constructor property of a DOM node should return.

Opera and Google Maps compatibility is a victim of spec fragmentation. The DOM specifications over there are not sufficiently aware of the details in that other spec over there to cover the issue.

The funny thing is that the public-html mailing list has had loooong-winded discussions where a vocal group argued that the HTML5 spec was too comprehensive and should be split into multiple stand-alone specifications.

Looking beyond the "what I'm not interested in should not be in the spec" shoot-out, they have a point: the spec as-is might not be "reviewer-friendly" to someone who, for example, is only interested in discussing the semantics of the HTML language. ("Spec reviewers" are completely ignored in the HTML5 "priority of constituencies" design principle).

However, fragmenting the spec increases the risk of dropping another "constructor" underspecified on the floor. Google Maps basically found an omission in DOM2's ECMAScript bindings. Opera simply followed the spec and implemented the omission..

Let's do whatever it takes to make the next HTML spec comprehensive enough. To quote Jamie Zawinski: "your needs are big because the Internet is big". So fasten your seat belts, HTML5 will be huge, it'll be complex, it'll have the weird DOM bits and parser bits - and those bits are necessary to make it a really solid specification.

The main point of the standards is of course to align the megaliths rendering and scripting engines of standards-compliant browsers. I've already discussed some of the problems arising when specs and implementations clash, and the dilemmas we face when we have the choice between being strictly standards-compliant and breaking some major website or breaking the spec, aligning with the other browsers, and making the site(s) work.

Never an easy decision to make!

I can now confirm a few points where Opera 9.5 is going to deviate somewhat from the specs or from our earlier implementations in order to be better aligned with other browsers. If you are a webmaster or JavaScript author please check that the changes will not cause problems for you!

We'll do the following:

• Capturing event listeners will fire on target. This behaviour will be kept until Firefox fixes their bug. Opera's new implementation is not expected to cause problems since most sites using the addEventListener API will have been tested against Firefox. A possible exception is widget code which may expect Opera's old behaviour.
  
• Implementation of capturing load event handlers changed: load events from within the document will now only be captured when the capturing event listener is attached to the document. In other words, earlier you could do

  window.addEventListener('load', func, true)

  and expect func to be called for all "load" events inside the document. From 9.5 you will have to do

  document.addEventListener('load', func, true)

  to get this capturing behaviour. (Please do not copy that code unless you know what the "true" part of it means!) This will probably become the official specification. Again, this will probably only break code specifically written for Opera such as widgets, and the change is known to fix large number of websites that use event capture by mistake.
  
• We've changed the relationship between body.clientHeight and documentElement.clientHeight in standards mode. The value of body.clientHeight is now viewport height, documentElement.clientHeight is document height. This is AFAIK what Firefox does too. This change can break your website if you use browser detection before reading clientHeight from body or documentElement.
  

I believe Firefox will support adding capturing load events to document. There you go, alignment of megaliths!

(Like the post title? Found it on Wikipedia .)

A forum thread asks the oft rehearsed question "why can't Opera be more compatible with websites". Allow me to respond with a blog post since that discussion is somewhat noisy already..

Simply: Opera 9 already supports the W3C and ECMA standards that are widely used, it also supports lots of non-standard and pre-standard features to the extent that these are documented by others or by our own testing. Compatibility today is in the details. A big, generic question like "what technologies should Opera support to be more compatible" won't get you far. If a problem isn't caused by sniffing or similar broken code, the likely culprit is a minor detail being different across browsers, not having to support another "technology". Hence, discussing compatibility needs to dive into the technical details and avoid sweeping generalisations - and I hope to provide some examples that are specific enough to perhaps bring the discussion forward:

event.button

Careful and correct implementation of standards goes a long way, unless standards differ from reality. Values of event.button is a damned-if-you-do, damned-if-you-dont scenario: IE and the DOM standard numbers mouse buttons differently, and there is lots of content depending on either of these. Depending on how the site detects "IE" custom scroll bars, mouse-drags and other fancy mouse usage stops working. Options?

• Break this half of the web
  
• Break the other half of the web
  
• Cry and tear your hair out
  

Bohoo.

XMLDocument object
This is an interesting Firefox thingy. It's not in any standard, Firefox just happens to make it available to scripts because internal Firefox architecture has an "XMLDocument" class. While this probably makes sense for Firefox's architecture it doesn't make sense for Opera's. We didn't really notice the XMLDocument object until we investigated why Sarissa doesn't work 100% in Opera. Yahoo Mail Beta uses it too. Options?

• Complain to Mozilla because cross-browser compatibility suffers from their exposed internals?
  
• Spend time tracking how it works and implement it as yet-another-nonstandard-we-must-support?
  
• Bug Mozilla to have it standardised so that we can implement it?
  
• Nag websites about not using it, or detecting it correctly?
  

XUL
Some stuff mentioned in the forum thread uses XUL. Yes, this is a reasonably open, non-standard technology invented by Mozilla which Opera has chosen not to support. The main reason we don't is very pragmatic: there hasn't been that much demand for it. Lack of this technology is very unlikely to be the reason why your Opera-newbie friends complain about sites not working in Opera. Options?

• Support
  
• Not support
  
• Wait for standardisation, then support.
  
• Wait for actual usage growing on the web, then support
  

Windows MediaPlayer scripting
Now this is truly a skeleton in the compatibility closet. WMP's latest scripting support only works with the ActiveX control. Since Opera doesn't support ActiveX we're stuck with an outdated version of WMP's scripting engine, and quite a lot of fancy video player user interfaces on the web are not written to fall back to use the older version if ActiveX isn't available. Did you say "standard"? Bah, there is no such thing in the world of plugin vendors - vendor lock-in is the name of the game here. WMP, Real, QuickTime, Flash - the scripting interfaces are all different, and once a developer has learnt one and a company has invested in building a website on one the cost of switching is naturally high. Options?

• Support ActiveX?
  
• Fake ActiveX?
  
• Add some other kind of compatibility layer?
  
• Continue the endless quest of web openers?
  

Dark matter

They say most of the universe consists of dark matter. Many, many compatibility problems are caused by the dark matter known as convention. All the stuff that we have to do simply because other browsers do. If a site forgets to add a closing SPAN tag, the parser constructing the DOM is going to have to try to compensate. When it turns out that IE's innerHTML in such a case apparently is inconsistent with the DOM it has built, kaboom - pictures.aol.com is broken in Opera. Hit by the dark matter..

Some things can't and won't be standardised (who would have thought that simply taking less than one second opening a popup window breaks a page??) but one of the greatest frontiers of web compatibility work is to expand the standards to cover much more of the "dark matter". WHATWG rises to the challenge and W3C seems to be getting the message that the dark matter of convention is one of the most serious threats to cross-browser compatibility.

Launching in browser.js tomorrow or so: OddPatch beta, solving many of the compatibility issues with the service formerly known as OddPost, now Yahoo!Mail beta.

It's a complex patch sorting out a daunting number of issues on both sides. When things get this complex, it's rarely only "their fault" or "our fault" - the testing has uncovered several bugs in Opera and several mistakes in their JavaScript. Here's a walk-through of the entire patch:

// browser sniffing workaround - walking in through the back door
if( location.href.indexOf( '/dc/system_requirements?browser=blocked' ) >-1){
location.href='/dc/launch?sysreq=ignore';
}

Y!Mail has three modes for browsers: supported, possibly working, or blocked. Opera is blocked, but luckily there is a backdoor that will bypass the sniffing.

The irony here is that when they block us, they are making their work on Opera-compatibility much harder than necessary. If we get access, we'll do our best to make things work: test, find bugs, even decide to support things we haven't supported previously (we'll have selectSingleNode soon because Y!Mail uses it heavily, and their code was also a very important reason why DOM2 Style support was prioritised for Opera 9.0!). Blocking us makes it much harder for us to make their life simpler.

if( top.location.href.indexOf('/dc/launch')>-1 ){ 
// Gecko compatibility library uses defineGetter and defineSetter. We need to fake them.
//* Patch below is required but causes trouble.. 
Object.prototype.__defineGetter__=      function(){}
Object.prototype.__defineSetter__=      function(){}

This is known as "fake it until you make it". We won't have getters and setters anytime soon, but things will work anyway if we pretend we do.

// IEism called loadXML, basically a DOMParser / DOMLS equivalent
// must handle XML fragments without root element!
Element.prototype.loadXML=function(s){  
try{
var d=new DOMParser().parseFromString(s, 'text/xml'); 

This, I think, is a bit of the strange world IE lets you into if you put an XML tag in a page. That tag takes on a life of its own and starts behaving in some contexts like a document, it aquires several methods and properties - and though I at first thought I could simply fake it with a DOMParser I had to think again because...

}catch(e){ // DOMParser could not parse fragment, probably because of missing single root element. Workaround time..
var d=document.implementation.createDocument('', this.tagName, null), el=d.createElement('el');
//?? why did I use this.tagName there?
el.innerHTML=s;
for(var i=0 ; i<el.childNodes.length;i++){
d.appendChild(el.childNodes[i].cloneNode(true));
}
}

Yes, they are not always playing with well-formed XML fragments. Oh well, we'll pull out good-old-tagsoup-parsing .innerHTML and eat their strings anyway. Then we move on and fill in some other required bits and pieces of IE's XML DOM. Right now I'm not sure if all the stuff in this block is required, but there it is.. I'll be the first to admit that both code and comments are evidence of the somewhat chaotic process of late-night patching..

// faking IE-style XML element DOM - separate documents with documentElement within the main doc's DOM
this.documentElement=d.documentElement||d.firstChild; 
//?? firstChild is probably leftover from earlier versions using documentFragment?
this.XMLDocument=d;
// address book loading checks .parseError.errorCode
this.XMLDocument.parseError={ 'errorCode':0 };
return d;
}

And then is a peculiar mystery, who would ever need a function called isSameNode when == would presumably do the job?

// some method called isSameNode is called. Not sure where it comes from but simple enough to fake..
Element.prototype.isSameNode = function(n){
return n===this;
}

Here we go, more delicacies from IE's internals: the handy XPath method selectSingleNode. I have quite some reservations against loadXML and the other XML DOM stuff above, but selectSingleNode should be written into a standard as soon as possible because document.evaluate needs too many arguments for lazy JS coders and the returned object is too fiddly too.

// selectSingleNode support
var realSelectSingleNode=function( expr, resolver ){
var result=(this.ownerDocument?this.ownerDocument:this).evaluate( expr+'[1]', this, resolver, XPathResult.ORDERED_NODE_SNAPSHOT_TYPE,null );
return ( result && result.snapshotLength ) ? result.snapshotItem(0) : null;
}
Node.prototype.selectSingleNode = function (expr, resolver)
{ 
  if (!resolver)
    if (this.nodeType == Node.DOCUMENT_NODE){
      resolver = document.createNSResolver (this.documentElement);
    }else if(this.nodeType == Node.ELEMENT_NODE && this.ownerDocument && this.ownerDocument.documentElement ){
      resolver = document.createNSResolver (this.ownerDocument.documentElement);
    }else{
resolver = document.createNSResolver (this);
    }
  return realSelectSingleNode.apply (this, [expr, resolver]);
}

Now, up till now the patching has been quite ordered. A patch is a bit of a kludge anyway, but so far it's been a nice kludge. Here come the problems that were too ugly for a nice kludge. Certain issues just required search and replace operations on the script source code, nothing else to do about them - and because Yahoo source is compressed and variable names random, search and replace must take that into account and go for seriously complicated and ugly regular expressions.

opera.addEventListener('BeforeScript', function(e){
// This is the riskiest patch
// Fixing typo: missing ' after attribute value
e.element.text=e.element.text.replace( /\):\(">"\)\),/, "):(\"'>\"))," );
e.element.text=e.element.text.replace( / id='_test_add_folder>/, " id='_test_add_folder'>" );

Yep, twice in their source code they say things like <tag class="foo> omitting the closing quote. That caused Opera to parse it as a text node instead of an element, meaning source code would appear here and there in the interface. Oops.

// WebForms2 problem: button attribute "action" is a URL in WF2
e.element.text=e.element.text.replace( /\.(action\b)/g, ".js$1" );

Specs and implementations collide again: if you set input.action to a value like 'markAsSpam' the WF2 spec means it will be resolved as a URL, so when the script reads it again it will see 'http://mail.yahoo.com/dc/markAsSpam' which is not at all what it expected.

// send button not working - attribute nodes must be in the document they will be used
e.element.text=e.element.text.replace( /(\w)\.setAttributeNode\((\w)\)/, "$1.setAttributeNode($1.ownerDocument.importNode($2, true))" );

Hm, is Firefox sloppy with exceptions on cross-document node usage again?

// workaround for getting the documentElement.xml  markup
e.element.text=e.element.text.replace( /(([\w\.]*)documentElement).xml/g, "(document.implementation.createLSSerializer()).writeToString($1)" );

IE's XML DOM rides again. Elements over there have an .xml property which is basically the equivalent of .innerHTML for an HTML element, showing the inner serialised markup.

I didn't take a long and hard look at how Y!Mail used it but some of that seemed very weird. I had the impression that they read .xml of the contacts list only to pass it around as a string and use loadXML later on. Why would they serialize markup just to parse it right into a DOM tree again? Oh well, there is probably some complex reason..

// for...in on objects run into our faked __defineGetter__ and __defineSetter__
// we try to add an exception to any for...in loops
e.element.text=e.element.text.replace( /(for\((var |)(\w*) in \w*\)\{)/g, "$1if($3.match(/^__define/))continue;" );

It turns out "fake it until you make it" wasn't such a good idea after all. The site called our bluff with code like

function foo(obj){
for( p in obj )return false; return true
}

var bar = {}; if(!foo(bar)) return;

and what exactly they meant by that I don't know either, except to check that a newly created object really REALLY had no properties. Huh?

// To: / CC: autocomplete fails
// we support IE's TEXTAREA.createTextRange but unfortunately not its boundingLeft property. Improving object detection..
if(e.element.src&&e.element.src.match(/ac\.js$/))e.element.text=e.element.text.replace( /if \( editCtrl\.createTextRange \)/, "if ( editCtrl.createTextRange  && editCtrl.createTextRange().boundingLeft )" );

This is a typical trap of piecemeal implementation of something: we support whatever of IE's stuff was deemed important to get some plaintext formatting JavaScript to work with 8.x. .boundingLeft wasn't on the list back then. Sorry. Look a bit harder when you look for something.

// Preferences not read correctly from XML attributes
// IE has an attribute node .text property. .nodeValue will work in Opera..
e.element.text=e.element.text.replace( /\.selectNodes\((\w*)\);\s*\}return\((\w*)\.length\)\?(\w*)\[0\]\.text:/g, ".selectNodes($1);}return($2.length)?$2[0].nodeValue:" );

IE again. text alias nodeValue, enough said.

// We throw an unwanted exception if both arguments to insertBefore are the same node
e.element.text=e.element.text.replace( /var (\w*)=(\w*)\?(\w*)\.nextSibling:(\w*)\.firstChild;\s*(\w*)\.insertBefore\((\w*),(\w*)\);/, "var $1=$2?$2.nextSibling:$4.firstChild;if($6!=$1) $4.insertBefore($6,$1);" );

Now this is plainly a bug. The DOM spec says we should throw an error if the node you insert is a parent of the reference child, but we also did so if the inserted node was the reference child itself.

The code doesn't make sense, mind you... Why do you want to replace an element with itself?

// Opera 9.00 and 9.01 has a bug that means createContextualFragment on table elements is unreliable
// easily the worst patch.. but then it works around a really tricky bug..
if( navigator.userAgent.indexOf('9.01')>-1 || navigator.userAgent.indexOf('9.00')>-1 ){ 
// UA detection to target specific bug in specific version is OK
e.element.text=e.element.text.replace( /(\b(\w*)\.selectNodeContents\((\w*)\);\s*var (\w*)=(\w*).createContextualFragment\((\w*)\))/, "if($3.tagName=='TBODY'||$3.tagName=='TR'){ $2.createContextualFragment=function(s){var n=s.match(/<(\\w*)/)[1]; var e=document.createElement('div');e.innerHTML='<table><tbody>'+s+'</tbody></table>';return e.getElementsByTagName(n)[0];  } }$1" ); 

Just as ugly as it looks, just an attempt to make the code work in 9.01.
Imagine if Yahoo!Mail blocked us until they one day decided that it was necessarily to start working on Opera compatibility? If we never had gotten to test their system, they would probably have to add such an ugly workaround to their application to get around this bug. Developers everywhere: please, don't sniff, just leave more of the burden of compatibility on the UA's table (and listen to feedback!).

Hey, we're done with the replacements! It wasn't pretty, and I look forward to deleting one by one while things are fixed on either side. That will also give us a nice performance lift. Their scripts are huge. At some point I did some profiling of the above replace calls and found that up to half of the time it took Opera to load Y!Mail was spent applying the above patches.

}
}, false)

// No scrollbars appear for message list..
// uses an "overflow" CSS property to control scrollbars. e.overflow="-moz-scrollbars-vertical", and some odd clipping as well..
document.addEventListener( 'load', function(){ setTimeout( function(){ var divs=document.getElementsByTagName('div');for(var i=0,div;div=divs[i];i++)if(div.className&&div.className.indexOf('fakeScrollBar')>-1){div.style.overflow='auto';div.style.clip='auto';}},500);}, false );

Yes. That problem. It uses some CSS I still haven't fully understood.. I think they wanted to show an element with a scroll bar but clip the whole element away so only the scroll bar would be visible.

// redraw problem hides To: field in compose screen
document.addEventListener( 'load', 
function(){ if(top.document.frames['newmessage']){
setTimeout( function(){try{top.document.frames['newmessage'].document.body.className+=' ';}catch(e){}},1000);
}}, true);

This is another tricky one, it has to do with timing and I still haven't quite captured the sequence of events. Basically the "To" and "Subject" fields in the compose screen disappear until you click "Show BCC".

// sluggish performance due to unintended event capture
(function(ael){ 
window.addEventListener = function(type, func, capture){ ael.call(window, type, func, false); }
})(window.addEventListener);

..and just to top if off, they had to capture events by mistake. Of course. That's from the curriculum of "How to code Opera-incompatible websites 101".

opera.postError( 'Yahoo mail patched' );
}
}

Yippee! We did it!

Now, this patch is (repeat after me) in beta! There are problems that I'm aware of but haven't fixed, and there are problems that I'm not aware of and haven't fixed. And while I was working on this stuff, code changes in Yahoo mail would break things again every few days (and even break differently on the U.S. and the U.K. sites!). So, hurry up and try it while you have a chance! I'll try to keep the patch maintained, and we sure hope that we'll get all the issues sorted out from either side for a fast, friendly, responsive experience - somewhere in the future..