miscoded

the web is a hack

websites playing timing roulette

Saturday, March 7, 2009 12:01:03 AM

, ,

For a couple of days in November, the New York Times looked serenely white in Opera:


You might think that nothing newsworthy was happening on the planet. Unfortunately it was not the start of a new and peaceful world order, it was merely their JavaScript playing timing roulette:

if (typeof callback == 'function') { 
if (document.addEventListener) { 
  window.setTimeout(function(){ 
    document.write('<script type="text/javascript" charset="utf-8">(' + callback.toString() + ')();<\/script>'); 
  }, 0) 
}

What is that doing? This code says "have a 0 millisecond break, then add this SCRIPT tag to the document". (By "0 millisecond" it sort of means "as soon as you get around to it".) The problem with that is that if the browser happens to complete loading the document before running that document.write() statement it will replace the current document with a single, invisible SCRIPT tag.

Opera, doing its best at trying to show you the page as soon as possible did finish loading the document before getting back to the timeout - so the page was overwritten and disappeared.

Now, by the above it sounds like it is a drawback to be fast (again?). Wait until you see an extract of some code that broke image upload on Orkut:

<head>
<script>setTimeout( function(){ document.body.appendChild(el) }, 0 )</script>
<script>/*other stuff here*/</script>
</head><body>


In this case, if the timeout runs before we've seen the BODY tag - it will break, because no document.body exists yet. Wait, at New York Times parsing quickly was a mistake and here..we're punished because we didn't get to BODY yet when running the timeout?? So we're simultaneously too fast for NYTimes and too slow for Orkut..

This is JavaScript timing roulette - such things happen when websites are written according to the timing of specific browsers, or even the speed of the network connection the web site developer uses! What would happen if you're visiting Orkut on a really slow connection and the network has a small hiccup-pause between HEAD and BODY? By Orkut's code I would not be surprised if certain network delays - only a few milliseconds - would break the site entirely.

And it gets worse. AOL sites often try to launch slideshows in popup windows. In Opera that doesn't go very well:



AOL is another gambler addicted to playing the timing roulette. Bear with me, because this gets complex - but they have something like

<SCRIPT src="http://www.aolcdn.com/ke/swfobject/ke_kit_popup_includes.js" type="text/javascript" language="javascript" charset="utf-8"></SCRIPT> 
    <DIV id="gallery-holder"> 
       <DIV id="news-news_popup_foobar">


The popup_includes.js contains code that appends external scripts to HEAD. The final external scripts (ke_kit_refresh.js) contains further inline code that calls rederPopupPage() which calls a method in the opener window that eventually calls the embedswf() method which starts looking for the element with the ID "news-news_popup_foobar" (where 'foobar' is actually the short name for the article).

Since loading scripts block parsing, we have not yet reached the DIV when the slideshow is supposed to be inserted. So when the script asks "do you have the element news-news_popup_foobar?" Opera responds "no, not yet anyway" and the script just gives up.

So, apparently we're supposed to keep parsing forward while we're running scripts dynamically added earlier in the DOM?! The W3C never told us that, I think. It's the web's dark matter striking again.

X-Talisman-Compatible: messupSite patching article on the core concerns blog

Comments

Andrew Gregory Saturday, March 7, 2009 2:58:10 AM

So, are you saying that NYTime and Orkut working in other browsers is just pure luck? Or are there some unwritten rules you're trying to figure out?

BTW, *should* document.write ever overwrite the current document? I thought it always added to the document and that document.open was the function that wiped out the document?

Anonymous Saturday, March 7, 2009 3:49:59 AM

Anonymous writes: Yes, this is how browsers work. It has nothing to do with spec. This is why modern frameworks poll document.readyState to find out when the body has loaded and only then is the code executed. It's pretty lame to see that big sites like NYT aren't aware of this stuff.

Anonymous Saturday, March 7, 2009 4:01:13 AM

Anonymous writes: This is why you should use Safari 4 or Firefox 3 instead. They seem to work correctly with all of these issues.

Anonymous Saturday, March 7, 2009 4:28:53 AM

snuxoll writes: 'correctly' is subjective, I'm not an Opera user (don't much care for it at all, actually), but developers should know better. I'd pull in jQuery *JUST* for $.ready() these days and avoid these kinds of issues.

Anonymous Saturday, March 7, 2009 4:33:37 AM

Anonymous writes: > This is why you should use Safari 4 or Firefox 3 instead. Awesome idea. Actually, let's go back in our time machine, back to 2002 when IE6 was the only browser that works awesomely everywhere, and all those marginal browsers like Safari and Firefox kept breaking because they didn't implement some feature that worked really well in IE6. Or.. we actually try to uphold web developers to a higher standard than "well, it works on my box, so your browser must suck, obviously."

Anonymous Saturday, March 7, 2009 7:31:41 AM

Anonymous writes: snuxoll hit the nail on the head, JQuery has dealt with this logic rather well, and you can pull it in just for this function alone. Yes, I know, absurd overhead. That being said, why anybody on a big site isn't using flag vars for this sorta thing is insanity. How do you get hired at a high traffic site without knowing how to use flag vars (Booleans in any normal language) to watch for this sort of thing? I saw I pickup truck in Florida that said "Quality websites for $297" in the back window. I wanted to strangle that guy for undermining our profession.

edvakf Saturday, March 7, 2009 8:00:17 AM

You are absolutely right, hallvors. It's very surprising other browsers could properly handle uncertain setTimeouts...

By the way, Firefox and Safari (in their beta versions) implemented Speculative loading, which loads external scripts and cache them until the proper script tag appears. It makes a browser far robust against "network hiccups" and, imho, it isn't hard for Opera as well to implement it.

When will we see it..?

Anonymous Saturday, March 7, 2009 10:24:46 AM

Anonymous writes: > BTW, *should* document.write ever overwrite the current document? I thought it always added to the document and that document.open was the function that wiped out the document? Yes, it should. If the document has completed loading, document.write wipes it out. It's probably in a spec somewhere, but that is the behaviour of IE6 - IE8, all the Firefoxes, Opera, and anything Webkit based.

Andrew Gregory Saturday, March 7, 2009 1:32:14 PM

@edvakf: I'm not sure, but maybe the "Delayed script execution" opera:config option does that?

As for document.write, it's always nice to see one function doing two different things at different times. I don't *want* to know which idiot thought of that one sad

Charles SchlossChas4 Saturday, March 7, 2009 3:41:38 PM

have the 2 sites read your post? The pages work now.

This is why you should use Safari 4 or Firefox 3 instead. They seem to work correctly with all of these issues.



Safari does use site-specfic hacks, I am not sure about FireFox but my guess is something simalar. So they also have trouble.

Omega JuniorOmegaJunior Saturday, March 7, 2009 4:22:14 PM

There's a reason for using document.onload and other forms of script deferrence. When manipulating the dom, you want to make sure the dom actually is loaded. It's not like these techniques are brand new, or anything. I really don't want browsers to go back to guestimating what it could possibly be the author intended. We've seen too much of that already. Authors should know what they're doing, and otherwise get fired.

Anonymous Saturday, March 7, 2009 8:03:06 PM

Jessta writes: Who creates a site like that? Who is confident enough in the platform a user will be using to create something that mad crazy? WTF happened to this web? I'd the crackheads decide it was more profitable to make websites than steal cars and change professions?

Hallvord R. M. Steenhallvors Saturday, March 7, 2009 11:31:07 PM

So, are you saying that NYTime and Orkut working in other browsers is just pure luck? Or are there some unwritten rules you're trying to figure out?


I think it is a bit of both. The final issue is probably an unwritten rule - it's a bit counter-intuitive that if you have a document with

SCRIPT id=a
SCRIPT id=b
DIV id=c

in the markup, neither script can see the DIV. However, if you have

SCRIPT id=b
DIV id=c

and "script b" contains code that inserts "script a" through the DOM, script a should see the DIV. You end up with exactly the same DOM structure yet "script a" runs when the DOM is in a different state, and in the AOL page's case behaves very differently.

have the 2 sites read your post? The pages work now.



NYTimes fixed it quite quickly - I don't think we even got around to complaining to them about it. Pretty good turnaround time when they needed to fix a mistake yes. The Orkut problem was sort of masked by some other problems - I'm about to add some browser.js patches solving an avatar upload issue, and with that issue out of the way I came across this. And AOL's popup slideshows are of course still broken..

@edvakf and Andrew Gregory: delayed script execution is indeed similar to the "speculative loading" - though perhaps a little more ambitious. If it's bug-free enough and improves perceived performance (which it logically should) it would be nice to enable it by default wink

The larger point in this article is that writing code that depends on a certain non-specified, implicit timing of events is a bad idea. Period. Do you want a site that breaks if a browser improves its parsing and DOM performance? Do you want a site that breaks if the TCP packet containing BODY start tag is a little bit delayed by a slow network? Of course not.

Charles SchlossChas4 Sunday, March 8, 2009 2:44:43 AM

What AOL link trigers the glitch?

Anonymous Monday, March 9, 2009 6:48:40 AM

Anonymous writes: so you've shown us the problem. what are you going to do about it? because world is brutal, and people want to see the page instead of some geeky explanations why 'other browsers are doing wrong showing you the pages' if safari/ff use speculative loading or any other trick that makes such mess work - I want opera use it too. I dont want to read why browser does not work, I want browser to work. there are many other browsers around, and they DO 'work' on such sites...

Anonymous Monday, March 9, 2009 12:22:11 PM

Anonymous writes: "so you've shown us the problem. what are you going to do about it?" I guess Opera needs to slow down. Performance be damned. If being too fast breaks sites, Opera damn well better make sure it's as slow as other browsers!

Anonymous Monday, March 9, 2009 12:23:55 PM

Anonymous writes: > people want to see the page instead of some geeky > explanations This blog is all about "geeky explanations", dumbass. Posting in a blog isn't going to fix anything but since some people actually want to educate themselves (unlike you), blog posts that explain the geeky stuff behind the scenes are great news. I guess you are allergic to knowledge, because you somehow think that not posting about the geeky details would magically fix the site.

Hallvord R. M. Steenhallvors Monday, March 9, 2009 3:11:56 PM

Chas4: most of or all the "feature stories" on e.g. http://www.aolhealth.com/ have this problem - click through to the article, then try to start the slideshow - only the empty popup appears.

so you've shown us the problem. what are you going to do about it?



Well, Mr. anonymous, the usual drill: make sure bugs are reported, analysed, given sufficiently high priority, write the test cases, suggest politely to webmasters that these sites might want to move their SCRIPT elements below the elements they rely on, write browser.js patches that make the sites work even if they depend on undocumented magic timing luck, regression test the fixes when developers write them, remove browser.js patches if the fixes work, and blog about any interesting observations in the process. If there is anything else you think I should do about it, I'm all ears.

Hallvord R. M. Steenhallvors Monday, March 9, 2009 5:17:54 PM

Looks like it might be. Thanks smile

FataL Monday, March 9, 2009 8:50:46 PM

Not only sites play timing roulette, Opera also does:
http://my.opera.com/community/forums/topic.dml?id=266662
http://globeprgroup.com/tests/opera-bugs/javascript-jump-to-anchor/

I think those bugs is pretty relevant to a post...

Zotlan Tuesday, March 10, 2009 11:15:07 AM

This page, posted in this thread, seems to show similar behaviour. At least, it only works with javascript turned off.

Anonymous Wednesday, March 11, 2009 1:44:24 PM

huh writes: @FataL, why are you hijacking the comments section with irrelevant stuff?

FataL Wednesday, March 11, 2009 5:51:50 PM

@huh, I think it is relevant since it's also JavaScript and timing related.

Anonymous Thursday, March 12, 2009 1:26:35 PM

huh writes: @FataL, just because it mentions JavaScript doesn't mean that it's about anything and everything. This is clearly about specific sites and not JS in general or anything like that.

Anonymous Thursday, March 12, 2009 5:53:07 PM

Daniel Colascione writes: And yet again, we re-learn the lessons the operating systems people learned in the 80s. Lesson of the day? The web doesn't make race conditions magically disappear. You need to be as competent in this field as in any other.

Omega JuniorOmegaJunior Saturday, March 14, 2009 9:11:17 AM

Daniel wrote: "You need to be as competent in this field as in any other."

This is quite true. I've worked in the field of software programming for over 15 years. Anyone working in any field any length of time, sees new people coming in and making the same mistakes.

This is a problem of lack in education. Usually, this problem starts with the project manager.

edvakf Saturday, March 14, 2009 7:39:44 PM

@hallvors

Apologies for the off-topic again.

The Delayed Script Execution works similarly to the Speculative Loading. After all, however, it's a functionality for a mobile browser as far as I feel. Yes it's turned on in the Opera Turbo edition that was just unveiled, but I can easily imagine a script (both page script and user script) not working when it's turned on.

One obvious thing I found was that Opera with Delayed Script Execution won't fire DOMContentLoaded event any more. I'm quite sure it will affect browser.js behavior as well.

Since it is apparently only Opera that doens't already implement Speculative Loading among the 5 major browsers, I strongly hope Opera either to refine the Delayed Script Execution to be trouble free, or to have the Speculative Loading.
http://stevesouders.com/ua/

Just a thought anyway.

Hallvord R. M. Steenhallvors Tuesday, March 17, 2009 9:35:02 AM

Hi edvakf,
first of all: when I tested with a simple TC, Opera seems to send DOMContentLoaded also when delayed script execution is enabled. If you see such problems, could you point me to a page where this occurs?

(The test I used was as simple as putting this snippet in a HTML file:
document.addEventListener('DOMContentLoaded', function(){ passed=true; }, false)

so there may well be scenarios this test didn't catch where it fails)

Secondly, enabling delayed script execution should not fix the AOL problem I mention, actually. This is because the implementation of delayed execution goes to great lengths to make sure the script that runs after parsing sees the DOM as it would be if the script ran normally. Doing things differently caused some problems - one example I remember was a reload loop on milonic.com where a frame breaker checking top.frames.length was confused by an IFRAME after the script. Since top.frames.length was larger than expected the script tried to "break out" of the presumed frameset.. So the intention behind delayed execution is that enabling it should make no difference whatsoever from the running script's point of view.

Hallvord R. M. Steenhallvors Tuesday, March 17, 2009 9:42:43 AM

(And BTW - here is one of the problems with implementing "speculative parsing" or any sort of parsing-forward-while-running-script stuff: whether it will fix or break things is a matter of emulating the timing of the other browsers that do so. Imagine a site that requires forward-parsing like AOL apparently does. Imagine that it was tested with browsers that while waiting for an incoming script parse on average 15 elements onwards. Imagine Opera implementing this and optimising the parser to parse 35 elements onwards instead - that might not be what the script expects either. The frame breaker experience shows that we will likely break some site, somewhere.)

edvakf Tuesday, March 17, 2009 10:47:33 PM

@hallvors

I tested this UserJS.
// ==UserScript==
// @include http://*
// ==/UserScript==

document.addEventListener('DOMContentLoaded',function(){opera.postError(0);},false)
document.addEventListener('DOMContentLoaded',function(){opera.postError(1);},true)
window.addEventListener('DOMContentLoaded',function(){opera.postError(2);},false)
window.addEventListener('DOMContentLoaded',function(){opera.postError(3);},false)
opera.addEventListener('BeforeEvent.DOMContentLoaded',function(){opera.postError(4);},false)
opera.addEventListener('BeforeEvent.DOMContentLoaded',function(){opera.postError(5);},true)
opera.addEventListener('AfterEvent.DOMContentLoaded',function(){opera.postError(6);},false)
opera.addEventListener('AfterEvent.DOMContentLoaded',function(){opera.postError(7);},true)


WITHOUT Delayed Script Execution turned on, the console messages are below;
JavaScript
Event thread: DOMContentLoaded
0
JavaScript
Event thread: AfterEvent.DOMContentLoaded
7
JavaScript
Event thread: AfterEvent.DOMContentLoaded
6
JavaScript
Event thread: BeforeEvent.DOMContentLoaded
5
JavaScript
Event thread: BeforeEvent.DOMContentLoaded
4
JavaScript
Event thread: DOMContentLoaded
2
JavaScript
Event thread: DOMContentLoaded
3
JavaScript
Event thread: AfterEvent.DOMContentLoaded
7
JavaScript
Event thread: AfterEvent.DOMContentLoaded
6


on both http://portal.opera.com/upgrade/ and http://www.google.ca/

Then I turned on the DSE, I get ABSOLUTELY NO messages on http://portal.opera.com/upgrade/, but exactly the same message as above on http://www.google.ca/

I'm on Opera 9.64, by the way.

PS. I'm completely for the idea that websites should not rely on specific browsers' implementation. I was just wondering if Opera was left behind from the crowd.

Hallvord R. M. Steenhallvors Friday, March 20, 2009 9:29:22 PM

I can not reproduce this in Opera 10 (haven't tested 9.6x yet) so hopefully it's fixed ..

Kamaleshkamalesh Tuesday, April 28, 2009 8:49:52 PM

Thanks for the insight into how Opera is handling and sometimes suffers from this type of JS-timing in web site design. Very interesting to see how Opera is unexpectedly fast, in some cases.

Btw, I'm trying out "delayed script execution" in the lates Opera alpha build, and it seems to a bit faster... Will keep playing with it and give feedback... wink

Hallvord R. M. Steenhallvors Tuesday, May 12, 2009 9:27:57 PM

By the way edvakf, we figured out the DOMContentLoaded bug: it occurs when delayed execution is enabled and sites use document.write() to add scripts. This will get fixed!

Write a comment

You must be logged in to write a comment. If you're not a registered member, please sign up.