A Blog From Behind the Trenches

Attack of the Bugs

Microsoft's IE9 standards tests vs. reality

, , , ,


The good news is that Internet Explorer 9 supports SVG!

The bad news is that Microsoft's standards support table could mislead people into thinking that IE9 is more standards compliant than other browsers.

Since I'm already talking about SVG, let's look at that as an example. If you read the description on their page, you will notice that it doesn't actually show SVG compliance as such. It shows how each browser does when running the 31 tests Microsoft created, when even the SVG 1.1 Tiny test suite has more than 150 different tests.

CodeDread has a published list detailing SVG support in different browsers. As you can see, IE9 still does poorly compared to other browsers. So while Microsoft's own page would give you the impression that IE9 has excellent SVG support, that is not the reality.

It's great that IE9 will support SVG, but I think Microsoft's page is rather misleading. Let's hope they are planning to make use of the full test suite at some point.

In conclusion: Microsoft should update the page to make it more obvious what it actually shows. It currently gives the false impression that IE9 beats all other browsers at standards compliance.

Not cool, Microsoft. Not cool.

Open letter to Microsoft regarding video on the WebBrowser tests, clarifications, and refreshing honesty

Comments

eliotcougar Wednesday, March 17, 2010 1:30:55 PM

Ha!.. Did anyone have any doubts?..

Daniel Aleksandersendaniel Wednesday, March 17, 2010 1:43:19 PM

Isn’t all their marketing like this?

Krio LythKriolyth Wednesday, March 17, 2010 1:58:18 PM

Unfortunately, all SVG test pages are unavailable now at Microsoft Test Center. We can't even say now if their tests were broken or not.

However, if we assume that their tests stick to standards, it's "clean" marketing - no lying while presenting good sides and concealing ugly ones.

FransFrenzie Wednesday, March 17, 2010 2:12:37 PM

I was already wondering how they came up with their values.

Brian HuismanGreyWyvern Wednesday, March 17, 2010 2:46:06 PM

Even though they spun their results to compare other browsers to the short list of things IE9 does support, it is good to see them releasing yet another test suite. More test suites are always good news. Let's get the other browsers moving on the tests that IE has cherry-picked for this list.

FransFrenzie Wednesday, March 17, 2010 2:51:37 PM

Originally posted by Brian Huisman:

Let's get the other browsers moving on the tests that IE has cherry-picked for this list.


Assuming they are correct. I recall seeing one or two wrong CSS2.1 tests when I looked over those Microsoft-submitted testcases and I only randomly clicked through a few.

Rijk Wednesday, March 17, 2010 3:31:10 PM

@Frans: the numbers here are more manageable, so I suppose these tests will get reviewed quickly enough. For a moment I thought they were *only* summitting tests where at least one competitor fails, but they aren't that nasty smile

BTW, IE9 does pass all the test in the automatic Selectors text, which is very nice.

sirnh1 Wednesday, March 17, 2010 4:04:00 PM

Originally posted by haavard:

but I think Microsoft's page is rather misleading.


It's Microsoft, what else did you expect? wink

(I would really like to see test results, from test not done by microsoft, because all I have is windows XP and that one doesn't support IE9 sad ..., so thank your for the link to codedread smile )

Plazzmex Wednesday, March 17, 2010 4:40:43 PM

People. I think we need to share this post as for much people as we can. Post it on your blogs, on your websites, forums etc..

Luchio Wednesday, March 17, 2010 6:57:19 PM

Well, they are friendly enough to provide test cases and tell you where each browser fails. Aside from the doubtful marketing standpoint (releasing only tests that they pass!), I think this page can be of use for Opera's, and other browser's development teams, as it identifies very specific bugs.

prd3 Wednesday, March 17, 2010 8:21:24 PM

Originally posted by Luchio:

I think this page can be of use for Opera's, and other browser's development teams, as it identifies very specific bugs.


The official testsuite already does that. Also, Microsoft's tests contain several errors. So no, Microsoft's tests are not useful. If they had bothered to let other people check them first, perhaps. But right now it's just wrong marketing garbage.

Cutting Spoonhellspork Wednesday, March 17, 2010 9:31:58 PM

"This is how many tests (in each set) our browser will pass"

...or that's the message *I* took away from it.

http://ie.microsoft.com/testdrive/Default.html

Opera 10.51 RC1 handles most of that crap easily. On an N280 netbook running XP, in Power-Saver settings, with 10.10 running more than a hundred tabs in the background. They've got a ways to go.

d4rkn1ght Wednesday, March 17, 2010 9:48:10 PM

Originally posted by mgillespie:

It's Microsoft, the most corrupt corporation on this planet, they will do what they need to do to ensure domination and control in every sector they trade in.


And don't forget that some people still swear by that serious flawed company. They love to put down on Opera, Apple, Linux, or any other company that truly had innovated.

masterofopera Wednesday, March 17, 2010 10:32:37 PM

Thanks for 100% clearness, haavard. smile

At the first moment I saw the chart, I was shocked,
but I thought later, this this can not be true, Opera "was"
the the browser with the best SVG support.
Now I feel reassured. cheers happy
Some things seems to never change at Micro$oft.

Edit: I hope the people will spread it.

Daniel HendrycksDanielHendrycks Wednesday, March 17, 2010 10:55:05 PM

Can they just be honest, once?

AtTheOpera Thursday, March 18, 2010 1:12:10 AM

Haavard - Do you do anything other than complain? All you seem to like to do is stir up your own fanboys. Microsoft is truly making positive strides but all you seem focused on is chopping them down.

David BruantDavidBruant Thursday, March 18, 2010 1:26:55 AM

My impression is that they are trying to have their Acid test. The same way everyone was pointing at IE for their ridiculous score on Acid3, IE's trying to point at everyone else by submitting tests on which other major browsers fail.
Their dark green "100% line" is a very efficient communication idea.

Daniel HendrycksDanielHendrycks Thursday, March 18, 2010 1:47:30 AM

Originally posted by AtTheOpera:

Do you do anything other than complain?


He is informing people that Microsoft is lying.

Originally posted by AtTheOpera:

Microsoft is truly making positive strides


True, but they should not lie about it.

AtTheOpera Thursday, March 18, 2010 2:03:08 AM

The tabs below provide details and links to each of the tests we submitted to the various W3C working groups to help the web become more interoperable.



Note the emphasis on "tests we submitted". It doesn't claim to be all tests or the best tests, but ones that MS submitted.

Daniel HendrycksDanielHendrycks Thursday, March 18, 2010 2:16:46 AM

Originally posted by AtTheOpera:

It doesn't claim to be all tests or the best tests, but ones that MS submitted.


But, it certainly was quite a diminuendo. Don't worry, Haavard was just disliking the fact that they used tests created by themselves to make it look 5x better than what it is. Kinda like Chrome and the V8 benchmark. And, obviously, he did not like that they did not implement Theora. cry

Cutting Spoonhellspork Thursday, March 18, 2010 2:37:04 AM

Originally posted by AtTheOpera:

Not the emphasis on "test we submitted". It doesn't claim to be all tests or the best tests, but ones that MS submitted.


Oddly, your typo makes your statement "correct". There was NOT real emphasis on the submission component, at a glance it looks like the inclusive listing of all important tests. Why? Well, most people seem not to even know what a "browser" is...

Pity that Apple's flawed xServe comparison page is no longer up.

prd3 Thursday, March 18, 2010 10:44:52 AM

Originally posted by AtTheOpera:

Haavard - Do you do anything other than complain? All you seem to like to do is stir up your own fanboys. Microsoft is truly making positive strides but all you seem focused on is chopping them down.


So it's OK for Microsoft to post misleading claims, but not to point out when they do so?

If Microsoft is making such positive strides, why can't they do it without lying?

What good do positive strides to if they are going to keep lying to people?

It's like one step forward, two steps back.

You know as well as everyone else that the page is blatantly misleading, and now lots of people think IE9 has better support for open standards than other browsers.

Why are you defending liars and attacking people who expose liars?

prd3 Thursday, March 18, 2010 10:46:47 AM

Originally posted by AtTheOpera:

Note the emphasis on "tests we submitted". It doesn't claim to be all tests or the best tests, but ones that MS submitted.


Never mind the fact that there are huge errors in their tests... If Microsoft had been honest, they would have run browsers through the full existing testsuites instead of cherry-picking a tiny amount of tests they optimized IE for.

Why didn't they optimize IE for the existing tests instead? Why did they have to write their own (with multiple errors)?

We both know the answer to that.

FransFrenzie Thursday, March 18, 2010 10:53:40 AM

Originally posted by AtTheOpera:

Haavard - Do you do anything other than complain? All you seem to like to do is stir up your own fanboys. Microsoft is truly making positive strides but all you seem focused on is chopping them down.


Complaining that Opera didn't support border-radius while it supported much more useful CSS3 selectors near-perfectly for ages is chopping down. Pointing out that Microsoft is making misleading statements is chopping down? Hah!

Originally posted by AtTheOpera:

Note the emphasis on "tests we submitted". It doesn't claim to be all tests or the best tests, but ones that MS submitted.


It's more like the small print than "the emphasis."

By the way, those CSS3 selector tests (and various others) are very inaccessible behind iframes etc. so frankly I can't be bothered to look at them. However, I'm quite inclined to agree that at least some of them are probably nonsense as prd3 says (I assume he checked them or some such? prd3?). Like I said, a bunch of those CSS2.1 ones sure were.

Charles SchlossChas4 Thursday, March 18, 2010 12:55:40 PM

Back to their old tricks

Tho I wonder how they maintain 4 different versions of IE at the same time? 5 tru 8

Luchio Thursday, March 18, 2010 2:04:53 PM

Originally posted by prd3:

Originally posted by Luchio:


I think this page can be of use for Opera's, and other browser's development teams, as it identifies very specific bugs.

The official testsuite already does that.


For SVG, yes, but there are other subjects such as CSS that are tested there.

Originally posted by prd3:

Also, Microsoft's tests contain several errors.


Not trying to defend Microsoft here, but could you give specifics? Or do you just assume that the tests are wrong because they come from the MS Factory?

FransFrenzie Thursday, March 18, 2010 2:09:15 PM

Anybody with Vista SP2 or higher, how does IE9 do on this test?

Oh, and this one?

icare Thursday, March 18, 2010 6:15:47 PM

There's something strange with their test page. With the SVG, for instance, they say Opera fails at "Default values of 'x', 'y', 'width', and 'height' attibutes for the 'svg' tag" and at the "Zero-length paths and 'stroke-linecap' attribute" and indeed on their test page, it fails.
Now, if I'm not wrong the first one refers to chapter 5.3, yet the official test, on the w3.org website seems to work pretty well.
The second one seems to refer to chapter 11.4 and yet the official test seems to work pretty well again.
Now, I don't know a lot about this coding stuff so maybe I'm all wrong, maybe I didn't see something but I'd like someone more experimented to tell me if I'm wrong or Microsoft did "forgot" something in their tests.

FransFrenzie Thursday, March 18, 2010 6:39:45 PM

I don't think the test you linked to for the first one tests the same thing? Anyway, given Fx' and Chromium's behavior this does indeed seem to be a bug in Opera, though I can't seem to find the relevant spec describing the behavior.

I think the second one is more related to pathLength here? http://www.w3.org/TR/SVG11/paths.html#PathElement

FransFrenzie Thursday, March 18, 2010 8:48:14 PM

Daniel HendrycksDanielHendrycks Thursday, March 18, 2010 9:34:52 PM

Originally posted by Luchio:

Not trying to defend Microsoft here, but could you give specifics?


Coding errors. There are invalid pages according to the W3C Validator.

FransFrenzie Thursday, March 18, 2010 9:36:43 PM

Originally posted by Daniel James Hendrycks:

Coding errors. Live invalid pages according to the W3C Validator.


I thought they took the SVG Validator down?

AgentCROCODILE Friday, March 19, 2010 1:28:52 AM

I want to see IE9 pass Acid3 with flying colir

Originally posted by AtTheOpera:

Haavard - Do you do anything other than complain? All you seem to like to do is stir up your own fanboys. Microsoft is truly making positive strides but all you seem focused on is chopping them down.


Positive strides such as making fraudulent "standards tests" and lying and producing porkies, as well as making their products a lot more bloated and slow than they should be?

Cutting Spoonhellspork Friday, March 19, 2010 6:48:03 AM

Let's see if they become the first to release a "slow" hardware-accelerated interface.

Keldian.-Keldian Saturday, March 20, 2010 12:51:26 AM

Just when I was getting enthusiastic about IE 9 and its makers' new commitment with REAL web development, this comes to light...



Personal memo: Don't be too enthusiastic with new IE release announcements, ever again. doh

FransFrenzie Saturday, March 20, 2010 8:53:38 AM

Originally posted by Keldian.-:

ust when I was getting enthusiastic about IE 9 and its makers' new commitment with REAL web development, this comes to light...

Personal memo: Don't be too enthusiastic with new IE release announcements, ever again. doh


In all fairness, they are improving leaps and bounds. I just wish they wouldn't lie about how much they improved.

FransFrenzie Saturday, March 20, 2010 9:16:42 AM

That is good news, though I really don't see the use for the table. I would think it should be fairly easy for any browser vendor to pick a number of tests they pass (or write new ones and make sure they pass, same difference), consequently perhaps do some extra testing to increase the number of tests failed by other browsers and then publish it. IE9 passing all of its own testcases is meaningless and I can only think of intent to mislead or lack of thought as the reason behind publishing it.

Cutting Spoonhellspork Saturday, March 20, 2010 9:19:43 AM

Well, it keeps their name in the press.

FransFrenzie Saturday, March 20, 2010 9:25:12 AM

I'd say that falls under intent to mislead, though probably no worse than most marketing when interpreted like that.

icare Saturday, March 20, 2010 7:00:31 PM

Originally posted by Frenzie:

I don't think the test you linked to for the first one tests the same thing? Anyway, given Fx' and Chromium's behavior this does indeed seem to be a bug in Opera, though I can't seem to find the relevant spec describing the behavior.



The microsoft URL contain the words "/chapter_05.3.svg" and "/chapter_11.4.svg" at the end. That's why I thought it was refering to this and this (plus the word "stroke-linecap" in the second test). But you're probably right, if Fx and Chrome pass the test, it must be something in Opera...
Thanks for your attention.

PozsonyiMarkCsendesMark Saturday, March 20, 2010 11:33:35 PM

Microsoft?
Failsoft?

They can't hide the truth from the experts ninja headbang

Quadunit404 Sunday, March 21, 2010 2:06:42 AM

Originally posted by codedread:

IE9 Preview 1 is the first Microsoft browser to natively support any form of SVG. Let's hope they achieve parity with Firefox 1.5 by the time it is released.

lol

coolfigaro Monday, March 22, 2010 9:16:40 PM

While I agree that at first glance the IE Testing Center page does make the IE9 preview look more stellar than it might be, I don't think it was that egregious.

In fact, I think Haavard's inclusion of his doctored screenshot is far worse. He removed the other browsers, the relative standards the tables were referring to, AND the column that indicates these were MS submitted tests. His inclusion of that image misrepresents the situation far worse than MS's original page.

The page has good text describing how these were specific ambiguities that MS wanted to clarify in the specs. Their preview is intended for developers. When a developer takes a look at a page describing standards (or really anything technical for that matter), he or she should know better than to just look at the graphics. I sincerely hope you don't take that approach to reading the standards themselves. That's how you end up with crap like this: http://imgur.com/oMgGG.png

Bottom line: if you really like web standards, then you should be encouraging the work they're doing to catch up and to clarify/improve the standards.

Cutting Spoonhellspork Monday, March 22, 2010 9:40:38 PM

1) Whose site put forth that test? I fail to spy an address.

2) A cropped graph? Not nearly so bad as Shankland's manipulated graphs on cNet a while ago. (Cut the first 300 out of one, showed only Dromaeo DOM for Dromaeo total)

FransFrenzie Monday, March 22, 2010 10:36:25 PM

Originally posted by coolfigaro:

In fact, I think Haavard's inclusion of his doctored screenshot is far worse. He removed the other browsers, the relative standards the tables were referring to, AND the column that indicates these were MS submitted tests. His inclusion of that image misrepresents the situation far worse than MS's original page.


Yeah, the original page showed other browsers doing far worse than Opera. He didn't capture quite what kind of an incorrect image the table gives. As I already said, saying "we pass our own tests" is either a clever marketing ploy (misleading without really saying anything untrue) or lack of thought.

Originally posted by coolfigaro:

I sincerely hope you don't take that approach to reading the standards themselves. That's how you end up with crap like this: http://imgur.com/oMgGG.png


What about crap like this? What about Opera's DOM compliance compared to IE's? It's nice that they're moving forward, but the notion that IE is better in anything but a few meager details is ludicrous.

prd3 Tuesday, March 23, 2010 7:29:59 AM

Originally posted by coolfigaro:

In fact, I think Haavard's inclusion of his doctored screenshot is far worse. He removed the other browsers, the relative standards the tables were referring to, AND the column that indicates these were MS submitted tests. His inclusion of that image misrepresents the situation far worse than MS's original page.


How does the illustration image attached to the blog post misrepresent the situation?

It doesn't, and of course you know that, but you couldn't resist attacking the messenger, could you?

The fact is that the page falsely gives the impression

You are dishonestly relying on people to read the fine print to understand what the page is all about, but the proof is in the pudding: Hardly anyone got it, which is why "IE9 is more standards compliant than other browsers" was spreading across the web.

The page has good text describing how these were specific ambiguities that MS wanted to clarify in the specs.


It had a wall of text, a fine print, that most people didn't bother to read. And Microsoft knows this.

Bottom line: if you really like web standards, then you should be encouraging the work they're doing to catch up and to clarify/improve the standards.


If only they could stop lying.

Jimtoyotabedzrock Friday, March 26, 2010 2:26:56 PM

Originally posted by Frenzie:

I was already wondering how they came up with their values.



The same way banks come up with there own tests. It's literaly like telling a college student to write the questions for there own test.

beritaku Wednesday, April 14, 2010 1:30:37 AM

Is this comparison table cart legit and creditable?

Write a comment

You must be logged in to write a comment. If you're not a registered member, please sign up.