Haavard

A better benchmark: Benchmarking Browsers with Real Websites

Friday, 3. July 2009, 11:18:47

opera, benchmarks, performance, browsers

We are all used to all the artificial JavaScript benchmarks out there by now. What they have in common is that they all test small parts of JavaScript, and none of them are testing real-life performance.

Now someone has spoken up, and pointed out that JavaScript-only benchmarks are a bad idea:

The problem with browser benchmarks today is that they are extremely synthetic. A majority of them are Javascript only because its easier to test.

Better yet, they created a benchmark which measures the actual loading time of different sites in various browsers. This means that the benchmarks do not rely on network speed, nor are they narrowly focusing on JavaScript.

The results will probably vary depending on what sites you choose to test, but the average loading times in this particular test should put to rest the misconception that Opera has somehow fallen behind when it comes to performance.

Results:

Opera
Safari
Chrome
Firefox
IE

I don't think Carakan is going to have much of an impact on these more realistic benchmarks. But people will probably continue to quote the flawed JS-only benchmarks in the future, so it will be nice to beat everyone else there too.

Opera Unite powered BitTorrent trackerIs Apple's WebKit a patent minefield?

Comments

xErath # 3. July 2009, 11:26

Originally posted by haavard:

so it will be nice to beat everyone else there too.

Nice statement

godjonez # 3. July 2009, 11:41

I have to agree on my own experience. While I have seen that currently Opera's JavaScript performance is not on par with other browsers (Firefox and Chrome both seem to be much better in this regard), the overall browsing experience still feels smoother with Opera. This is especially on sites that focus on lots of content and less JavaScript.

Can't wait for Carakan so that Opera goes back on top with JavaScript too.

dbloom # 3. July 2009, 12:41

To be fair, this guy's tests aren't very realistic either. They don't factor in network latency, which would benefit Chrome and IE8 because they can parse ahead to speculatively download external resources.

rafaelluik # 3. July 2009, 13:09

That post is awesome!
Opera Rules!

haavard # 3. July 2009, 13:09

It seems that he specifically wanted to leave out network factors. Those can be extremely variable. He also addresses it in the FAQ:

Other features like DNS look-ahead (likely cached) or different networking engines probably costs microseconds compared to the milliseconds it takes for a page to load.

sandalian # 3. July 2009, 14:38

Agree, we can't judge a browser's performance just based on javascript test.

dbloom # 3. July 2009, 15:24

Originally posted by haavard:

It seems that he specifically wanted to leave out network factors. Those can be extremely variable.

Not if you test it right. There are proxies out there that can simulate latency and limited bandwidth in a consistent manner. These factors could even have been simulated by simply using a PHP script that sleeps to delay the response.

Edit: Really, I'd like to see a test of multiple browsers against multiple network connection and system performance factors, so users can see what is the fastest browser for a slow computer on a fast connection, a fast computer on a high-speed but also high-latency connection (sattelite?), etc.

Khaled-Khalil # 4. July 2009, 05:47

i just tested the benchmark (with the same sites) by my own, and got even weirder Opera/FF ratio.
AMD Athelon Neo 64 1.6 GHz 2 GB memory
LinuxMint 7 64 (kernel 2.6.28-11)
Opera 10.00 Beta Build 4464 (Site Average: 89.7 ms)
Konqueror 4.2.2 (Site Average: 194.4 ms)
FireFox 3.0.11 (Site Average: 285.4 ms)

to be fair, keep in mind my experience was applied on latest Opera snapshot, while for Firefox and Konqueror they are latest stable release.

Khaled-Khalil # 4. July 2009, 05:55

Originally posted by haavard:

I don't think Carakan is going to have much of an impact on these more realistic benchmarks

as long it will not be worse it will be more than welcomed by users, Opera is already excellent in CSS and DOM, most of the claims about its downsides goes for JS, so if it could be handled as good as CSS, that of course make difference.

xErath # 5. July 2009, 13:29

iPhone OS 3 on iPod touch
http://www.pcmag.com/article2/0,2817,2349677,00.asp

And despite the much faster JavaScript performance, web pages load slightly slower in OS 3.0 than in 2.2.

Which means, javascript performance is hardly significant most of the times.

AgentCROCODILE # 8. July 2009, 09:10

Also depends on config of the computer. People quote ridiculous memory "requirements" simply looking at the task manager. What they are not seeing is the VM pool being used etc.

I can run Opera under 96MB of RAM and still have room for other apps like MS Paint and File Manager and Task Manager and VLC Media Player (Windows NT)

Can't do that with Firefox which supposedly uses "less memory" (but can't run under 96MB of RAM according to system reqs and needs Windows 2000 or above....Hmmm...tells me something about the "not-so resource hungry" Firefox)

And IE wouldn't be 5th if you tested other browsers (such as Lynx which has super fast SVG rendering capabilities)

oke # 26. July 2009, 17:29

@Haavard
I know Opera does not give dates for proposed releases but can you at least give us some estimates when we might start to see action on Carakan? (this year still, first half 2010?)

I mean the latest news on Carakan was 5 Feruary only.

hellspork # 22. August 2009, 21:39

Well Opera ran great on my sister's laptop. 9.64 she's got.

800MHz Coppermine Celeron.
128MB RAM.

No skipping on YouTube, and with VLC plugin she could play 720p content in the browser.

Funny Opera's recommended requirement is Firefox's minimum. And it still runs on practically no memory. Even win98/32MB won't kill a person.