Opera Denied Gzip - What to do?

Opera Denied Gzip - What to do?

This problem has been mentioned before but I think it should be taken much more seriously. Some very large and well-known websites send gzip-compressed content to Internet Explorer and Gecko browsers, while Opera gets the uncompressed content that is much bigger and takes longer to load. You can read more about content compression at the Web Reference.

Which sites do not send Opera compressed content? Here is a list I've compiled, there are certainly more.

• Google including all Google services (Google News, Google Maps, GMail, etc)
  
• Yahoo! including all Yahoo services (e.g. Yahoo! Mail), but not Yahoo! Search
  
• eBay
  
• Amazon.com
  

Sites decide upon the classical user agent sniffing. Most sites send gzip compressed content only if the browser's user agent string contains "MSIE" or "Mozilla". Now, Opera's default user agent string is the IE one, but that doesn't always help. For instance, Amazon.com doesn't send compressed content to Opera with any of Opera's user agent strings, because it detects the word "Opera" and leaves the content uncompressed.

How do you detect such sites? Usually, Opera's Info panel will do. Load the page ID'd as Opera, and check the page size. Then reload with Mozilla and with IE, and see if the page size gets smaller. If you feel suspicious, you can also use the Proxomitron to hide Opera from your UA completely, or check the size in another browser (Firefox or MSIE).

What should be done? Well, I see some entries in ua.ini which might be there because of websites not sending compressed content. But the sites I listed above are not there and shouldn't be either - if they were, the number of Opera users would drop to zero in their stats! These sites should be contacted pushed to make a change.

Interesting thread. So, what do you suggest?

Hi again. I'm sorry I had to finish the post in a hurry. I'd like to add a few things.

Browsers do in fact send an HTTP header which mentions all supported compression methods. The header is called Accept-Encoding and all browsers up to date, including IE, send it. Opera doesn't do it differently. There is no need to sniff for browsers to try to guess whether it supports gzip or not (if it didn't, the user would only see a screenful of junk), because this header tells it. Yet the problematic sites I mentioned check the user agent string for MSIE or Mozilla.

Someone (non-troppo IIRC) suggested that they do this because of browser history - Netscape4 sends the header saying it supports gzip even though it doesn't. However, I have Netscape Communicator 4.8 which seems to support gzip well. So there's still no reason to do browser sniffing. And you could still check Accept-Encoding and just exclude Netscape4 from its user agent string.

Practically the only thing that can be done is to contact these websites and ask them to correct the browser sniffing (change to Accept-Encoding sniffing). Opera's default UA setting is MSIE which works for some websites. However, as I said, for example Amazon.com seems to actually deliberately exclude Opera, because none of the Opera-ized user agent strings work. And as I said, ua.ini (which hides "Opera" completely from the user agent string) is not a solution, because this way Opera wouldn't be shown in the statistics of those sites, which could be harmful.

Just for completeness, here is the feedback I sent to Google:

[url]http://groups.google.com/group/GMusers/browse_frm/thread/e925b652d797d9cc/68b7750d6fd58e41#68b7750d6fd58e41

I did get back a reply by a real person sometime later suggesting to look at this but no further detail emerged. I remember sending feedback to Yahoo too.

Interestingly, [url]http://www.independent.co.uk compresses for Opera and IE but doesn't compress for Mozilla! Quite why I don't know.

And it was me who mentioned the Netscape 4 / IE masks as NS 4 theory (proposed to me by Tarquin); and yes a fix is trivial; not as trivial as you say though. Sadly even the Apache docs suggest basic browser sniffing to handle who gets compression, so I think developers have gotten carried away by this and Opera has suffered as a result.

The only way for this to change is for people to write and send feedback to the sites responsible...

Originally posted by non-troppo
The only way for this to change is for people to write and send feedback to the sites responsible...

Yes, but you have to admit just sending emails to Google, Yahoo, eBay and others isn't a piece of cake. Such market monopolists don't need to listen to their customers, do they? I see this is as rather serious browser discrimination and I would like Opera Software to intervene into the issue.

Now I also realize if Opera 8.10 is going to default to the Opera ID, this will result in non-gzipped pages with the default configuration also for those sites that don't deliberately exclude Opera (i.e. where the Opera MSIE user agent string works). Did someone mention it in this thread?

Thanks for pointing this out. Some quick stats for me (measures: main page/inline elements) with Opera and then UA 4 and 5 on pages you mentioned that I also frequent:

Amazon Opera: 75K/113K
Amazon Mozilla: 16K/74K (font is not rendered properly on Mozilla)
Amazon IE: 17K/200K (why inline elements is higher I don't know)

(With IE emulation on Amazon you get the balloon popup when you hover over the "See all 31 product categories" tab, something I never knew before.)

My Yahoo Opera: 154K/37K
My Yahoo Mozilla: 30K/37K
My Yahoo IE: 155K/37K (this time it's the main page size not helped)

Google News Opera: 125K/73K
Google News Mozilla: 28K/73K
Google News IE: 28K/73K (finally)

I remember someone from Opera (unofficially) said they would use their contacts in Google to ask about this; I wonder what ever happened to this?

Forbes sends Opera uncompressed pages, no matter the ID.

Didn't know about this. I agree with Jere, it's a serious problem.
It's strange that things like this happens so often with Google, which is a partner of Opera. I'm thinking of the "Google.com in English" problem - among other things.

Those of you interested in hunting Opera discriminating websites, consider using this small Perl tool I wrote:

Gzip Test

Screenshot...

gziptest.png

• Mail & Guardian Online sends uncompressed content to all Opera IDs.
  
• The Times of India sends uncompressed content when ID is Opera.
  
• The Globe and Mail sends compressed content only to IE. (Apparently it still deliberately excludes Opera because Opera's spoofed MSIE UA doesn't work)
  
• AltaVista sends uncompressed content when ID is Opera.
  
• Ask Jeeves sends compressed content to IE only (again, Opera is deliberately excluded)
  
• Weather Underground sends uncompressed content when ID is Opera.
  
• Bloglines sends uncompressed content when ID is Opera.
  
• CDnow Part of Amazon - same problem (Opera excluded)
  
• MySpace sends uncompressed content when ID is Opera.
  
• Target.com sends uncompressed content to Opera (all IDs)
  

During my gzip test session I also stumbled upon something weird. Travelocity sends gzip compressed content to Opera ID'd as Opera and MSIE, but not ID'd as Mozilla, nor any other browser

Jere: why don't you make the gzip tester an online one? I had actually half-written something similar in PHP some time ago but never finished it; I suppose it will be easier for you to get the test to spit out HTML...

Sure. I can make a version that spits out HTML. But I don't want to host it... if people are going to use it as much as I did yesterday night, (talk about crappy Friday nights?) I don't want it to use my bandwidth But indeed, an HTML version (could be run on one's localhost) could come very handy, as you could add a button / bookmarklet that lets you test a page directly.

K... updated: [url]http://www.vonpluten.com/gziptest/

Originally posted by non-troppo
Jere: why don't you make the gzip tester an online one? I had actually half-written something similar in PHP some time ago but never finished it; I suppose it will be easier for you to get the test to spit out HTML...

+1

as far as I understand this tool requires a pearl interpreter being installed on my machine, which I don't have. I'd like to test only 2 sites for this stuff and don't want to install a pearl interpreter just for that. Could anyone try www.spiegel.de and www.heise.de for me? The latter seems to load faster in FF than in Opera for me, and I wonder whether this gzip stuff may be the reason.

Originally posted by Jere
Sure. I can make a version that spits out HTML. But I don't want to host it... if people are going to use it as much as I did yesterday night, (talk about crappy Friday nights?) I don't want it to use my bandwidth But indeed, an HTML version (could be run on one's localhost) could come very handy, as you could add a button / bookmarklet that lets you test a page directly.

Maybe Javascript? Then you won't have to host it.
A UserJS that checks the active page when you double click on it or something.

Originally posted by WildEnte
as far as I understand this tool requires a pearl interpreter being installed on my machine, which I don't have. I'd like to test only 2 sites for this stuff and don't want to install a pearl interpreter just for that.

You're right; a Perl interpreter is required. (And a web server running locally if you want the HTML output). I'm sorry about that, but seeing that the tool requests a page up to 100 kB by average five times per test, I'm not fond of hosting it online.

Originally posted by WildEnte
Could anyone try www.spiegel.de and www.heise.de for me? The latter seems to load faster in FF than in Opera for me, and I wonder whether this gzip stuff may be the reason.

Sure.

Spiegel.de seems to send gzip compressed content to MSIE and Opera, but not Mozilla or Konqueror based browsers. So it should actually be slower in Firefox. (Shows that Opera is not completely alone with the gzip problem...)

Heise.de seems to send gzip compressed content to all browsers that support it.

Originally posted by WildEnte
Maybe Javascript? Then you won't have to host it.
A UserJS that checks the active page when you double click on it or something.

Are you thinking of XMLHttpRequest? Unfortunately, that is not possible, at least not in Opera. Opera doesn't let you request content from outside the domain of the page that uses the XMLHttpRequest object. And Opera doesn't support custom request headers.

Originally posted by Jere
Are you thinking of XMLHttpRequest? Unfortunately, that is not possible, at least not in Opera. Opera doesn't let you request content from outside the domain of the page that uses the XMLHttpRequest object. And Opera doesn't support custom request headers.

Too bad. Maybe Opera software can host the perl script?

OK, I quickly hacked up (i.e. 10 minutes so its ugly!) an online tester, uses cURL libraries via PHP to do the requests:

http://nontroppo.org/tools/gziptest/

button

Cool!

Seems your tester doesn't do HTTP redirects though (from what I can tell from the headers it shows for the results)

I set cURL to follow up to 10 Location: redirects, but the cURL library doesn't have any other options AFAIK, so for example:

http://nontroppo.org/tools/gziptest/?url=http%3A%2F%2Fgoogle.com&uas=

You'll see a 302 and redirect being followed. I don't download content bodies (just HEAD requests) so it cannot follow META directives; because of that it also don't bother analysing the difference in body sizes for compressed / uncompressed which your tool does nicely (but then it is much faster!). If I downloaded the bodies then these maybe possible, but I don't want to spend so much time on it. You're welcome to the source to improve if you know PHP, I can still host it if you don't want to risk the bandwidth.

BTW you don't check for deflate with your tool AFAICS from your source.

Originally posted by non-troppo:

BTW you don't check for deflate with your tool AFAICS from your source.

Hm, no I don't, but I haven't seen it being used anywhere?

Ok, so it should just be a matter of adding 'deflate' to the Accept-Encoding header, and modifying the script a bit to let the user know which method was used.

Yep; my regex is simply: Content-Encoding:.*(gzip|deflate) - I don't bother saying which one is used as, well, I'm lazy

Small update made to the online tester to fix some sites (www.independent.co.uk) that fail to compress for HEAD requests.

BTW, Yahoo feedback is here: http://cgi.yahoo.com/bin/feedback

ebay: http://pages.ebay.com/help/contact_us/_base/index.html

amazon: http://www.amazon.com/gp/help/contact-us/general-questions.html

Maybe you could host a "wall of shame" (or rather a list of discriminating sites). If we can get Opera to prominently link to it, such as in a press release, we may get a few of them to come around.

Is google still sending uncompressed stuff? Gmail always seems to take longer to load in Opera than FF, but I thought I read that Google would be supporting Opera fully around the time 8 came out. Wrong?

Also, I've noticed gmail has started not displaying possible contacts as I type their names in. Normally if I type Rob, gmail displays the email address of someone on my contact list called Rob and enters it when I press tab. Recently it has stopped doing this in Opera, but it still works in FF.

I wouldn't be surpised if Google starts heavily funding FF development and creates GoogleFox or something soon. I don't see why they are not changing a few bits of code to support Opera. Bet it wouldn't take a day.

Originally posted by AndyP:

I wouldn't be surpised if Google starts heavily funding FF development and creates GoogleFox or something soon.

off topic: Google registered the domain gbrowser.com a long time ago.

Originally posted by AndyP:

Is google still sending uncompressed stuff?

Yes, unfortunately.

Google have finally pulled their heads out of their a$$ and fixed this...

http://my.opera.com/dstorey/blog/show.dml/121098

Well, a look to www.webwarper.net says:

Your Web browser (Netscape Communicator 0 or compatible) does support GZip format of Web pages, which is necessary to save your traffic by WebWarper. But the Web server received the information that GZip is not supported!

You probably need to enable HTTP 1.1 procotol in your browser. It is a standard protocol supported by all sites and browsers today; however, the use of this protocol in the browser may be disabled sometimes.

For Internet Explorer:
please open Tools / Internet Options... in the main menu;
select "Advanced" tab;
find "HTTP 1.1 settings" section;
turn on both flags "Use HTTP 1.1" and "Use HTTP 1.1 through proxy connections".
After this, please restart the browser (close all browser windows and start the browser again).

Note: the changes will not affect static Web pages which are cached by your browser; but new pages downloaded to your computer will be compressed by GZip.

...
and since yesterday, I noticed that Opera is requesting the pages without compression(from PHPINFO()):
HTTP_ACCEPT_ENCODING text

how is that possible that it was changed? and how can I get it back???

Um, I still get the compression headers sent just fine in V8.51 or V9TP:

GET / HTTP/1.1
User-Agent: Opera/9.0 (Windows NT 5.1; U; en)
Host: news.google.com
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: en
Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
TE: deflate, gzip, chunked, identity, trailers
Connection: keep-alive

Are you using a proxy?

No, I'm not... I edited the ini files (no special settings, just added the wikipedia search engine, some open with xxxx application) and now that I'm trying again... it sends again the headers...

HTTP_ACCEPT_ENCODING deflate, gzip, x-gzip, identity, *;q=0

HTTP_TE deflate, gzip, chunked, identity, trailers

and what I think is going on(about why some sites doesn't send the content compressed) is because of the

Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0

Originally posted by atomo64:

Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0

Of course, content in format * isn't accepted

Originally posted by xErath:

Originally posted by atomo64:

Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0

Of course, content in format * isn't accepted

AFAIK * is for ANY encoding; and anyway, if what you say is what it means, then there's a mistake there, because Opera says that it supports encoding, but that you should not send encoded content, so the Accept-Encoding header would be like this:

Accept-Encoding: deflate, gzip, x-gzip, identity[COLOR=red];q=0.9,[/COLOR] *;q=0

and btw, dunno what's going on, I just opened Opera and again is requesting pages without encoding; there must be something wrong somewhere

Are you using Outpost Pro? Or AdMuncher? Both disable GZIP.

Originally posted by atomo64:

if what you say is what it means, then there's a mistake there

* means any content not otherwise listed. So Opera's field says, "send me deflate, gzip, x-gzip, identity or don't encode"; the field is OK.

Again, any firewall/ad blocker/proxy/virus checker may disable compression...

Originally posted by jp10558:

Are you using Outpost Pro? Or AdMuncher? Both disable GZIP.

none of them, nor any other soft like that

Originally posted by non-troppo:

Originally posted by atomo64:

if what you say is what it means, then there's a mistake there

* means any content not otherwise listed. So Opera's field says, "send me deflate, gzip, x-gzip, identity or don't encode"; the field is OK.

ok, but I think that the value of q should be changed then, so it would say that it prefers encoded than plain text

Originally posted by non-troppo:

Again, any firewall/ad blocker/proxy/virus checker may disable compression...

firewall?=>I use isafer, which is a winsock firewall and doesn't do anything on the content, so it is not that
ad blocker=> none, I use the own Opera iltering system
virus checker=> I use norton 2004, so... no.

so I still don't know why sometimes the pages are requested with compression and sometimes are not

Originally posted by atomo64:

ok, but I think that the value of q should be changed then

if q is not given it defaults to 1 IIRC so you don't need to specify it in the header

How are you testing whether your compression is changing? Can you give a header log to see if other headers change?

I'm using the phpinfo() function.

Originally posted by jp10558:

Are you using Outpost Pro? Or AdMuncher? Both disable GZIP.

Thanks - I was unaware of this. I use Agnitum Outpost, but I thought I had stripped it down sufficiently to stop this from happening.

When I turn it's off (it's safe at the home network, as it is NATted), the following header gets added, or actually, not removed:

accept-encoding
	deflate, gzip, x-gzip, identity, *;q=0 

The HTTP header tool I used


[edit]Searching at the Outpost KB for "gzip" yields this page:

If you want to enable page compression run Regedit from the command line and set the "Enablegzipencoding" key value in HKEY_LOCAL_MACHINE\SOFTWARE\Agnitum\Outpost Firewall to 1. However, please note that the Ads, Content and Active Content plug-ins will not work for sites that use compression.

IMHO it should be known by more people that firewalls block GZIP like this, even when it's not necessary. I don't use the ad blocker features.
[/edit]

As much as I hate dredging up old threads, but...

I've found a bug in some Apache 1 servers that refuse to gzip compress pages when the browser explicitly claims identity support, like Opera does. It may not always be web site browser sniffing! Apache 2 servers seem to be fine. I don't know the specific Apache 1 versions that are affected, but it would seem to be recent version 1s at the least.

(I actually discovered this a couple of months ago, but forgot about it until now, so details are a bit hazy, sorry.)

Any changes on the sites?

The four mentioned in the first post (Google, Yahoo, eBay, Amazon) all gzip the main HTML page and almost all the CSS and JS too. I think I might have seen a JS file on Yahoo that wasn't gzipped, but didn't compare in any other browser.

You can check yourself by using the Developer Tools in an Opera 10 beta and using the Network tools. Check the headers for a "Content-Encoding:gzip".