Testing My Opera
By Esteban Manchado Velázquez. Friday, 19. September 2008, 13:16:40
Here at Opera, we make more things than the desktop browser. Apart from the browser for all the other platforms and Opera Mini, we make some internal systems, some server-side support for the browser, and some public sites. Probably the most well known, non-browser Opera product is My Opera (yes, this very site!). Being such an important product, it's in the hands of very smart people (and me), and it goes through extensive automated testing to catch as many problems as possible, as early as possible.
The automated testing for My Opera is based on three things:
The functional test suite is built with Test::WWW::Mechanize and uses special My Opera testing installations, each one connected to its own private database. That way, we can recreate the database before each test and ensure a stable and reliable environment for the test execution. This is currently the biggest and most mature one, and it has helped a lot in avoiding regressions when refactoring code and fixing bugs.
The unit test suite is a standard Perl test suite that covers the most important modules of My Opera. Again, we use private databases for the tests that do need them, so we ensure a reliable testing environment.
Last, but not least, we have everything hooked up in continuous integration so we have testrun information for every commit. When any test suite catches errors, we are automatically notified, and we can track which changes produced which errors.
All in all, we are quite happy with the automated tests for My Opera, as they give us freedom to work with the code without fear of breaking things without noticing. There is, of course a lot of room for improvement, and we are always tweaking and enhancing it.
The automated testing for My Opera is based on three things:
- A functional test suite.
- A unit test suite.
- Our beloved continuous integration server, that executes the test suite at every commit and shows the results to the whole team.
The functional test suite is built with Test::WWW::Mechanize and uses special My Opera testing installations, each one connected to its own private database. That way, we can recreate the database before each test and ensure a stable and reliable environment for the test execution. This is currently the biggest and most mature one, and it has helped a lot in avoiding regressions when refactoring code and fixing bugs.
The unit test suite is a standard Perl test suite that covers the most important modules of My Opera. Again, we use private databases for the tests that do need them, so we ensure a reliable testing environment.
Last, but not least, we have everything hooked up in continuous integration so we have testrun information for every commit. When any test suite catches errors, we are automatically notified, and we can track which changes produced which errors.
All in all, we are quite happy with the automated tests for My Opera, as they give us freedom to work with the code without fear of breaking things without noticing. There is, of course a lot of room for improvement, and we are always tweaking and enhancing it.
WIDGETIZE
Maulkin # 22. September 2008, 00:21
PaulHammant # 26. September 2008, 21:38
I'm one of the Selenium developers. Selenium being the defacto standard testing tool for web-apps these days. Its open source, and tests all your competitor's browsers including the one that Google released recently (they contributed the code days after the release). It also works on Windows, Linux and Mac as testing platforms. Selenium-RC is the bit that needs the most help reliably bringing up Opera for a test suite.
Anyway, we sorta have an Opera capability, but its second class compared to the others. Could you help out? We're pretty happy about getting help from wherever we can, and we really love Opera.
Here's hoping !
- Paul
http://selenium.openqa.org
PS - you could also use it to test MyOpera too ;-)
zoso # 29. October 2008, 09:31
Sorry for the (very) late reply.
About Selenium, yeah, I knew it. I had used it before, with Opera even, but when I started building the test suite for My Opera I couldn't figure out how to make file uploads work (and I imagined it would be a different battle for every browser, so I didn't want to spend so much time with it). That, coupled with a site that almost didn't have any Javascript, made me go for a purely HTTP-based system.
Actually I love Selenium myself, but I think we're looking at Watir, so I'm not sure I'll have the "chance" of using it...
As for helping with the Opera compatibility, what's in there to do? I'm not sure I can help myself, but if there's some bug in Opera that needs to be fixed, maybe I can nudge someone to bump the priority a bit.
PaulHammant # 29. October 2008, 22:31
Selenium does not support file uploads (yet). That not withstanding it's *an order* more popular than its competitors. Google run a farm of Selenium-RC servers and run (last published figures) run 55,000 tests a day for various of their apps. I have not heard of Google using WATIR at all, at least all their talking on the topic cites Selenium rather than WATIR. My employer (ThoughtWorks) also has customers that have grids of Selenium-RC instances. Gomez.com have a grid of Selenium-RC instances that similarly churn tests away day after day for their customers.
Selenium's first-class citizens are Firefox and Internet Explorer. Selenium-RC worked with Chrome the same day of its launch (Google have some team members on the Selenium family of open source projects). There is partial support in Selenium-RC for Safari and Opera.
What we need in order to get Opera to the "first class" place for Selenium-RC compatibility:
1) a way to launch the executable with a command line switch specifying a proxy server for that instance only. typically we run Selenium-RC on localhost:4444 and it is a perfect HTTP proxy server that adds Selenium test capability to the pages it is proxying.
2) an enhanced mode for testing that allows stepping past the same origin policy. For Firefox this is 'chrome' mode and built in. Its not a vulnerability as you have to direct the browser from the command line towards that local resource. For IE it is "HTA" mode. We can do "proxy injection" as an alternative, but nothing is as good as being able to bring up the browser in a special mode from the command line.
3) profile management from the command line, In firefox we by default launch without plugins - using a vanilla profile that's been enhanced to recognize the Certificate that Selenium-RC specifies. With greater profile management we're able to launch Firefox with a configurable set of plugins, because some advanced tests demand it. Again having a command line switch to allow this configuration would be great.
Thoughts?
- Paul
zoso # 5. November 2008, 12:39
Well, for us file upload is an absolute showstopper, and the "testing with a real browser with Javascript engine and such" wasn't that important. I didn't mean that Selenium is not powerful enough of course, as I said I love it myself and have used it in some personal project with very good results (very limited scope, but still).
About your concrete requests to improve Selenium-RC for Opera, thanks a lot! I'll send them to the appropriate parties, and will try to follow up on them. About (3) in particular, you can *choose* profiles from the command-line in Opera (-pd option), but you can't configure them AFAIK. Isn't that enough, preparing a generic Opera profile (and maybe ship it with Selenium-RC or similar) and use it?
hsiboy # 9. December 2008, 16:45
I'm a QA tester for a large (read top 10) website in the UK, and we are trying to automate all of our testing using Selenium Grid (much like google), and for me not having robust Opera support in selenium is a real kicker.
While Opera users only make up a small amount of our user base (0.9% of our daily user base) its still around 5,000 visitors viewing 200,000 pages daily that may not be getting an optimal experience.
Come on Team Opera, help make the web a better place for your devoted users!
Stuart.
--
Edited for poor grammar.
GoJoeGo # 9. December 2008, 17:58
Originally posted by hsiboy:
So what are you expecting from them? Somehow, you forgot to mention that
hsiboy # 11. December 2008, 21:25
i'm asking them (team opera) to support driving opera with selenium so more of us selenium users can test (reliably) with opera. If i cant use selenium to test with opera i wont. ergo the only people who will lose out if the Opera team don't help with selenium support are the opera devotees. the net result is that opera users will think opera is broken, when in fact it could be the website that's broken, however the developers of said site couldn't automatically test in opera, so they didn't.
Stuart.
zoso # 16. December 2008, 10:39
I have filed tasks and bugs internally for the concrete things Paul mentioned. Hopefully that will fix the problems with Opera and Selenium, but unfortunately we don't have a timeline for them.
If you have any other concrete problems with Opera, please tell me, and I'll file them too.
hsiboy # 30. December 2008, 11:23
That is great news.
If Team Opera can implement just points 1 & 2 from Paul's suggestion we will be in much better shape for testing with Opera.
Stuart.