Skip navigation.

Log in | Sign up

REST ASSURED

Quality rants by Opera QA

Fabulous Feedback

By Tim Altman. Friday, 8. August 2008, 19:59:18

Though Opera Software's Quality Assurance (QA) department works tirelessly to find problems that affect end-users (that's you), it's impossible to test all possible configurations and to find all possible problems. Thus, we rely on end-users to help find the problems we've missed.

There are several methods end-users can use to communicate their feedback to us:

The BTS is the best method when end-users run into issues they believe are bugs and they want them fixed. QA employees at Opera Software spend much of their time in the BTS, filing and analyzing bug reports, communicating with developers, and verifying bug fixes. The system makes it easy to track issues and compile relevant information, so any developer viewing a bug report can quickly understand the issue and spend time fixing it rather than analyzing it and gathering information.

In most cases, the bug reports we receive have all the required information, so there is no reason to contact bug reporters. In a small number of cases, we need additional information, so the bug reporter is contacted through the BTS.

The Opera Community Forums and newsgroups are a great place for end-users to discuss issues and gather information before filing bug reports. The forums and newsgroups are not an official way of contacting Opera's QA dept., though we do frequent both the forums and newsgroups. Thus, there's no guarantee that an Opera employee will see your post, let alone act upon it and follow-up internally. Developers generally do not visit the forums and newsgroups, as they're busy coding!

Under the Help menu on Opera for Desktop, users can report problems with the site they're currently viewing and provide a short description of the problem. This information is sent directly to Opera where it's collated in a database. We use this information to augment bug reports by indicating the frequency of problem reports. Reporting a site problem this way does not submit a bug report. Users comfortable reporting bugs should use the BTS instead.

Many of our development teams maintain blogs to communicate with end-users, such as the Desktop Team blog, Mac Team blog, Core Team blog, Opera Mobile blog, and Opera Mini blog. Though these blogs do allow comments, blog comments are a suboptimal means of leaving feedback on a software release. Instead, forums or newsgroups should be used for discussing issues and the BTS should be used for submitting bug reports.

Your feedback is important, so please take a moment to think about which method is most relevant for the feedback you want to provide.

Alexa Global Top 500 Validation ResearchYou can be a site compatibility wizard!

Comments

Mickeyjoe_irl 8. August 2008, 20:43

Very useful information :up:

Muzzlehatch 11. August 2008, 07:13

Hiya Junyor,

good article, although I feel unaccountably somewhat chagrined for my approach to submitting bugs: checking with you first whether something is a bug! There is, however, glaring omission in your article is that the current bugtracker is a bit of a black hole and that this is more by accident than by design. I'm sure it helps people to know if their bug has been accepted or not or whether it's a duplicate of an existing issue, etc.

Junyor 11. August 2008, 13:40

@Charlie_X: As the article says, forums are a great place to discuss bugs before submitting them to the BTS. The BTS is working as designed.

AsaDotzler 12. August 2008, 07:27

Hey again :-) It's obvious that you don't have enough people or systems in place to test every possible configuration -- no organization does. But I'm interested in what kind of resources you do have?

Can you describe the Opera QA team? I'm interested in the number of people you have, how you're organized around features and platforms, etc.

It'd also be cool to know a bit more about your basic toolset like your bug/issue tracker, your automation setups for desktop and mobile, your performance and stability testing and metrics, etc.

Finally, can you tell us about your team's normal daily routines, functional testing, bug reporting, automation development, bug triage and testcasing, etc. etc.


On another subject, and one mentioned by Charlie X above, why isn't your bug tracker open to those who submit reports? That's "working as designed?" You're basically saying "please help us by donating your time and energy to report problems, but you can't follow along and that's the way we like it."

Can you explain why you designed the process to be so opaque? What harm would there be in allowing reporters to follow their own report through the process of fixing? You'd even get a free verification on that fix on the very configuration you lacked in the first place. It makes little sense to me why you'd prefer the blackhole to something more open and participatory.

Thanks for starting this blog. QA rocks!!

- Asa

Junyor 12. August 2008, 14:41

Asa: I can provide some information about the Desktop QA team. As indicated in the Desktop Team FAQ, there are seven people on the Desktop QA team. Each member of the team has various feature responsibilities, which overlap so we have (at least) a primary and secondary contact. Most of our team runs multiple operating systems, though we do have test managers for each of our major platforms (Windows, Mac, and Linux/UNIX).

Our bug tracker is currently a heavily modified version of Bugzilla. The BTS is closed for a couple of reasons: 1) as part of agreements with our partners, 2) corporate policy, 3) to protect the identity of our volunteers, and 4) technical limitations of the system we use. We periodically revisit this policy and evaluate different solutions for the reasons I mentioned.

Our main automated testing system is a proprietary system originally designed by Ian Hickson (and described at in his blog). We also use various proprietary tools for stability, memory, and performance testing, as well as tools like fuzzers and SunSpider.

WillYum 19. August 2008, 16:37

:lol: While the lecture was intersting the stats on the testing methodology were far more insightful. Cool and thanks. One question on the virtual thwaping, "blog comments are a suboptimal means of leaving feedback on a software release."

I can't see how they'd be suboptimal.. Sure they don't get into the system but how can you tell someone who sees a window to a developer that it's suboptimal, especially if they are discussing Opera?

Yum
P.S. I realize that this is valuable info on how Opera works and how it wants to interact with the OC, however, I believe a little form forcing function would be more appropriate. If you want forum discussion instead of comments, then display links to the forums instead of a comments box in official blogs. :smile:

Mickeyjoe_irl 19. August 2008, 22:04

If you want forum discussion instead of comments, then display links to the forums instead of a comments box in official blogs.


I don't know if they should get rid of the comments box, but including a bit of "boilerplate" text with the links and instructions on the best way to report bugs/issues is a good idea.

Junyor 20. August 2008, 07:25

@WillYum: You're assuming that developers read through the comments to the blog posts. That usually isn't the case.

@Mickeyjoe_irl: It's in the Desktop Team FAQ.

Mickeyjoe_irl 20. August 2008, 09:21

No offense, but I never even knew there was a FAQ until you posted the link in your earlier comment.

The more obvious you make the "correct" feedback procedure, the more likely you are to get the feedback you want in the method you want it. Sometimes you have to spell these things out for people (like me :wink: ).

WillYum 21. August 2008, 00:46

Here, here, Mickeyjoe :smile: And I often seen dev responses in the Desktop Team blog comments... or at least things that look like responses. They reenforce the improper behavior by responding.

Yum

Mickeyjoe_irl 29. August 2008, 22:00

Just saw the announcement for the latest 9.60 snapshot.

Thanks for taking my suggestion on board Junyor. :smile:

Junyor 30. August 2008, 05:50

:smile:

53north 18. December 2008, 17:25

I think as long as opera do tests with all the major search engine portals then 70% of complaints should be dealt with.
My beef at the moment is google translator select boxes not working in opera mini. Babelfish is ok.

Write a comment

You must be logged in to write a comment. If you're not a registered member, please sign up.