Opera Core Concerns

Posts tagged with "core"

Automated regression testing of the browser core

By Wilhelm Joys Andersen. Tuesday, 13. October 2009, 14:13:15

The cornerstone of all testing done on the core of the Opera browser is our automated regression testing system, named SPARTAN. The system consists of a central server and about 50 test machines running our 120 000 automated tests on all core reference builds. The purpose of this system is to help us discover any new bugs we introduce as early as possible, so that we can fix them before they cause any trouble for our users.

Step one: Preparing a build

Before SPARTAN can test anything, it will require a build to test. Our build system automatically creates builds every night and pings SPARTAN when they are ready. Developers and testers can also request their own builds from the build system, using any build tag they want, to test stuff from their own experimental branches before this is eventually merged into the stable mainline we base our products on.

Unlike other browser vendors we ship our browser on a variety of different platforms. So our core build packages do not contain just one binary, but several. One for each general product category. Each of these profiles have the same feature set and memory constraints as the platform they correspond to. The whole set of tests are run on each of these profiles.

Step two: Testing

When the SPARTAN server is informed about the existence of a new build it will add this build to its testing queue and distribute a few hundred tests to each of the test machines the next time they ask for more work. Each test machine works independently with its assigned tests. It will download the Opera binaries it has been told to use, and run its assigned tests on it. Once it has finished its batch of tests, it will pass the test results back to the SPARTAN server, and again ask what to do next.

If it ever runs out of new builds to test, for example during the weekend, it will look back at older builds and run any newly added tests on them too. This to ensure that we have a full history for each test, and at any time can determine when a fix or regression was first introduced without having to manually test things again.

We have several different types of tests:

Unit tests: Unit tests (or selftests), written by the same developers who write the running code, tests individual functions and APIs.
JavaScript tests: Our JavaScript tests test a wide array of different features on a functional level. This includes for example tests for the Selectors API, tests for common JavaScript frameworks, or any other feature we can interact with through JavaScript.
Watir tests: Many tests require some sort of user interaction. To test forms, for example, one must click buttons or checkboxes or type in text fields. To avoid having to do this manually, we have implemented support for the cross-browser Watir API. While others use this API to test their web applications, we use it to test the browser itself.
Visual tests: To test our stylesheet and graphics code, we need to test that our visual test cases look right. Some of our visual tests automatically compare two pages or two elements to determine whether they are the same. On other tests, the test machines will take a screenshot of the rendered page and pass it back to the SPARTAN server. If the SPARTAN server has seen this screenshot before, it will know whether that particular rendering means PASS or FAIL. If SPARTAN has never seen it before, the screenshot must be labeled as PASS or FAIL by a human. This is labour-intensive work we intend to further reduce through reftests.
Performance tests: A modern browser must not only pass tests for all its different features. It must also be fast. SPARTAN runs a number of different performance tests, both internally and externally developed, on our builds. If Opera becomes slower at any particular test, this will be flagged as a regression.
Crash tests: We create test cases for every single bug we analyze and fix, and SPARTAN runs most of these. Among our bug-based test cases are crash tests. If Opera can load these tests without crashing, the test has passed. If it crashes, we have reintroduced an old crash, and must fix it.

All in all, we currently run about 120 000 tests on each configuration in each build, but this number changes daily. We continuously write new test cases for bugs or test suites for new or old features, and we also copy any publicly available test suites we find useful. Right now we are also working on automating many of our previously manual tests, including memory tests.

Step three: Human intervention

Once the machines are done with their part of the job with any particular build, they will send an email to a human who will continue the work. SPARTAN will generate a report of changes between this build and the previous build. In most builds there are some tests that go from FAIL to PASS because we have fixed something. But there are also often regressions—tests that go from PASS to FAIL—because we accidentally broke something while fixing something else. This is expected, and is the reason for why we do regression testing. We know there will always be regressions, and need to find them as quickly as possible in order to fix them before they can cause any trouble for users or customers.

The human tester will analyze each regressed test. If a hundred different tests started failing at the same time, they could all have broke because of one regression, or there could be several different ones. For each unique regression identified the human tester will report a new bug and assign it to the developer responsible for the code that broke. Once a fix is ready, we will run all our tests again.

25 comments

Carakan

By Jens Lindström. Wednesday, 4. February 2009, 23:29:35

JavaScript, core, opera

Over the past few months, a small team of developers and testers have been working on implementing a new ECMAScript/JavaScript engine for Opera. When Opera's current ECMAScript engine, called Futhark, was first released in a public version, it was the fastest engine on the market. That engine was developed to minimize code footprint and memory usage, rather than to achieve maximum execution speed. This has traditionally been a correct trade-off on many of the platforms Opera runs on. The Web is a changing environment however, and tomorrow's advanced web applications will require faster ECMAScript execution, so we have now taken on the challenge to once again develop the fastest ECMAScript engine on the market.

The name Carakan, like the names of Opera's previous ECMAScript engines, Futhark, Linear A and Linear B, is the name of a writing system, or "script".

We have focused our efforts to improve upon our previous engine in three main areas:

Register-based bytecode

The last couple of generations of Opera's ECMAScript engine have used a stack-based bytecode instruction set. This type of instruction set is based around a stack of values, where most instructions "pop" input operands from the value stack, process them, and "push" the result back onto the value stack. Some instructions simply push values onto the value stack, and others rearrange the values on the stack. This gives compact bytecode programs and is easy to generate bytecode for.

In the new engine, we've instead opted for a register-based bytecode instruction set. In a register-based machine, instead of a dynamically sized stack of values, there's a fixed size block of them, called "registers". Instead of only looking at the values at the top of the stack, each instruction can access any register. Since there is no need to copy values to and from the top of the stack to work on them, fewer instructions need to be executed, and less data needs to be copied.

Native code generation

Although our new engine's bytecode instruction set permits the implementation of a significantly faster bytecode execution engine, there is still significant overhead involved in executing simple ECMAScript code, such as loops performing integer arithmetics, in a bytecode interpreter. To get rid of this overhead we are implementing compilation of whole or parts of ECMAScript programs and functions into native code.

This native code compilation is based on static type analysis (with an internal type system that is richer than ECMAScript's ordinary one) to eliminte unnecessary type-checks, speculative specialization (with regards to statically indeterminate types) where appropriate, and a relatively ambitious register allocator that allows generation of compact native code with as few unnecessary inter-register moves and memory accesses as possible.

On ECMAScript code that is particularly well-suited for native code conversion, our generated native code looks more or less like assembly code someone could have written by hand, trying to keep everything in registers.

The register allocator is designed to be architecture independent, and so is the code generation "frontend" where most of the complicated decisions are made. We have initially implemented a backend for 32- and 64-bit x86, but some preliminary work has already started to generate native code for other CPU architectures, such as ARM.

In addition to generating native code from regular ECMAScript code, we also generate native code that performs the matching of simple regular expressions. This improves performance a lot when searching for matches of simple regular expressions in long strings. For sufficiently long strings, this actually makes searching for a substring using a regular expression faster than the same search using String.prototype.indexOf. For shorter strings, the speed is limited by the overhead of compiling the regular expression.

For more complex regular expressions, native code generation becomes more involved, and improves performance less since the regular expression engine is already reasonably fast. The base regular expression engine is a newly developed one, but it's already made its debut in Presto 2.2 (the browser engine used in Opera 10 Alpha). It's fundamentally a typical backtracking regular expression engine, but does some tricks to avoid redundant backtracking, which usually avoids the severe performance issues a backtracking regular expression engine can have on specific regular expressions.

Automatic object classification

Another area of major improvement over our current engine is in the representation of ECMAScript objects. In the new engine, each object is assigned a class that collects various information about the object, such as its prototype and the order and names of some or all of its properties. Class assignment is naturally very dynamic, since ECMAScript is a very dynamic language, but it is organized such that objects with the same prototype and the same set of properties are assigned the same class.

This representation allows compact storage of individual objects, since most of the complicated structures representing the object's properties are stored in the class, where they are shared with all other objects with the same class. In real-world programs with many objects of the same classes, this can save significant amounts of memory. It can be expected that most programs that do create many objects still only have a few different classes of objects.

The shared lookup structures for properties also allow us to share the result of looking up a property between different objects. For two objects with the same class, if the lookup of a property "X" on the first object gave the result Y, we know that the same lookup on the second object will also give the result Y. We use this to cache the result of individual property lookups in ECMAScript programs, which greatly speeds up code that contains many property reads or writes.

Performance

So how fast is Carakan? Using a regular cross-platform switch dispatch mechanism (without any generated native code) Carakan is currently about two and a half times faster at the SunSpider benchmark than the ECMAScript engine in Presto 2.2 (Opera 10 Alpha). Since Opera is ported to many different hardware architectures, this cross-platform improvement is on its own very important.

The native code generation in Carakan is not yet ready for full-scale testing, but the few individual benchmark tests that it is already compatible with runs between 5 and 50 times faster, so it is looking promising so far.

161 comments

Vega - Opera's vector graphics library

By Tim Johansson. Wednesday, 4. February 2009, 09:39:41

svg, 3d, core, opera ...

In my previous post here at Core concerns I wrote a bit about hardware acceleration of Opera's vector graphics library. In this post I will go into some more details about our vector graphics library.

The history of Vega

Vega was created shortly after we started working on SVG support. When we added SVG support in Opera we needed a vector graphics library. We looked into what was available to use and met our requirements (fast, low memory usage and works on platforms ranging from phones to TVs and desktop computers). We did not find and good match for our needs, so we decided to write our own.

Shortly after we created Vega we added <canvas> support, which also uses Vega.

The most recent addition to Vega is the ability to use a hardware accelerated back-end. The back-ends we are using at the moment are OpenGL and Direct3D.

HTML rendering

In the core version we are currently developing, Presto 2.3, we have made it possible to use Vega for all rendering in Opera. This means that we can replace the platform specific code for rendering with Vega. In the future it might be mandatory to use Vega for rendering, but in Presto 2.3 it is still possible to use the old rendering back-ends.

There are three reasons for doing this. The first reason is that the new CSS3 background and borders standard is much easier to implement using a vector graphics library. Presto 2.3 adds partial support for CSS3 backgrounds and borders, but only when using Vega for rendering.

The second reason is to support hardware acceleration of our vector graphics. In order to be able to render SVG and <canvas> in hardware we must also be able to directly draw the rendered vector graphics to the screen since reading back the rendered image from the graphics card is usually slower than rendering in software.

The final reason is that it enables us to easily add advanced graphical effects to our UI and to web pages.

Hardware requirements

The hardware back-end of Vega will unfortunately not work on all graphics cards. The good news is that we detect at runtime if your graphics card is supported or not and fallback to the software back-end of Vega if the graphics card is not supported. This means that everything will work regardless of your graphics cards capabilities.

The first requirement of you graphics card is that it has fast stencil buffers. If you are on a desktop computer that will not be a problem, but on some mobile phones that will be a problem. The reason for this requirement is that we use the stencil buffer for rendering some complex shapes instead of tessellating them and render triangles.

Since some web standards (for example opacity, SVG and <canvas>) require us to render to an off-screen buffer which is composited onto the screen we also require some kind of render to texture to be able to use Vega hardware back-end. This means a DirectX 9 compatible graphics card, or support for the framebuffer object (FBO) extension in OpenGL. It would be possible to do the same thing with pbuffers in OpenGL, but we need to do many render target switches and with pbuffers that is too slow.

The final requirement is pixel shaders 2.0, or fragment shaders 2.0 as it is called in OpenGL. We also require GLSL support on OpenGL. The shaders are required for filters. Filters is a part of SVG which performs an operation such as blur or color transforms on an image. Filters are also used in <canvas> and to support text shadows and box shadows in HTML.

For a PC user these requirements are met by any DirectX 9 compatible graphics card. So with the correct drivers you should be able to use the hardware back-end of Vega if you have a DirectX 9 compatible graphics card.

Update 2009-02-05:

I was not very clear about the multiple back-ends of Vega. If the graphics card is not capable of running the hardware back-end of Vega the software back-end will be used as a fallback and all features will still work. No features depends on hardware acceleration. All of them, including CSS3 backgrounds and borders, will work in the software back-end of Vega which does not depend on hardware accelerated graphics.

36 comments

Presto in the the Opera User-Agent string

By Arve Bersvendsen. Thursday, 4. December 2008, 13:43:14

presto 2.2, presto, core, user-agent ...

As previously explained, Opera has a common foundation on every platform and release Opera ships on called the core, which consists of all the platform-independent bits and pieces that make the browser, such as the layout engine or ECMAScript engine.

The Opera Core, called Presto, is released internally at Opera at a schedule independent of any product releases, including Opera on desktop, Opera Mobile, Mini or Opera for devices. Normally, core makes three to four releases every year.

In the past, a developer could not directly determine which core version of Opera shipped with a particular release of Opera, but this changed with Opera 9.5 on desktop, where the User-Agent string changed.

In the past, a typical Opera User-Agent string would look like this:

Opera/8.54 (Windows NT 6.0; U; nl)

Starting with the previous release of Presto, the Opera core, we started identifying the version of Opera core in the User-Agent string:

Opera/9.62 (X11; Linux i686; U; en) Presto/2.1.1

The Presto components of the User-Agent string are on the format Presto/[Major version].[Minor version].[Maintenance version].

What to expect?

When developing against Opera, you can generally expect features to have the same level of support and issues, regardless of on which device. For instance, you can expect SVG, ECMAScript, CSS or DOM across all Opera releases shipping with a certain Presto version to behave the same.

This does not, however, mean that you should expect that all features from any given core release are available. Through a set of tweaks, a given shipment of an Opera product might disable or enable a certain feature. A good example of this is the web fonts feature for Presto 2.2.0 - it may be disabled for certain platforms, due to constraints on these platforms, such as low bandwith or low memory.

As for the maintenance versions: They are usually released a relatively short time after the minor version, and contain bug fixes that have surfaced since the release. In general, these versions will not affect any APIs or compatibility.

If you want to play with the latest and greatest Opera core, Opera 10 alpha 1 is shipping with the brand new Presto 2.2.0 release - an overview of the new features are to be found at dev.opera.com. A more detailed changelog can be found from our desktop team's announcment.