Skip navigation.

exploreopera

| Help

Sign up | Help

Web Applications Blog

Posts tagged with "Programming"

avatar

Search engine friendly multi-language support on widgets.opera.com

, , , ...

What, why and how

We have just released a new version of widgets.opera.com where we have reworked our multi-language solution to support search engines. Instead of storing the user selected language in a cookie, which hides all languages except the default from search engines and indexers, we now have unique URLs for every language which the site is translated into.

In this blog post we present both the important client-side effects, and our technical Python and Pylons-based solutions on the servers. So if you are a user of widgets.opera.com, a developer who wants a similar site structure, or just passing by - read on!

Read more...

avatar

widgets.opera.com 2.0 - the best place for you to share and find Opera Widgets

, , , ...

It's a been a while since we unleashed widgets.opera.com 2.0 on the world so we're long overdue a blog post from the devs regarding what we have done and where we intend to take things.

Read more...

avatar

Scalable Server Sent Events

, , , ...

Poking through my newsreader I realised that it's been a long time since the Web Applications Team actually shared any real code samples with you guys, and I think it's about high time we fixed that.

Back in September last year Arve Bersvendsen (virtuelvis) introduced you to Event Streaming to Web Browsers along with a few primitive code samples to get you started.

Some of you, quite rightly, pointed out that with a regular apache configuration the concept of keeping open many long lived requests doesn't hold. Also many of you requested that we provide more detailed examples of code working with this. I hope that today's blog post will go some way to addressing both issues.

The Theory

First for the boring part, a quick discussion of how to tackle the issue of
scalability.

In a normal web application when a server receives a HTTP request, an operating system thread is allocated from a pool and it is not released until the whole request is complete. When using SSE the request does not complete until the user closes the html document that has requested the event stream, this means that while a user is using your application the thread that is dealing with the SSE request can not be returned to the pool to serve requests for other users even if the SSE channel is currently idle.

The solution to this problem is to, while there is no data to be written to the outbound socket, release the thread so that it can process another client's request. Most languages have a socket APIs that can be exploited to implement this kind of behaviour however it is only recently that this technology is being exploited fully by web server developers in response to the AJAX wave of technologies.

The Solution

As far as hosting sites on apache is concerned I have not yet found any solution for doing this in a scalable manner in "traditional" languages such as PHP.

In python is is reasonably easy to hack together your own HTTP server using asyncore or socketserver however I don't feel that everyone should be writing their own HTTP 1.1 implementation just to achieve a scalable solution. This is where I think twisted.web has a good shot at being an elegant solution. The python twisted framework is an event based networking layer that allows reasonably easy access to the networking techniques we need and twisted.web is a HTTP layer built on top of that. In fact the existing SSE based web chat is a sort of compromise here, the twisted framework was used for low level connectivity but we did write our own, very primitive HTTP layer on top of that. One thing I'm interested in doing this year is having a go at this myself but using the full twisted.web stack rather than writing my own HTTP implementation.

In the Java world (where my heart lies) I'm ashamed to say that each vendor appears to have gone their own way regarding how to implement threadless waiting. Tomcat 6 features the newCometProcessor interface (why oh why did they choose a name that tied it to Comet:doh:) whereas Jetty have implemented continuations in the server. For a more complete assessment of the options in the Java world, see Greg's blog article on the subject (he also has GWT related code samples if you look around.

After looking at the above options I find that I am in general agreement with Greg, and have decided to proceed with the rest of this blog article to show
how to implement an SSE service in Java running on Jetty using their continuation support to achieve scalability.

The Code

The first part of the process was to model the Events themselves. Using the DOM Level 3 Events Specification and the Web Applications 1.0 Draft I came up with the following two classes and also decided to make them responsible for serialising themselves to a format that can be pushed out onto the output stream.
public class ServerSentEvent implements Serializable
{
    /** DOM 3 Event namespace */
    private static final String DOM3_EVENTS_NAMESPACE =
            "http://www.w3.org/2001/xml-events";    
    private String type;    
    private String target;
    private Boolean bubbles;    
    private Boolean cancelable;    
    private String namespace;
    // *snip * The usual getters and setters
    
    protected void generateEventContent(StringBuffer sb) {}
    
    public final String toString()
    {
        // Provide default event properties
        StringBuffer sb = new StringBuffer("");
        sb.append("Event: ").append(type).append("\n");
        sb.append("Class: ").append(this.getClass().getSimpleName()).append(
                "\n");
        if (target != null)
        {
            sb.append("Target: ").append(target).append("\n");
        }
        if (bubbles != null)
        {
            sb.append("Bubbles: ")
            .append(bubbles.booleanValue() ? "Yes" : "No").append("\n");
        }
        if (cancelable != null)
        {
            sb.append("Cancelable: ").append(
                    cancelable.booleanValue() ? "Yes" : "No").append("\n");
        }
        if (namespace != null)
        {
            sb.append("Namespace: ").append(namespace).append("\n");
        }
        
        // Provide method for subclasses to provide more event info
        this.generateEventContent(sb);
        
        // Terminate the event with a blank line
        sb.append("\n");
        return sb.toString();
    }
}

public class RemoteEvent extends ServerSentEvent
{
    /** Namespace for remote events */
    private static final String REMOTE_EVENT_NAMESPACE =
            "uuid:755e2d2d-a836-4539-83f4-16b51156341f";
    
    private List<String> data = new ArrayList<String>();
    
    public RemoteEvent(String type)
    {
        super(type, REMOTE_EVENT_NAMESPACE);
    }
    
    public void addData(String data)
    {
        this.data.add(data);
    }
    
    protected void generateEventContent(StringBuffer sb)
    {
        super.generateEventContent(sb);
        for (String currentData : data)
        {
            sb.append("data: ").append(currentData).append("\n");
        }
    }
}
There are also a couple of interfaces defined for listening to and generatingserver sent events in a similar manner to how such event listeners are handled in swing.

Next we need to implement a class to represent a client connection to the system, this is where the real meat lies.
public class SSEClient implements ServerSentEventListener
{
    public SSEClient(HttpServletRequest request, HttpServletResponse response)
    throws IOException
    {
        continuation = ContinuationSupport.getContinuation(request, this);
        this.response = response;
        this.request = request;
        
        writeHeaders(this.response);
    }
    
    private void writeHeaders(HttpServletResponse response) throws IOException
    {
        // Set the necessary headers
        response.setContentType("application/x-dom-event-stream");
        response.setCharacterEncoding("utf-8");
        response.setHeader("Cache-Control", "no-cache");
        response.setHeader("Transfer-Encoding", "Chunked");
        response.flushBuffer();
    }
    
    private void writeEvent(ServerSentEvent event) throws IOException
    {
        byte[] eventBytes = event.toString().getBytes();
        ServletOutputStream os = response.getOutputStream();
        os.println(Integer.toHexString(eventBytes.length));
        os.println(event.toString());
        os.println();
        os.flush();
        response.flushBuffer();
    }
    
    public void run()
    {
        try
        {
            // This outer while loop ensures correct operation when emulating
            // continuations i.e. not running in Jetty
            while (true)
            {
                synchronized (this)
                {
                    while (events.peek() != null)
                    {
                        log.debug("Found an event, writing it to the stream");
                        writeEvent(events.remove());
                    }
                }
                continuation.suspend(WAIT_TIME);
            }
        }
        catch (IOException e)
        {
            this.disconnect();
        }
    }
}
Finally we need a service class to manage the currently connected clients that can be called on by servlet code. This is so simple I'll let you poke at that in the download.

And that's it! My entire JavaSSE framework in ~400 lines of code.

Using this framework I have created a couple of very simple demo services that rely on no major libraries such as spring in order to keep them short and sweet. All of this is covered by a BSD style license.

They can be found in the zip file below along with a maven pom that will allow you to run the demo with the command
$ mvn jetty:run
JavaSSE.zip

The Results

The Hardware

In order to make this a sensible test I comandeered a couple of spare blade servers that I had left over from testing a previous project. Both have the following specification.

* 2 dual core 3.20GHz Xeon processors
* 3Gb RAM
* Debian Etch amd64 kernel

The Proceedure

In order to test out the scalability of this implementation I thought it would be sufficient to just throw apache bench at the SSE stream for the server time example in the package above. This gives very rough indications.

On the Jetty machine I placed a war file of the project in the jetty webapps
directory and fired up the server process with
$ java -XX:MaxPermSize=128m -Xmx3072M -jar start.jar
Then on the Apache bench machine I set up a bunch of requests with a command such as
$ ab -n 50000 -c 2000 -t 18000  http://jettyserver:8080/JavaSSE/ServerTime

The Results

Using apache bench I was able to step up the number of simultaneous connections to 4000 before I experienced any problems with the service. This is a huge improvement on the 200 you would get out of a standard servlet container on default configuration but still nothing stellar performance wise.

Note that these tests were done on a default configuration of Jetty and so by tweaking the config perhaps I can squeeze even more out of the container.

One final point, after running apache bench for about an hour the jetty process froze, my best explanation for this is that there is a memory leak somewhere in the application. I don't have enough time to investigate this now but I'm sure we could find a t-shirt for the first person that does.

Wrap Up

I'm hoping that I get to maintain my current workload and therefore can continue to post more technical articles. I have plans to make a beefed up version of the chat that was made before and perhaps I will get through this exercise with twisted.web. If anyone has any requests let me know in the comments.
avatar

Event Streaming to Web Browsers

, , , ...

One cool feature we added to Opera 9 is Server-Sent Events from the WHATWG Web Applications 1.0 specification. Using SSE you can push DOM events continously from your web server to the visitor's browser. This creates a lot of exciting opportunities for web application authors.

Traditionally, when building an Ajax application, the browser continually polls the server, sending requests to the server, asking to get data back, making new HTTP requests for every single poll, putting more strain on the server than needed.

The event streaming approach instead opens a persistent connection to the server, sending data to the client when new information is available, eliminating the need for continuous polling. This method for doing remoting offers a tremendous advantage, since the server no longer has to handle the overhead associated with clients asking for new data. Instead, the server simply sends back data every connected client when appropriate, thus reducing the load on the server, with the added advantage of offering instant feedback to the user.

Opera Web Chat

Opera Web Chat Screenshot To provide you with a starting point on how to build your own event streaming application, we have built Opera Web Chat. This is a web based chatroom offering some of the features from the built in to the Opera IRC client. Currently the chat only offers one single chatroom. (A screenshot is available here)

Keep in mind that it is an experimental service, which means it may not always be available for use.

How to use Server-Sent Events

To use Server-Sent Events in a web application, add an <event-source> element to the document, with a src attribute pointing to a event source URL. This URL should provide a persistent HTTP connection that sends a data stream containing the events. The connection must use the content type application/x-dom-event-stream.

It is possible to send events with any name, and specify the properties of the event object. Opera 9.01 only supports the data property of the event, so this is what we are going to use in these examples.

The server side event source writes the events whenever they occur, and sends them over HTTP to the client. This is a basic example of event data. This is more thoroughly explained in the specification.

Event: server-time
data: [time on the server]

Event: the-answer
data: 42

This will send two events to the browser, and it's possible to catch them as DOM events. The following JavaScript example listens for the "server-time" event, and alerts the content.

document.getElementsByTagName("event-source")[0]
        .addEventListener("server-time", eventHandler, false);

function eventHandler(event)
{
    // Alert time sent by the server
    alert(event.data); 
}

This is a very simple Python CGI example which sends a new event every 3 seconds. Every event is named "server-time", and sends an event with the data property set to the current time of the server in seconds.

Keep in mind that when a CGI script outputs data, there is no guarantee that it is sent immediately. There are often caching mechanisms and so on in place. For this reason it may be necessary to explicitly flush the output.

Here is the example code written in python.

#!/usr/bin/python
import sys
import time
print "Content-Type: application/x-dom-event-stream\n\n"

while True:
    print "Event: server-time"
    print "data: %f\n" % (time.time(),)
    sys.stdout.flush()
    time.sleep(3)

The same example written in PHP:

<?php
header("Content-Type: application/x-dom-event-stream");
while(true) {
    echo "Event: server-time\n";
    $time = time();
    echo "data: $time\n";
    echo "\n";
    flush();
    sleep(3);
}
?>

Opportunities

In addition to the chat application we made, there are lots of different applications that can be made with Server-Sent Events. For instance games or instant messaging clients, such as MSN Messenger, Jabber or AIM. You could also build stock and news tickers, status and log file monitors, or anyhing you can come up with.

What will you build?

avatar

Moving and resizing widgets

, , , ...

One of the last additions we made to widgets in the stages before releasing Opera 9.0 was giving widgets the ability to resize and move themselves.

The main use for resizing behavior is in those cases where you want to emulate UI widgets (yes, that other kind of widget) like maximize, restore, and regular resize and drag handlers.

interface Window {
  void resizeTo( in int w, in int h );
  void resizeBy( in int delta_w, in int delta_h );
  void moveTo( in int x, in int y );
  void moveBy( in int delta_x , in int delta_y );
};

window.resizeTo()

The window.resizeTo() method takes two positive integers as arguments: A width w and height h. These sizes are the desired dimensions of the widgets in pixels, and correspond to the width and height elements in a widget's configuration file. The method sets the window.outerWidth and window.outerHeight properties. Example:

// Resize the widget to 400x300 px
window.resizeTo( 400, 300 );

window.resizeBy()

The window.resizeBy method takes two integers as arguments: A delta_w that is the desired change in width, measured in pixels, and a delta_h that is the desired change in height, again measured in pixels. The width and height of the widget is then recalculated to be equivalent to the following pseudocode:

window.resizeTo( window.outerWidth + delta_x, window.outerHeight + delta_y );

In all cases of window resizing, the resizing happens from the lower-right corner of the widget window, while the upper-left corner remains static.

// Grow the widget by 100px horizontally
window.resizeBy( 100, 0 );

window.moveTo()

The window.moveTo method takes two integers as arguments: An x that is the desired x coordinate for the upper-left corner of the widget, and an y that is the desired y coordinate for the upper-left corner of the widget, with both values mentioned in pixels.

// Move the widget to the upper-left corner of the screen
window.moveTo( 0, 0 )

window.moveBy()

The window.resizeBy method takes two integers as arguments: A delta_x that is the desired change in x position, measured in pixels, and a delta_y that is the desired change in y position, again measured in pixels. Both values can be negative. Example:

// Move the 50px left and 50px down
window.moveBy( -50, 50 )

Closing note

If you change these settings for a widget, the settings are volatile, and the change will be lost if the widget is closed and then reopened. This means that if your widget relies on being a certain size, which may be different from the install-time default, you should make sure that you store the widget size in the persistent storage. The widget storage interface is compatible with Apple's interface, using the methods preferenceForKey and setPreferenceForKey.

avatar

The 'Opera-2dgame' Canvas Context

, , , ...

Since we added support for the canvas element in Opera 9 tp1, we have also supported a custom context on the canvas, named "opera-2dgame". As suggested by the name, this context has functionality specifically geared toward game authoring. Let's have a quick look at what we can do with it. However, please remember that this functionality is still experimental, and that the interfaces may change at any time before a final version of Opera is launched.

Using the 'opera-2dgame' context

To use the custom context, you simply call the 'getContext' method on the canvas, with the argument 'opera-2dgame', much the same as you would do if you were working with the regular canvas.:

// get the first canvas element on the page
var canvas = document.getElementsByTagName('canvas')[0];
// get the game context:
var gctx = canvas.getContext('opera-2dgame');

Pixel-by-pixel manipulation using getPixel() and setPixel()

From reading the canvas developers mailing list it seems that one of the more requested features is the ability to read and modify pixels on an individual basis. The opera-2dgame context has just that.

The getPixel() method

If you want to read a pixel value directly, this is achieved by using the getPixel method, which takes two integer arguments, representing the coordinate for which one wants to get the color value:

// Read the pixel at 0,0
var myPixel = gctx.getPixel(0,0);

The value returned by getPixel is a string, and is returned in one of two forms:

  1. If there is alpha information in the pixel, the value returned is on the form rgba( r, g, b, a )
  2. If the pixel is fully opaque, the value is on the form #rrggbb

getPixel() and security

For the security-concious out there, Opera enforces a same-origin policy on the canvas: Trying to call the getPixel method on a canvas element that contains data from a third-party server (that has foreign content) will raise a SECURITY_VIOLATION error. For what it's worth, the same restriction is enforced on the toDataURI method on the HTMLCanvas interface.

A canvas element is considered to have third-party content if drawImage has been used with an HTMLImageElement as an argument, and this image is from a third-party server , or if drawImage has used with a HTMLCanvasElement as argument, and this HTMLCanvasElement has previously been determined to contain third-party content.

setPixel()

While getPixel on its own has some meaning, the real power is revealed when we add the setPixel() method. The method takes three arguments: The x and y coordinates, represented as integers, and a string representing the color, represented as a CSS color.

// set the pixel at 32,24 to the value 'rgba(255,127,0,0.5)'
gctx.setPixel(32,24,'rgba(255,127,0,0.5)');

// set the pixel at 32,25 to the value '#f33'
gctx.setPixel(32,24,'#f33');

// set the pixel at 32,26 to the value 'red'
gctx.setPixel(32,24,'red');

Examples

Here are a few examples of using the getPixel and setPixel methods to perform image manipulation in much the same ways filters in popular graphics application works

Collision detection

Indeed. The opera-2dgame context does collision detection as well, and makes it really simple for you: The checkCollision method on the 2d-game context takes an x and y value as arguments, and when called, will check whether the given coordinate is within the currently open path on the '2d' canvas context. The checkCollision() method returns 1 if the point is within the open path, and 0 otherwise. The checkCollision method does not take any translatation on the regular '2d' context into consideration; the x and y values are relative to the top left corner of the canvas.

// draw a rectangle from 0,0 to 100,100
// ctx and gctx are previously created references to the
// 2d context and the opera-2dgame contexts on a canvas element.
ctx.beginPath();
ctx.fillStyle = "blue";
ctx.beginPath();  
ctx.lineTo(120,0);
ctx.lineTo(60,90);
ctx.lineTo(0,0);
ctx.fill();

// Output '1', since the point 60,25 is inside the open path
alert( gctx.checkCollision(60,25) ); 

// Output '0', since the point 150,150 is outside the open path
alert( gctx.checkCollision(150,150) ); 

Examples

We've prepared two small checkCollision() examples for you to look at:

  • Click the blue triangle - when clicking the blue triangle, you should be told whether you've clicked on the triangle or somewhere else.
  • Hit the ball - A little "How to get RSI ASAP"-game where the aim is to click the ball.

Optimizing canvas performance using locking

Finally, if you write high-performance canvas applications, such as games, you might want more control over the canvas redraw. The opera-2dgame context provides you with ultimate control using two methods:

lockCanvasUpdates( bool Lock) takes a boolean value as argument. When this function is invoked with the argument , updates on the 2d canvas context are locked. When a canvas is locked for updates, you can perform operations on it as you normally would do, but the updates will not update the display of the canvas. The advantage of this is that if you need to do multiple, possibly complex operations on the canvas, such as display a frame in a game with all heros, monsters, backgrounds and fluff, you won't waste any time rendering this to screen.

When you are ready to render your frame and update the canvas, you first call the lockCanvasUpdates(false) to allow updating the canvas. Next, you call the updateCanvas() function to force the canvas to repaint, and you relock the canvas again afterwards, using lockCanvasUpdates(true) as illustrated below

// We start by simply locking the canvas:
gctx.lockCanvasUpdates(true);

function renderFrame(){
  /* 
    First, lots of code rendering a frame
    in a complex scene 
  */
  // When we've rendered everything, we refresh the canvas:
  gctx.lockCanvasUpdates(false);
  gctx.updateCanvas();
  gctx.lockCanvasUpdates(true);
}

These additional methods should provide authors with the means neccessary to write graphics applications and high-performance games, in widgets or on web sites. Use your creativity.