Covert Tomato

Spending CPU cycles associating random words

Subscribe to RSS feed

Handling POST requests in Yusef

, , , ...

In its latest version, Yusef has introduced a new route API, extending the capabilities of the addSectionListener API. Basically, a route extends the concept of section/action. The route is matched by a regular expression (instead of a section name), and an action is attached to it. This new version also enforces security - a given for Opera - of the section/actions API. In particular, it was also possible to handle any POST in the sectionListener, without having a nonce. The nonce was however required for action handling. Sections were particularly useful when using AJAX components relying on POST to pass information back to the application: these component may not have a sufficient level of customization to pass that nonce. This worked all good in the previous version. However, the newer version of Yusef makes sure that every POST has a nonce attached. Otherwise, the application returns HTTP error 400, Bad Request. How can we deal with these AJAX components then? By adding "disablePOSTSecurityCheck: true" to the section listener options. AddSectionListener Options With all these options around, I thought it might be a good idea to get a table of every option. Although it is not currently the ultimate reference, I hope it will get more and more complete with the time:
Name Default value Plugin Description
disablePOSTSecurityCheck false Yusef core Whether POST are accepted without a nonce being present
ui false ui Use a template on the page (Note: for the content and _index sections, the default value is true).
acl {} acl Restrictions applied to the section/action.
acl.type "" acl Who can access the section ('private', 'public' or 'limited').
acl.level private acl Who can access the action (one of the levels of accesstypes defined in Yusef.plugins.acl.getAccessTypes()).
acl.strict "" acl Whether the access can be modified later on.
translation true translation Whether the content must be translated.

Yusef: new version

A new version of Yusef has been released and is available in the Hello Yusef package.

This brings a new version of markuper, with a new neat data attribute: data-unwrap. When a node has data-unwrap="true" (or for the matter, any expression that evaluates to true), the node is replaced by its content. This avoids many situations where a div/span/other element is used just for the sake1 of evaluating some keep-if, remove-if or list data-attribute, resulting in a loose semantic in the target document.

As far as I can see, this new version also brings fixes to the library loader and to Yusef. A neat change in Yusef is the addition of a PSO statement that automatically subscribes the current user to his debug statements. The ID for those messages is "user:your-user-name".

Other remarkable changes include the addition of many plugins to Yusef:
  • help: an integrated help system
  • comm: used for shoutbox... scarcely documented, unfortunately.
  • embed: to get representation of an url from a third party site
  • shoutbox: does what its name states


And indeed, it works with 10.50 beta smile

  1. Not the japanese beverage

Translation using plural forms

,

Using the translation plugin of Yusef, and the appropriate {%trans%} and {%endtrans%} tags along with some JSON, localizing an application is a breeze.1 The only problem is with pluralisation. So if you follow the documentation, you should get something like this
{%trans with
   "{{myData1}}" as counter and
   "{{myData2}}" as other   and
   "{{moreData}}" as etc %}
   There is {counter} foot
{%plural%}
   There are {counter} feet
{%endtrans%}

This won't work unless you fix the translation thing. The first problem lies within the underscore function: when variables are fetched, they are retrieved as string. However, when testing for plural, the following line does the wrong thing:
if (variables && variables.counter && variables.counter !== 1) 
{ /* plural stuffs */ } 
else 
{ /* singular */

Since the "!==" comparison is used, the test fails. Instead, we can rely on javascript conversion and use != 1

On a side note, there seem to be no support for localizing Date related strings. Even the so nice humanReadable addon of the datelibrary has no support for anything else than English. sad

  1. Except if many hardcoded stuffs are in your code.

Debugging Unite apps: take 2

, ,

One of the particularly interesting things with Unite is that you can simply drag and drop the config.xml file over Opera to install the application while debugging. In comparison with using a full .ua file, this already allows for changing application resources on the fly. For example, modifications in documents such as markuper templates or stylesheets will be directly available. However, application scripts will stay the same.

To change application (or libraries) files, the application must be reloaded. Although you can do it using the Start/Stop button in the Unite side-pane, a more efficient way is to use the refresh button of Dragonfly. The refresh button has the advantage that breakpoints won't be lost.

Creating dynamic images

, ,

Now that you have played a bit with Yusef, what's still missing is the possibility to create dynamic images for your very own 2000 web counter or captcha. Or is it missing? Recent web browsers support for the so called Canvas. The power of element, which appeared with HTML5, resides in dynamic images generation.

In this post, we will first take a look at Yusef's graphics plugin, which harnesses canvas to create images thumbnails. Then we will move on to create a custom web counter.

An Overview of Graphics

Graphics' main functionality is to make thumbnails out of larger pictures. In addition to this, thumbnails are cached and served in response to HTTP requests. Please note that in the following we use the more recent Graphics version bundled with Photo Sharing, as kindly suggested by p01.

Graphics API is offers two functions: serveResizedImage serves a thumbnail on a given connection and resizeImage creates thumbnails and caches them. The actual prototypes are:

serveResizedImage( connection, image, maxWidth, maxHeight, crop, quality, originalImageFile )
  • connection: The connection over which the thumbnail is to be served.
  • image: A File (from the FileIO library) pointing to the image to be resized. If originalImageFile is also specified, this File instead refers to a previously generated thumbnail.
  • maxWidth: Thumbnail width.
  • maxHeight: Thumbnail height.
  • crop: Whether the thumbnail should occupy the whole space (true) or if it should be sized down to fit in the rectangle (false).
  • quality: The quality from 0.0 to 1.0 for jpeg generation.
  • originalImageFile (optional): The full-sized picture File (from FileIO) if the image argument already refers to a thumbnail.


resizeImage( image, maxWidth, maxHeight, crop, quality, callback, originalImageFile )
  • callback: A function called when the thumbnail is generated. The function is called with a data argument, describing the generated thumbnail. This argument is hash contains the following values: {file: The thumbnail file, canvas: The canvas used to generated the thumbnail.}
  • Other arguments have the same meaning as those in serveResizedImage.

I will not describe the serveResizedImage further. Not only this function is way out of the scope of this post, but also the code speaks for itself.

Drawing on the canvas

The resizeImage function proceeds as follows to draw the thumbnail: first, it loads the image asynchronously; second, the thumbnail is rendered on a canvas at the appropriate size; third, the image is saved to a file; fourth, the callback is called.
To load the image asynchronously, an new Image object is created. All that is required is to set its onload handler to some function before setting the image source. Unfortunately, the execution does not stop and will end up with Yusef cleanup code disposing the connection. For this kind of asynchronous scenarios, Yusef can be told that the connection is still pending:
connection.pending = true; /* Yusef won't kill the connection */

Once the image is loaded, the image's onload handler is called. This handler retrieves the actual img element from the event.srcElement argument, and compute various height and width for the drawing. The image element is passed to the canvas for rendering:
var context     = canvas.getContext( '2d' );
context.drawImage( srcImage,0,0, srcImage.width, srcImage.height, (canvas.width-tw)/2,(canvas.height-th)/2,tw,th );

A bit of caching
The resizeImage function caches its results for faster access. For each set of parameters (i.e. file name, thumbnail size and quality) unique filename identifies the thumbnail.
When resizeImage is called, the function first checks if the full sized image is in cache and whether it is still bears the same modification time stamp as when the thumbnail was generated. If the original image satisfies these conditions, the previous thumbnail is returned. Otherwise, a new thumbnail is generated.
The role of the _updateCache in this is to bookmark the original file, along with its modification date, in the cache. After registering that new entry, _updateCache also removes older cache entries1

Your very own counter
In this section, we will make a visit counter as an image. We are aiming for readability, rather than optimization, or graceful error handling. We will therefore leave out problems, such as caching results or testing existence of image files. These should be considered in real world applications.
We start of the Hello Yusef application2. The first step is to create ten 12x16 pixels images, one per digit (zero included), and put them in under a new "resources" folder at the application root. We then make an index page, serving some dumb HTML, for our testing:
var counter = 0;

/**
* Generate html content to visualize the counter
*/
Yusef.addSectionListener('_index', function(connection)
{
    counter++;

    return '<html><head><title>Counter</title></head><body>' + 
           '[html:img src="' + 
           opera.io.webserver.currentServicePath + 'counter/?c=' + counter +
           '" alt="' + counter + '"]</body></html>';
}, {ui: false});


Now, we are ready to draw the counter image. For this, we write a function which takes the counter value and paints it over a given canvas:
var digitImages = [];

/* pre-load every image of a digit */
(function() {
    var digitfile;
    var digitsDirectory = Yusef.mountpoints.application.resolve('resources');

    for (var i = 0; i < 10; ++i) {
        digitImages[i]      = new Image();
        digitfile           = digitsDirectory.resolve(i + '.png');
        digitImages[i].src  = unescape(digitfile.path);
    }

    /* Images are loaded asynchronously. 
       Since the images are stored locally, we expect that loading will be done
       by the time the first request occurs. */
})();

/**
* Paint a counter of specified value onto the canvas.
*
* @param {Number} value The counter value
* @param {HTMLCanvasElement} canvas The canvas over which the graphical
* counter is to be shown. 
*/
function paintCounter(value, canvas) {
    var DIGIT_WIDTH     = 12,
        DIGIT_HEIGHT    = 16,
        digits          = [],
        canvas, canvasctx, 
        digitfile, digitimg;

    /* Get each digit in the value */
    while (value != 0)
    {
        digits.splice(0, 0, (value % 10));
        value = Math.floor(value / 10);
    }

    /* Set the canvas to an appropriate size */
    canvas.width    = DIGIT_WIDTH * digits.length;
    canvas.height   = DIGIT_HEIGHT;

    /* All painting operations are made on a "context"; first clear the
     * drawing area */
    canvasctx       = canvas.getContext('2d');
    canvasctx.fillStyle   = '#fff';
    canvasctx.fillRect( 0,0, canvas.width, canvas.height );

    /* For each digit, paint the corresponding image onto the canvas */
    for (var i = 0; i < digits.length; ++i) {
        canvasctx.drawImage(digitImages[digits[i]], i * DIGIT_WIDTH, 0);
    }
}

At this point, if you are unfamiliar with the canvas element, you may want to take a look at other canvas articles, to get an idea of the kind of pictures that can be generated using more involved code. All that's left is to create the canvas element, and make unite serve the image it contains:
/**
* Handle requests for the counter image
*/
Yusef.addSectionListener('counter', function(connection)
{
    var value = connection.request.getItem('c'), 
        canvas;

    /* Ensure that the counter value is present */
    if (!value || value.length === 0 || (1*value[0] === NaN)) {
        connection.response.closeAndRedispatch();
    }

    value = value[0] * 1;

    /* Create a canvas and paint a counter on it */
    canvas = document.createElement('canvas');
    paintCounter(value, canvas);

    /* We use the writeImage function to give the image back to the browser.
     * This function accepts a canvas element and serializes it as PNG */
    connection.response.setResponseHeader( "Content-type", "image/png" );
    connection.response.writeImage(canvas);

    /* Try to help the garbage collection */
    canvas = null;
});


That's all for our counter! The full code is available in attachment.

counter.js

  1. This really means entries that are in RAM for the file, not the actual thumbnails files on the hard drive for some reason.
  2. Generally speaking, the libraries used in Hello Yusef are some of the latest available. Those over dev.opera.com are too often outdated.

Happy New Year

Happy New Year, everybody! Don't drink too much and get home safely!

Are you the person I spoke with, last time?

, , , ...

An important feature of web application nowadays is the ability to keep track of the users. For example, the Fridge application determines whether a note belongs to a user to allow him/her to remove their own notes.

There are basically1 three mechanisms that support storing user properties: session variables, query strings2, and cookies. Cookies and query string are client-side3 while session variables are server-side4. In other words, users can fiddle with the formers, but not (directly) with the later.

The basic idea is that data stored on the user side cannot be trusted while those on the server can, to some extent. However, deciding whether the user is still the same as last time for the server is not failproof. It is also worth noting that with cookies, the user is in control. If the user want his setting cleared when closing the browser, he/she can do so.

Concretely, your application may allow the user to change the display theme. The current theme may not be sensitive enough to use server resources in a session variable. The cookie solution is preferable. On the other hand, your application may need to associate items to users, to allow each user to edit his/her own items (e.g. Fridge). Users will try cheating and edit other's items; in such a situation, session variables are better.

Using session variables
The Yusef application framework provides session variables through the getSessionVariable and setSessionVariable functions. Their basic usage is illustrated in the following:
/* in some section or action handler, exhibiting a connection variable */
Yusef.setSessionVariable(connection, "User's items", [1, 2, 5]);
Yusef.setSessionVariable(connection, "The number", 42);
...
var items = Yusef.getSessionVariable(connection, "User's items");
Note that setSessionVariable will copy the value passed before storing it.

Session variables can also be removed when not more needed. This is done through the deleteSessionVariable function:
Yusef.deleteSessionVariable(connection, "Session Variable Name");

Sessions in Yusef can also be tweaked to some extend. For this purpose, session configuration options are exposed in core.js. One setting of interest is _config.sessionTTL. A user inactive for more than this time will be logged out.
Cookies
In bare bones Unite, cookies are handled as other request headers: the WebServerResponse.setResponseHeader and WebServerRequest.headers functions are used for this purpose.

Yusef, aguments WebServerResponse and WebServerRequest with the getCookie and setCookie functions. The following shows their usage:
/* in some section or action handler, exhibiting a connection variable */
var css_file = connection.request.getCookie("theme");
...
connection.request.setCookie("theme", "blue-red-gray.css");
Note that while session variables can store whole javascript objects, (get|set)Cookie only deal with strings.


  1. Other mechanisms are also worth considering but won't be discussed here.
  2. Parameters in the url. For example, in url http://www.example.org/search?id=123&q=hello, the id=123&q=hello part are options. These are more or less transcient, and will not be discussed in this post.
  3. On the user machine
  4. Inside unite.

Security Included: Using the ACL Plugin

, , , ...

The UI plugin adds by default a password protection to the application. This feature stands in the sidebar section "Privacy options for [Application Name]". In this post, I explain the mechanism allowing for such security features, which is implemented by the acl plugin of Yusef.

Overview

Basically, the ACL plugin does nothing more than associating a clearance level with each URL of the application. The clearance level is technically named "access type", while the URL is referred to as a "Resource". Furthermore, the association between an URL and an access type is called an "Access Node".

The acl plugin offers the following access types:
  • accessTypes.public: Unlimited access to the resource.
  • accessTypes.limited: The resource is only available if you have the password.
  • accessTypes.private: The resource is restricted to the owner of the application.


Apart from associating URL to access types, an access node has also a strictness parameter. If an access node is strict, the access type of a resource cannot be changed. Ever.

Getting Started: a Page for the Almighty Owner

Defining a Section
Now that the main terms are defined, we will build a owner-only option page as a practical example. This page will actually be a section of the application. The following code builds such a page and restricts the access to the sole owner:
Yusef.addSectionListener
(
  'options',
  function(connection) { /* render the options page */ },
  {acl: {
    type: 'private', 
    strict: true
  }}
);

This way of adding options may remind one of how the ui plugin is used. Indeed the registration process is roughly the same, except for the recognized options.

I also introduced the strictness parameter earlier. For our option page, we only want the owner to change parameter and never should it be a user. Since we should never change this access type, it makes sense to set a strict access node.

Defining an Action
Of course, this option page will be used to set options. For this purpose, we will use a Yusef Action. Actions are only associated with an access type (level) and are not considered as resources. The following code sets an action for the options page:
Yusef.registerUniteAction
(
  'set-options',
  function (connection) { /* handle the [URL=http://www.cgisecurity.com/xss-faq.html#vendor]user input[/URL] here */ },
  { acl: { 
      level: Yusef.plugins.acl.getAccessTypes().private
  }}
);

We may further wonder how an action can be made available to public or to limited users depending on whether the corresponding section access. One possible way is to set the level to public and check in the action handler whether the section is accessible. For this we can use the hasAccess function of acl, which takes as arguments the connection, and optionally the section and the path within the section. The following code demonstrates this, allowing for only allowed users to save the options:
Yusef.registerUniteActionListener("set-options", 
  function(connection)
  {
     /* Do we have access to the "option" section ? */
     if (!Yusef.plugins.acl.hasAccess(connection))
     {
        /* bad guy. In the actual code, you may want to record this event in some log */
        return false;
     }
  
     /* Clearance check OK; proceed with actually saving the options */
     /* ... */
  
  }, { "acl": Yusef.plugins.acl.getAccessTypes().public });


Changing the Access Type

We defined the strictness of an access node earlier, but nothing was said about how to actually change the access type. For this purpose, Yusef.acl provides functions to read and write an access node: setAccessNode, getAccessNode and hasAccess. These respectively define the access node of a resource, retrieve the current access node of a resource, and determine whether the user has access to a resource. These functions are defined as follows:
setAccessNode = function( serviceWidePath, type, strict )

  • serviceWidePath: The path to the resource relative to the application root, including a leading '/'.
  • type: A string representing the access type ('private', 'public', 'limited).
  • strict: The strictness of the access node.
  • Returns true if the access node could be set, as requested or false otherwise.

getAccessNode = function( connection, section, path )

  • connection: The connection from which the resource path is to be retrieved.
  • section: (optional) The application section of the resource.
  • path: (optional) The path under the application section of the resource.
  • Returns the function returns the accessNode corresponding to the path.

hasAccess = function( connection, section, path )

  • Arguments have the same definition as for getAccessNode.
  • Returns true if the user has access to the resource


While most of the parameters are clear, the actual meaning of the path to the resource may be discussed. The most obvious definition is the url to an actual page of the application. However, some applications (e.g. like Fridge) use the index page for various purposes (e.g. displaying user posts and application settings). However, posting new messages, deleting messages, and setting the maximum number of messages can require very different access. In such case, "virtual" urls can be associated to features requiring different security tokens. For example, using access nodes over /security/post, /security/delete and /security/options would solve the problem.

Integration with the UI

In this last section, we get back to where we started this post, and study how the acl and ui plugins are integrated. The UI plugin has indeed some interaction with acl: when a new user navigates to a limited page, a message is shown requiring him/her to provide a password. Another integration, provided in the right side of the default template, allows the owner to change the access type of a page.

To provide ui with these features, acl registers the following actions:
  • Yusef_plugins_acl.unlock: checks that the password provided by the user is right. In that case the user goes from "public" privileges to "limited" ones.
  • plugins.acl.actions.change: allows the owner to set the access type for the current page

As with any action, the proper unite-action and unite-nonce fields must be set. The other fields needed for these features can be found in the main.html file of default ui template.

Debugging Yusef Applications

, , ,

Preemptive comment: this post inspiration comes from this forum thread.

Contrary to normal Unite applications, you may notice that whenever you try to debug a Yusef application using Dragonfly, you end up with the application crashing. On the javascript side, Opera complains about not being able to reshare (ALREADY_SHARED_ERR) some file, and the application will only accept to point you to some "static" section of the site, which won't work.

The problem, as stated by Antonio Afonso (thank you) is that Opera core won't clear some registered path when it reloads the unite application. However, Yusef magics involves sharing some resources under a static section right after running the application. To overcome this problem, I've hacked the Yusef core lib to allow debugging as described in the next section. However, some warnings first:

First warning: This fix works well for me. And it might work for you as well, although I definitely can't guarantee it. Use it at your own risks. If it kills your pets1, don't blame it on me!
Second warning: Newer versions of Yusef or Opera may have this fixed. Check them out!

The HackWorkaround

The problem lies with opera.io.webserver.sharePath: this function can't be called twice on a same path by design. Since Opera won't clear it's shared files on reload (hence, on Dragonfly debug), we need to explicitly do it.

Yusef offers the following wrapper around that sharePath function: function _shareDirectory( dir, exposedPath, rootPath ) in core.js. My hack is as follows (replacing the existing if block):
if( !(_config.sections.static+path in _staticFiles) )
{
  /* replace existing shares silently */
  opera.io.webserver.unsharePath( _config.sections.static+'/'+exposedPath+path, file );
  opera.io.webserver.sharePath( _config.sections.static+'/'+exposedPath+path, file );
}

Yusef fixed! Well... nearly: the ui plugin also uses sharePath to make application skins available. We can actually make it go through Yusef's wrapper, as suggested by the existing FIXME:
// FIXED: this should use Yusef.shareStaticDirectory when that function starts using sharePath instead of shareFile
Yusef.shareStaticDirectory(skin.resolve('public_html'), self.PLUGIN_PATH + '/' + skin.name + '/');


How do I debug Unite apps anyway?

First, make sure you know how to run Dragonfly; for starters, the debug menu is probably the easiest way to get it running. With the dragonfly window open, in the little dragonfly popup menu, select your unite app. It can be identified by its index2 page title, or if no title is specified by its url starting with "widget://".


From there, on the (left-side) Script tab, you can select which file you want to debug, add breakpoints, step in/over/out, just as in eclipse/visual studio. Great job, Opera, and many thanks to Chris Mills who made me aware of this JavaScript debugger (Me is clumsy! whistle)


  1. I did not actually test this code with every kind of pet to enforce safety.
  2. The actual index.html at the root of the app. Not the _index section handler.

Unite - Redirecting after form submission

, ,

A common requirement in web applications is to have forms that (1) can display errors to the user (e.g. missing values in required field) and (2) that can redirect the user to a "Thank you" kind of page when the user has finally got the input right. Handling the first case is not so complicated; using Yusef, your typical controller for such a form may very well look like :
/**
 *   Retrieve data from post
 */
function fetchPostData(connection)
{
  var request = connection.request;
  if (request.method != 'POST')
  {
    return null;
  }

  var data = {
    field1: fetchField(request, 'name') 
  };
  data.valid = data.field1.length > 0;
  return data;
}

Yusef.addSectionListener(myapp.constants.MY_SECTION, function(connection)
{
  /* Parse input arguments */
  var data = Yusef.getData(connection);
  data.formdata = fetchPostData(connection) || {
    field1 : 'default', /* default values */
    valid: true /* don't show "required" in red */
  };

  /* Create the form as output */

}, myapp.constants.STD_OPTIONS);

Yusef.registerUniteActionListener(myapp.constants.DO_THE_STUFF, function(connection)
{
  var data = fetchPostData(connection);

  if (data != null && data.valid)
  {
    /* do the actual stuffs */
  }
});

The problem is in the redirection. Different redirection methods exist, and each situation has its problems. However, for the "Thank you" page, different sources agree on the HTTP 303. 303 is a status code, just like the infamous 404 and the more used by far 200. This status code however tells the browser "Hey, you should GET that url, which is not the content you asked for, but is just what you need".

So how can we put a 303 using Unite? Here is a possible way to adapt the previous example with a 303 redirect:
Yusef.addSectionListener(myapp.constants.MY_SECTION, function(connection)
{
  /* Parse input arguments */
  var data = Yusef.getData(connection);
  data.formdata = fetchPostData(connection);

  if (data.formdata !== null)
  {
    if (data.formdata.valid)
    {
        /* redirect */
        connection.response.setStatusCode( '303', 'See Other' );
        /* to the root page of our app*/
        connection.response.setResponseHeader( 'Location', data.servicePath );
        return null;
    }
  }
  else
  {
    data.formdata = {
      field1 : 'default', /* default values */
      valid: true /* don't show "required" in red */
    };
  }

  /* Create the form as output */

}, myapp.constants.STD_OPTIONS);