Covert Tomato

Spending CPU cycles associating random words

Subscribe to RSS feed

Posts tagged with "YUSEF"

Handling POST requests in Yusef

, , , ...

In its latest version, Yusef has introduced a new route API, extending the capabilities of the addSectionListener API. Basically, a route extends the concept of section/action. The route is matched by a regular expression (instead of a section name), and an action is attached to it. This new version also enforces security - a given for Opera - of the section/actions API. In particular, it was also possible to handle any POST in the sectionListener, without having a nonce. The nonce was however required for action handling. Sections were particularly useful when using AJAX components relying on POST to pass information back to the application: these component may not have a sufficient level of customization to pass that nonce. This worked all good in the previous version. However, the newer version of Yusef makes sure that every POST has a nonce attached. Otherwise, the application returns HTTP error 400, Bad Request. How can we deal with these AJAX components then? By adding "disablePOSTSecurityCheck: true" to the section listener options. AddSectionListener Options With all these options around, I thought it might be a good idea to get a table of every option. Although it is not currently the ultimate reference, I hope it will get more and more complete with the time:
Name Default value Plugin Description
disablePOSTSecurityCheck false Yusef core Whether POST are accepted without a nonce being present
ui false ui Use a template on the page (Note: for the content and _index sections, the default value is true).
acl {} acl Restrictions applied to the section/action.
acl.type "" acl Who can access the section ('private', 'public' or 'limited').
acl.level private acl Who can access the action (one of the levels of accesstypes defined in Yusef.plugins.acl.getAccessTypes()).
acl.strict "" acl Whether the access can be modified later on.
translation true translation Whether the content must be translated.

Translation using plural forms

,

Using the translation plugin of Yusef, and the appropriate {%trans%} and {%endtrans%} tags along with some JSON, localizing an application is a breeze.1 The only problem is with pluralisation. So if you follow the documentation, you should get something like this
{%trans with
   "{{myData1}}" as counter and
   "{{myData2}}" as other   and
   "{{moreData}}" as etc %}
   There is {counter} foot
{%plural%}
   There are {counter} feet
{%endtrans%}

This won't work unless you fix the translation thing. The first problem lies within the underscore function: when variables are fetched, they are retrieved as string. However, when testing for plural, the following line does the wrong thing:
if (variables && variables.counter && variables.counter !== 1) 
{ /* plural stuffs */ } 
else 
{ /* singular */

Since the "!==" comparison is used, the test fails. Instead, we can rely on javascript conversion and use != 1

On a side note, there seem to be no support for localizing Date related strings. Even the so nice humanReadable addon of the datelibrary has no support for anything else than English. sad

  1. Except if many hardcoded stuffs are in your code.

Are you the person I spoke with, last time?

, , , ...

An important feature of web application nowadays is the ability to keep track of the users. For example, the Fridge application determines whether a note belongs to a user to allow him/her to remove their own notes.

There are basically1 three mechanisms that support storing user properties: session variables, query strings2, and cookies. Cookies and query string are client-side3 while session variables are server-side4. In other words, users can fiddle with the formers, but not (directly) with the later.

The basic idea is that data stored on the user side cannot be trusted while those on the server can, to some extent. However, deciding whether the user is still the same as last time for the server is not failproof. It is also worth noting that with cookies, the user is in control. If the user want his setting cleared when closing the browser, he/she can do so.

Concretely, your application may allow the user to change the display theme. The current theme may not be sensitive enough to use server resources in a session variable. The cookie solution is preferable. On the other hand, your application may need to associate items to users, to allow each user to edit his/her own items (e.g. Fridge). Users will try cheating and edit other's items; in such a situation, session variables are better.

Using session variables
The Yusef application framework provides session variables through the getSessionVariable and setSessionVariable functions. Their basic usage is illustrated in the following:
/* in some section or action handler, exhibiting a connection variable */
Yusef.setSessionVariable(connection, "User's items", [1, 2, 5]);
Yusef.setSessionVariable(connection, "The number", 42);
...
var items = Yusef.getSessionVariable(connection, "User's items");
Note that setSessionVariable will copy the value passed before storing it.

Session variables can also be removed when not more needed. This is done through the deleteSessionVariable function:
Yusef.deleteSessionVariable(connection, "Session Variable Name");

Sessions in Yusef can also be tweaked to some extend. For this purpose, session configuration options are exposed in core.js. One setting of interest is _config.sessionTTL. A user inactive for more than this time will be logged out.
Cookies
In bare bones Unite, cookies are handled as other request headers: the WebServerResponse.setResponseHeader and WebServerRequest.headers functions are used for this purpose.

Yusef, aguments WebServerResponse and WebServerRequest with the getCookie and setCookie functions. The following shows their usage:
/* in some section or action handler, exhibiting a connection variable */
var css_file = connection.request.getCookie("theme");
...
connection.request.setCookie("theme", "blue-red-gray.css");
Note that while session variables can store whole javascript objects, (get|set)Cookie only deal with strings.


  1. Other mechanisms are also worth considering but won't be discussed here.
  2. Parameters in the url. For example, in url http://www.example.org/search?id=123&q=hello, the id=123&q=hello part are options. These are more or less transcient, and will not be discussed in this post.
  3. On the user machine
  4. Inside unite.

Security Included: Using the ACL Plugin

, , , ...

The UI plugin adds by default a password protection to the application. This feature stands in the sidebar section "Privacy options for [Application Name]". In this post, I explain the mechanism allowing for such security features, which is implemented by the acl plugin of Yusef.

Overview

Basically, the ACL plugin does nothing more than associating a clearance level with each URL of the application. The clearance level is technically named "access type", while the URL is referred to as a "Resource". Furthermore, the association between an URL and an access type is called an "Access Node".

The acl plugin offers the following access types:
  • accessTypes.public: Unlimited access to the resource.
  • accessTypes.limited: The resource is only available if you have the password.
  • accessTypes.private: The resource is restricted to the owner of the application.


Apart from associating URL to access types, an access node has also a strictness parameter. If an access node is strict, the access type of a resource cannot be changed. Ever.

Getting Started: a Page for the Almighty Owner

Defining a Section
Now that the main terms are defined, we will build a owner-only option page as a practical example. This page will actually be a section of the application. The following code builds such a page and restricts the access to the sole owner:
Yusef.addSectionListener
(
  'options',
  function(connection) { /* render the options page */ },
  {acl: {
    type: 'private', 
    strict: true
  }}
);

This way of adding options may remind one of how the ui plugin is used. Indeed the registration process is roughly the same, except for the recognized options.

I also introduced the strictness parameter earlier. For our option page, we only want the owner to change parameter and never should it be a user. Since we should never change this access type, it makes sense to set a strict access node.

Defining an Action
Of course, this option page will be used to set options. For this purpose, we will use a Yusef Action. Actions are only associated with an access type (level) and are not considered as resources. The following code sets an action for the options page:
Yusef.registerUniteAction
(
  'set-options',
  function (connection) { /* handle the [URL=http://www.cgisecurity.com/xss-faq.html#vendor]user input[/URL] here */ },
  { acl: { 
      level: Yusef.plugins.acl.getAccessTypes().private
  }}
);

We may further wonder how an action can be made available to public or to limited users depending on whether the corresponding section access. One possible way is to set the level to public and check in the action handler whether the section is accessible. For this we can use the hasAccess function of acl, which takes as arguments the connection, and optionally the section and the path within the section. The following code demonstrates this, allowing for only allowed users to save the options:
Yusef.registerUniteActionListener("set-options", 
  function(connection)
  {
     /* Do we have access to the "option" section ? */
     if (!Yusef.plugins.acl.hasAccess(connection))
     {
        /* bad guy. In the actual code, you may want to record this event in some log */
        return false;
     }
  
     /* Clearance check OK; proceed with actually saving the options */
     /* ... */
  
  }, { "acl": Yusef.plugins.acl.getAccessTypes().public });


Changing the Access Type

We defined the strictness of an access node earlier, but nothing was said about how to actually change the access type. For this purpose, Yusef.acl provides functions to read and write an access node: setAccessNode, getAccessNode and hasAccess. These respectively define the access node of a resource, retrieve the current access node of a resource, and determine whether the user has access to a resource. These functions are defined as follows:
setAccessNode = function( serviceWidePath, type, strict )

  • serviceWidePath: The path to the resource relative to the application root, including a leading '/'.
  • type: A string representing the access type ('private', 'public', 'limited).
  • strict: The strictness of the access node.
  • Returns true if the access node could be set, as requested or false otherwise.

getAccessNode = function( connection, section, path )

  • connection: The connection from which the resource path is to be retrieved.
  • section: (optional) The application section of the resource.
  • path: (optional) The path under the application section of the resource.
  • Returns the function returns the accessNode corresponding to the path.

hasAccess = function( connection, section, path )

  • Arguments have the same definition as for getAccessNode.
  • Returns true if the user has access to the resource


While most of the parameters are clear, the actual meaning of the path to the resource may be discussed. The most obvious definition is the url to an actual page of the application. However, some applications (e.g. like Fridge) use the index page for various purposes (e.g. displaying user posts and application settings). However, posting new messages, deleting messages, and setting the maximum number of messages can require very different access. In such case, "virtual" urls can be associated to features requiring different security tokens. For example, using access nodes over /security/post, /security/delete and /security/options would solve the problem.

Integration with the UI

In this last section, we get back to where we started this post, and study how the acl and ui plugins are integrated. The UI plugin has indeed some interaction with acl: when a new user navigates to a limited page, a message is shown requiring him/her to provide a password. Another integration, provided in the right side of the default template, allows the owner to change the access type of a page.

To provide ui with these features, acl registers the following actions:
  • Yusef_plugins_acl.unlock: checks that the password provided by the user is right. In that case the user goes from "public" privileges to "limited" ones.
  • plugins.acl.actions.change: allows the owner to set the access type for the current page

As with any action, the proper unite-action and unite-nonce fields must be set. The other fields needed for these features can be found in the main.html file of default ui template.

Debugging Yusef Applications

, , ,

Preemptive comment: this post inspiration comes from this forum thread.

Contrary to normal Unite applications, you may notice that whenever you try to debug a Yusef application using Dragonfly, you end up with the application crashing. On the javascript side, Opera complains about not being able to reshare (ALREADY_SHARED_ERR) some file, and the application will only accept to point you to some "static" section of the site, which won't work.

The problem, as stated by Antonio Afonso (thank you) is that Opera core won't clear some registered path when it reloads the unite application. However, Yusef magics involves sharing some resources under a static section right after running the application. To overcome this problem, I've hacked the Yusef core lib to allow debugging as described in the next section. However, some warnings first:

First warning: This fix works well for me. And it might work for you as well, although I definitely can't guarantee it. Use it at your own risks. If it kills your pets1, don't blame it on me!
Second warning: Newer versions of Yusef or Opera may have this fixed. Check them out!

The HackWorkaround

The problem lies with opera.io.webserver.sharePath: this function can't be called twice on a same path by design. Since Opera won't clear it's shared files on reload (hence, on Dragonfly debug), we need to explicitly do it.

Yusef offers the following wrapper around that sharePath function: function _shareDirectory( dir, exposedPath, rootPath ) in core.js. My hack is as follows (replacing the existing if block):
if( !(_config.sections.static+path in _staticFiles) )
{
  /* replace existing shares silently */
  opera.io.webserver.unsharePath( _config.sections.static+'/'+exposedPath+path, file );
  opera.io.webserver.sharePath( _config.sections.static+'/'+exposedPath+path, file );
}

Yusef fixed! Well... nearly: the ui plugin also uses sharePath to make application skins available. We can actually make it go through Yusef's wrapper, as suggested by the existing FIXME:
// FIXED: this should use Yusef.shareStaticDirectory when that function starts using sharePath instead of shareFile
Yusef.shareStaticDirectory(skin.resolve('public_html'), self.PLUGIN_PATH + '/' + skin.name + '/');


How do I debug Unite apps anyway?

First, make sure you know how to run Dragonfly; for starters, the debug menu is probably the easiest way to get it running. With the dragonfly window open, in the little dragonfly popup menu, select your unite app. It can be identified by its index2 page title, or if no title is specified by its url starting with "widget://".


From there, on the (left-side) Script tab, you can select which file you want to debug, add breakpoints, step in/over/out, just as in eclipse/visual studio. Great job, Opera, and many thanks to Chris Mills who made me aware of this JavaScript debugger (Me is clumsy! whistle)


  1. I did not actually test this code with every kind of pet to enforce safety.
  2. The actual index.html at the root of the app. Not the _index section handler.

Hacking the core: ordering your application components

, , ,

Yusef is a nice toolkit, but there are times where it simply won't cut it. One of the things I dislike about yusef is its promise to load every script in the serverScripts directory. Sure, I want each of my script of my app loaded. But I also need some parts loaded before some others (e.g. some utility functions before fetching data off the disk).

Because yusef is all javascript, hacking one's way through it is fairly easy. So let's force yusef to load your application component using the total order imposed by the file name.

Finding yusef's serverScripts loading procedure

One question I asked myself when I first started using yusef was "How does it know that my application is in serverScripts?". It turns out that it knows the directory because of some magic constant set in its configuration:
_config.serverScriptsPath       = '/serverScripts/';

Searching for this constant in the core will show the _loadServerScripts procedure up easily. This is indeed the one procedure we want to hack.

Ordering the files

The _loadServerScripts procedure fetches the script files in the serverScripts {File} variable. This variable is then iterated to load each script in turn.

From this point, ordering the files is a breeze. For my use, simply ordering them by name is enough, but smart coders out there may prefer to order them using some hyper sophisticated comparator. So here's the hack:
serverScripts.refresh();
        
serverScripts.sort(); /* Added: sort scripts by name */
        
for( var i=0,currentFile; currentFile=serverScripts[i]; i++ )

All that's left is to properly name files. For this task, I tend to use a two digits code which goes as follows:
  • 0x: utils and or models
  • 1x: controllers or views
  • 2x: Sorry, i'm very binary, i can't count up to 2

With x chosen to resolve other dependencies that may occurs, or to add some layering semantic.

Extending the system

Sometimes, extending your application with our hack is not enough. Some functionalities that are used over and over, are better fit for a library. It turns out that yusef provides a libraries loader that can be used for that purpose. On a side note, this libraries loader takes care of dependencies between libraries; therefore the naming hack won't be necessary here.

One library yusef does not provide you with is uuid generation. Adding such a library can be done as follows:
  1. Get some nice uuid script or write one yourself.
  2. Create a uuid subdirectory of the libraries folder and put your uuid script. We suppose in the following that the script is named uuid.js.
  3. Add a new file in your uuid subdirectory named setup.xml containing the following:
    <setup id="uuid">
        <script>uuid.js</script>
    </setup>

Setup.xml files describes how your library is to be loaded. It takes the following format:
<setup id="your-library-id">
    <script>script_1.js</script>
    ...
    <script>script_n.js</script>
    <depend>library_1</depend>
    ...
    <depend>library_n</depend>
</setup>

When parsing this file, the libraries loader will first resolve all libraries specified in depend. Then all script files will be loaded.

Extending yusef is cool!

yusef .ui and filter for markuper

, , , ...

The yusef toolkit example app comes with the ui plugin. This plugin is a big wrapper around markuper, intercepting calls to yusef's request handler, to apply a template to any page on the fly. In this post, I discuss some basic usage of ui, the inner working of its hooking into yusef, and markuper filters.

Using UI

Using ui is demonstrated in the yusef example app: there is nothing to do, except for exporting the content inside the content property of a hash. However, when trying to extend this behavior to other sections of an application, it simply won't work. This is because by default, ui applies its templates only to the 'content' and _index sections (no, it's not a bug, it's a feature!). To enable it on any page, one must add the 'ui' option in his/her listener, as follows:
Yusef.addSectionListener("mysection", 
                         function(connection) { /* your own listener here */ }, 
                         {ui: true});

An easy way to give users access to different sections of your application is to create tabs. The ui plugin also provides this through the ui.generateTabs function. The parameters are an array of hashes, each describing one tab, and the current connection. Tabs are described by the following parameters:
  • title (required): The title of the tabs; A lower-cased version of the title is used the section if none is explicitly specified.
  • section: The specific section to which the tab redirects
  • root: Overrides the section with the application root page.
  • hideFromVisitors: If the tab should only be visible to the administrator.

Once generated through the generateTabs functions, tabs must be exported by the listener as the tabs proerty of an hash, as follows1:
{
    /* your listener code here */ 
    var mycontent = document.createElement('div');
    div.innerHTML = "Hello ui!";

    var mytabs = Yusef.plugins.ui.generateTabs([
                { title: opera.io.webserver.currentServiceName, isRoot: true },
                { title: "My very own tab", hideFromVisitors: true, section: 'the_section/' }
                ], connection);

    return {
        tabs: mytabs,
        styles: [a list of your stylesheets, as stored in the public_html directory],
        script: [a list of your user-side scripts, as stored in the public_html directory],
        content: mycontent
    }
}


UI hooking

For those interested in the inner working of ui, and most notably how it intercepts the page handling, here is how it works. Yusef offers an extend method, basically allowing to override any of yusef core function. This extend function requires as arguments the name of a function to be overridden and a new, overriding, function. To this new function, yusef.extend adds a "proceed" method which allows the new function to call the overriden one. This allows for a call chain from the extending function to the (possibly many) extended function, towards the original yusef core method.

In particular, ui extends the Yusef.addSectionListener. This allows ui to intercept creation of sections, and in particular to catch and modify the listener used to render the page. Actually, ui replace the listener with a wrapper around the original listener. This wrapper will conveniently add the template. When yusef gets a request for the section, it calls the new wrapping listener which in turn calls the original listener to get the content, builds a new page with the template filled with this content, and finally sends the result back to yusef. This effectively wraps the content into a template.

This same technique is used in the acl plugin to prevent access to some content under defined conditions, as well as in the translator plugin to translate strings embedded in the page. Obviously, acl does not only get a hook on the sections, but also on the actions registration.

And now for something completely different: markuper filters
Data generated by a controller for a page may need some processing before being okay for the viewer. For example, escaping HTML or capitalizing the fist letter of a string are such common processing.

One way to deal with this problem is directly in javascript, for example using yusef HTMLEntities to escape strings:
escapedHTML = someHTMLstring.HTMLEntities();

Another way is to apply markuper filters. For example, to escape someHTMLstring, one can use the following built-in filter2:
This is some {{someHTMLstring|escape}}

But is it possible to add one's own filters? The answer lies in markuper.evaluator.registerFilter. The following example shows how to capitalize the first letter of a string:
markuper.evaluator.registerFilter('capitalizeFirstLetter', function(data)
{
    if( typeof data != 'string' ) { return input; };

    return data[0].toUpperCase() + data.slice(1);
});

Be warned though that filters are registered globally (i.e. for all templates). Maybe some form of template-local filter will be available in the future?

By the way, filters can be chained. The following example shows someHTMLstring escaped and capitalizeFirstLetter-ed:
This is some {{someHTMLstring|escape|capitalizeFirstLetter}}

Enjoy!

  1. The section listener will not listen to a section without a trailing /. The culprit the following line of yusef/core.js: connection.request.section = tmp.length<2?'':tmp.shift().toLowerCase();. Removing tmp.length<2?'': obviously resolves the issue, but this test is probably there for some good reason confused .
  2. Markuper seems to HTML-escape every {{string} anyway, so this example may not be the best.

PSO, Yusef and Unite

, , ,

I spent the last few days building a new application for Unite. So I read all the nice articles, downloaded the full hello world , and found myself stuck with more questions than answers. Maybe I missed some documentation, but for sure the libraries in the example app offer a helluva lot features. I plan to go over some of these libs in the next few post; hopefully, some official documentation/articles will get available soon. Generally speaking, while this may seem a bit rough for a first post, I plan to use this blog as a way of writing memos about technical stuffs; I'm not too sure about blogging, but since everyone else (and their dog) do so...

Anyway, let me get started by pointing out the PSO lib, used for debugging purposes. This lib acts as a wrapper around opera.postError, which gets error messages straight to the error console. PSO messages are required to have an ID, which is a string identifying the message origin or category.

One can subscribe to messages of interest by specifying IDs:
PSO.sub(id1, id2, ...)

where id1, id2 are the ids. This effectively filters messages out, when no subscription has been made.

Debug messages are published using:
PSO.pub(id, msg)

where id is the message ID, and message is the actual debug information.

Subscribing to the special 'new pub' id, one can subscribe to get a list of new message ids. Other predefined ids in use throughout Yusef can be found at the beginning of yusef/core.js, right under switch( opera.io.webserver.userName ). Some of these are useful when debugging/adding new features to Yusef, while others can tell you important information about why your code fail. Did you register your section properly? Subscribe to 'Yusef.addSectionListener' to know. Which section is requested when you click that link? Subscribe to 'yusef request handle'.

I personally found it useful to express ids as a hierachy (e.g 'fpicalausa.myapp.feature42'), in order to keep code clean component-wise. Going along with this convention, I also modified PSO (see attachment at the end), so that it allows me to subscribe to messages higher in the hierarchy (e.g. 'fpicalaua.myapp' to get all messages in myapp).

Enjoy!

pso.js