miscoded

I've just tried to help the Web-API WG specify the "DOM0" key events. We badly need a spec here, one that documents the current implementations and finds a reasonable and web-compatible compromise where they differ.

Below is the whole text I sent to the public-webapi list. Comments are welcome both here and on public-webapi.

The text below refers to a table of virtual key codes. This table can be viewed here:
http://spreadsheets.google.com/pub?key=p32PJW4aXyuBDbcEUj4dmaw

INTRODUCTION
The specification of key events describes processing of input from keyboard devices. Key events rely on hardware support and DOM applications should not assume that any specific keys will be available, meaning they should be designed so that no information or command can be accessed only through generating a key event.

On many operating system, the system will handle character mappings but on devices where such mappings are not available an implementation may include its own conversion tables and for example decide to send ENTER key events with key code 13 if a joystick-type control in a mobile phone UI is pressed down.

Key input normally has a default action, e.g. inserting text. The default action of a key depends on what sort of element has focus, for example if focus is in a TEXTAREA the default action of the ENTER key is to insert a line break. If focus is on a link, the default action for ENTER usually is to follow the link. What the default action for a certain key is can not be determined from an event listener. The default action can be prevented by calling the event.preventDefault() method or returning false from an event handler, but implementations MAY make this configurable and let the user specify either globally or on a per-key basis that event listeners are not allowed to override the defaults.

INPUT METHOD EDITORS
Input method editors or IMEs are frequently used to type for example Chinese, Japanese and Korean characters. An active IME captures all keypresses and processes them to determine what character the user wants to insert.

Depending on the IME and its settings, most keys will either initiate, contribute to or finish the conversion process. For example, with the standard Microsoft Japanese IME active in Hiragana mode, pressing the "K" key will start a conversion process. Pressing "U" will contribute to it and the IME will show the Hiragana "ku" character. Pressing Enter will complete the conversion and insert the character into the document.

In this specification, the expression "input used for IME character creation" means input the IME is processing to transform into the actual output character(s). The expression "IME character insertion" means the insertion that occurs when the IME is done and the user confirms the generated characters to insert them into the text.

Note that IMEs can also insert text without any key events occurring at all, e.g. with handwriting recognition or on-screen keyboards.

KEY TYPES
Some key types defined by ranges of Microsoft Windows virtual key codes:

• Alphanumerical keys: all keys with key codes in the ranges 48-57 and 65-90.
• Function keys: keys with key code in range 112 - 135
• Dead keys: keys that are used to add accents to the next key that is typed. One example is an U+00A8 DIARESIS key. On many keyboard layouts, dead keys are in the OEM range and their placement and key codes vary by keyboard layout. It is not possible to specify virtual key codes for them and the implementation must query the system whether the key that was pressed was a dead key or not.
• Navigation keys: keys with key code in range 33 - 40 (arrows, Home/End, page navigation)

RELATIONSHIP BETWEEN EVENT TYPES

There are two types of key events:

• Hardware reference events. The keydown and keyup events report that a key was pressed down and released. These events include keyboard reference information but do not confirm what character(s) if any will be inserted. Keyboard reference codes are hardware-, software-, locale- and system-dependent, but implementations should map as many keys as possible to virtual key codes as defined below.
• Text insertion events. The keypress and textInput events include complete information about the character the input is generating, if any, taking into account shift states, previous dead keys and other contextual information required to decide what character will be inserted by a key press. The textInput event also shows what character(s) were generated by an active input method editor. The keypress event MUST NOT fire when an IME is processing the input for conversion.

All keys except dead keys fire the hardware reference events. What other events are fired depends on the type of key that is pressed and response from the event listeners. Typically, alphanumerical keys and punctuation keys fire text insertion events while control keys and navigation keys do not. Dead keys may fire text insertion events if pressed twice or followed by a space. The first time a dead key is pressed fires no events at all.

KEY CODE CALCULCATION FOR HARDWARE REFERENCE EVENTS
The key code for keydown / keyup events is calculated as follows:

1. If input is a numerical character (0-9), return the ASCII code of the character.
2. If input is a lower case character (not limited to English a-z), return the ASCII code of the upper case equivalent. [TODO#1]
3. Look up key code in table of Microsoft Windows virtual key codes, return it if found. [NOTE#2]
4. Read virtual key code from system if possible. [TODO#2]
5. If no key code was found, return 0.

PROCESSING OF KEYPRESS EVENTS

When a key is pressed, and the keydown event has been processed an implementation must behave as if it implemented this algorithm when firing keypress events:

1. If the keypress follows a keydown event on the same key and the earlier keydown event had its default action cancelled, terminate this algorithm. [NOTE#1]
2. If the input is key input used for IME character creation, terminate this algorithm. [TODO#3]
3. If the key does not cause text input and is not the Escape key (i.e. if the key is not is an alphanumerical key, a punctuation key, a repeated press of a dead key or the Escape key), terminate this algorithm.
4. Set event meta key properties (shiftKey, ctrlKey, altKey, metaKey) depending on what meta keys are pressed, if any
5. For backwards compatibility reasons the character code property has two different names. Define charCode and keyCode, set both to the decimal value of the unicode reference number of the corresponding character.
6. Fire the event. If it was not cancelled with event.preventDefault() proceed to fire a textInput event.
7. If the same keystroke inserts several characters, fire one keypress event for each
8. If the key is held down, repeatedly fire one keydown event and one or more keypress events at a rate determined by the system's repeat key interval setting

NOTE#1: follow Firefox or IE on what to do if keydown's preventDefault() was called? Test results:

• IE, Safari: fire keydown, keyup. No text insertion. (Above text standardises this)
• Firefox: fires keydown, keypress (does not insert text), keyup (and apparently web content relies on this)
• Opera: ignores preventDefault() on keydown, does not insert text if preventDefault() is called on keypress

NOTE#2: Microsoft virtual key codes attached as spreadsheet (Open Office format). Since step 1 and 2 of this algorithm handle alphanumerical keys, this step will mainly deal with punctuation characters. Their placements and virtual key codes vary greatly between keyboard layouts on Windows. The reason key codes for punctuation characters are so weird in a typical MSIE implementation are that they are usually placed on the keys Microsoft defines as "OEM keys". These are numbered OEM 1 - OEM 7 (with another OEM_102 on some layouts) plus OEM_COMMA, OEM_PERIOD, OEM_PLUS and OEM_MINUS.

However, different keyboard layouts change what OEM reference a certain key has! So the letter "ø" on a Norwegian keyboard is considered an OEM_3 key (code 192) but on EN-US layout the very same key is called OEM_1 and has code 186. Thus, even striking the *very same* key gives different virtual key codes on different keyboard layouts. Neither are those virtual key codes mapped to actual input: typing the *very same* character on different keyboard layouts can produce different key codes because these keys are associated with different "OEM" keys under the hood.

OEM_COMMA, OEM_PERIOD, OEM_PLUS, OEM_MINUS are obviously meant to map to the corresponding punctuation on most keyboards, but even this is not consistently applied. For example, on a Norwegian keyboard layout the + character is on the OEM_PLUS key and returns the virtual key code 187 but on the Icelandic layout + is on the OEM_2 key and returns its virtual key code 191.

There is no really consistent way to figure out how this should work for any OS and device. The assignment of keys to virtual key codes in Microsoft's OEM range is too chaotic to emulate, and not useful to authors. The way forward would be to specify explicit keyCode/charCode values for the following keys that never change with keyboard layouts:

• Backspace
• Tab
• Caps Lock
• Shift
• Control
• Alt
• Enter

And those punctuation characters that are *meant* to not change virtual codes between keyboard layouts:

• Period (.)
• Comma (,)
• Plus (+)
• Minus (-)
• Decimal key (on numpad)
• Space

Then leave other punctuation characters implementation dependent and inform script authors that they should listen to keypress or textInput events to reliably detect those.

TODO#1: note that IE does NOT take the upper-case value of certain non-English character (for example ø/Ø on Norwegian keyboards). I believe doing so makes the model cleaner and is unlikely to cause compatibility problems - this needs investigation though. Here we also probably need to specify some specific algorithm for upper/lower-casing characters?

TODO#2: Step 4 of this algorithm is incomplete, probably needs to specify how to get a virtual key from system? The issue step 4 is trying to solve is: If a given key, say the I key is mapped to something else, say "Hiragana I" keydown/keyup will still have the key code of an upper-case I in reference implementations but not according to this algorithm without some magic in step 4. Hence we need to fall back to reading virtual key codes from the system in step 4, but how to do this exactly is underspecified and will probably vary between operating systems.

TODO#3: If the IME is NOT in the middle of a conversion, keys that do not initiate a conversion in that IME (such as ESC and Enter) *do* fire keypress events. Is the text clear enough?

TODO: dead keys pressed twice fire two keypress events. Dead keys followed by space fire keydown space, keypress for the dead key's accent, keyup space (!). Dead keys are currently a bit underspecified in the above text.


USEFUL LINKS ON WINDOWS KEY HANDLING
http://www.microsoft.com/globaldev/handson/dev/inputinwin.mspx
http://www.woodmann.com/fravia/sources/WINUSER.H
http://tonesnotes.com/blog/2003/03/keycodes_defined_in_winuserh.html
http://www.microsoft.com/globaldev/tools/msklc.mspx

The HTML input element already defines a type attribute so it was a matter of adding new values, and my experiments looked at doing so for number and date. This worked fine on most browsers, but the Opera browsers handling of the type and required attributes made it necessary to support datatype and needed as alternatives.

(source)
Well, that's the sort of problem you get when re-inventing wheels: you have to figure out how to fit them in next to the wheels already in place. Meet the JavaScript car mechanic:

  // partial work around Opera's WF2 implementation for type
  // same technique doesn't work for required attribute :-(

Doesn't work for "required"? Seems like the JavaScript mechanic left his "removeAttribute" tool at home.

  if (nativewf2)
  {
    fields = document.getElementsByTagName("input");

    for (i = 0; i < fields.length; ++i)
    {
      field = fields[i];
      type = field.getAttribute("type");

      if (type == "number" || type == "date")
      {
        field.setAttribute("datatype", type);
        field.setAttribute("type", "text");

Look at those final setAttribute statements: since Opera's WF2 wheels apparently aren't good enough they are dismounted and the fancy JavaScript-required-XForms-Lite-wheels fitted instead. Why can't the mechanic simply say "OK, this car already has wheels so I don't need to do any work here"?

Are the differences between XForms Lite "date" and WF2 "date" so severe?

Why would one "required" be better than the other (besides from "required" being invented there while "required" isn't, that is..)?

One of the points of having open, independently developed standards from the W3C is of course to achieve compatibility.

However, standards are not always clear and consistent, and browsers do not always get things right.

When facing a choice between being compatible with other browsers and websites OR the standard, we may have a third way: fixing the standard to align it with the common implementations and the content. That's the only way one can have the cake and eat it, be standards-compatible and compatible with the Web as it is. Is it feasible? Or even a good idea? Standards are supposed to be set-in-stone after all, to keep implementors happy so that we don't spend hours and money doing it this way only to find a rewritten standard tells us to follow that way instead..

Three real-life examples of cases where we do break or are about to break the standard:

1) getAttribute on non-existing attributes
Standard mandated returning an empty string. Other browsers returned null. Sites running into problems due to Opera's correct standards support included Yahoo mail.

We reached a general consensus that returning null is a better idea, so we have changed Opera and asked the relevant W3C group to consider changing the text in an errata or the next version.

2) Throwing WRONG_DOCUMENT_ERR on cross-document node usage
If a node created in one document is appended to another document, the standard clearly states that a WRONG_DOCUMENT_ERR should be thrown. Firefox doesn't, and sites taking the Firefox behaviour for granted and breaking in Opera included Blogger.com's rich text editor.

Opera will start to call adoptNode implicitly. I'm not sure what way the standard will go - nobody seems to intend to push for a change here. According to their bug report, Mozilla intends to fix it (which is great!) and as soon as they do and websites fix their coding errors, we can revert to spec-compatible mode. Meanwhile, we are not going to live with sites being broken in Opera due to Firefox's bugs.

3) addEventListener and firing capturing events on target
Setting a capturing "click" event listener on, say, an IMG tag has no effect, because the standard clearly says that event listeners should not fire on target. Again, Firefox gets the implementation wrong and sites where this has caused trouble include live.com (according to the Safari team, I have not seen this myself but their analysis is probably right).

Sure, we should fix the standard here and go for it - I don't see any specific benefit to not firing the event listener though I'm all ears if anyone knows why the spec is written like that. We haven't changed this yet but I guess we will follow Safari.

The lesson? To standards bodies: please care about existing implementations, experience and content when developing the standards. Specs that are seriously incompatible with prior implementations or web content cause implementation headaches, incompatibilities and suffering users.