miscoded

I've just tried to help the Web-API WG specify the "DOM0" key events. We badly need a spec here, one that documents the current implementations and finds a reasonable and web-compatible compromise where they differ.

Below is the whole text I sent to the public-webapi list. Comments are welcome both here and on public-webapi.

The text below refers to a table of virtual key codes. This table can be viewed here:
http://spreadsheets.google.com/pub?key=p32PJW4aXyuBDbcEUj4dmaw

INTRODUCTION
The specification of key events describes processing of input from keyboard devices. Key events rely on hardware support and DOM applications should not assume that any specific keys will be available, meaning they should be designed so that no information or command can be accessed only through generating a key event.

On many operating system, the system will handle character mappings but on devices where such mappings are not available an implementation may include its own conversion tables and for example decide to send ENTER key events with key code 13 if a joystick-type control in a mobile phone UI is pressed down.

Key input normally has a default action, e.g. inserting text. The default action of a key depends on what sort of element has focus, for example if focus is in a TEXTAREA the default action of the ENTER key is to insert a line break. If focus is on a link, the default action for ENTER usually is to follow the link. What the default action for a certain key is can not be determined from an event listener. The default action can be prevented by calling the event.preventDefault() method or returning false from an event handler, but implementations MAY make this configurable and let the user specify either globally or on a per-key basis that event listeners are not allowed to override the defaults.

INPUT METHOD EDITORS
Input method editors or IMEs are frequently used to type for example Chinese, Japanese and Korean characters. An active IME captures all keypresses and processes them to determine what character the user wants to insert.

Depending on the IME and its settings, most keys will either initiate, contribute to or finish the conversion process. For example, with the standard Microsoft Japanese IME active in Hiragana mode, pressing the "K" key will start a conversion process. Pressing "U" will contribute to it and the IME will show the Hiragana "ku" character. Pressing Enter will complete the conversion and insert the character into the document.

In this specification, the expression "input used for IME character creation" means input the IME is processing to transform into the actual output character(s). The expression "IME character insertion" means the insertion that occurs when the IME is done and the user confirms the generated characters to insert them into the text.

Note that IMEs can also insert text without any key events occurring at all, e.g. with handwriting recognition or on-screen keyboards.

KEY TYPES
Some key types defined by ranges of Microsoft Windows virtual key codes:

• Alphanumerical keys: all keys with key codes in the ranges 48-57 and 65-90.
  
• Function keys: keys with key code in range 112 - 135
  
• Dead keys: keys that are used to add accents to the next key that is typed. One example is an U+00A8 DIARESIS key. On many keyboard layouts, dead keys are in the OEM range and their placement and key codes vary by keyboard layout. It is not possible to specify virtual key codes for them and the implementation must query the system whether the key that was pressed was a dead key or not.
  
• Navigation keys: keys with key code in range 33 - 40 (arrows, Home/End, page navigation)
  

RELATIONSHIP BETWEEN EVENT TYPES

There are two types of key events:

• Hardware reference events. The keydown and keyup events report that a key was pressed down and released. These events include keyboard reference information but do not confirm what character(s) if any will be inserted. Keyboard reference codes are hardware-, software-, locale- and system-dependent, but implementations should map as many keys as possible to virtual key codes as defined below.
  
• Text insertion events. The keypress and textInput events include complete information about the character the input is generating, if any, taking into account shift states, previous dead keys and other contextual information required to decide what character will be inserted by a key press. The textInput event also shows what character(s) were generated by an active input method editor. The keypress event MUST NOT fire when an IME is processing the input for conversion.
  

All keys except dead keys fire the hardware reference events. What other events are fired depends on the type of key that is pressed and response from the event listeners. Typically, alphanumerical keys and punctuation keys fire text insertion events while control keys and navigation keys do not. Dead keys may fire text insertion events if pressed twice or followed by a space. The first time a dead key is pressed fires no events at all.

KEY CODE CALCULCATION FOR HARDWARE REFERENCE EVENTS
The key code for keydown / keyup events is calculated as follows:

1. If input is a numerical character (0-9), return the ASCII code of the character.
   
2. If input is a lower case character (not limited to English a-z), return the ASCII code of the upper case equivalent. [TODO#1]
   
3. Look up key code in table of Microsoft Windows virtual key codes, return it if found. [NOTE#2]
   
4. Read virtual key code from system if possible. [TODO#2]
   
5. If no key code was found, return 0.
   

PROCESSING OF KEYPRESS EVENTS

When a key is pressed, and the keydown event has been processed an implementation must behave as if it implemented this algorithm when firing keypress events:

1. If the keypress follows a keydown event on the same key and the earlier keydown event had its default action cancelled, terminate this algorithm. [NOTE#1]
   
2. If the input is key input used for IME character creation, terminate this algorithm. [TODO#3]
   
3. If the key does not cause text input and is not the Escape key (i.e. if the key is not is an alphanumerical key, a punctuation key, a repeated press of a dead key or the Escape key), terminate this algorithm.
   
4. Set event meta key properties (shiftKey, ctrlKey, altKey, metaKey) depending on what meta keys are pressed, if any
   
5. For backwards compatibility reasons the character code property has two different names. Define charCode and keyCode, set both to the decimal value of the unicode reference number of the corresponding character.
   
6. Fire the event. If it was not cancelled with event.preventDefault() proceed to fire a textInput event.
   
7. If the same keystroke inserts several characters, fire one keypress event for each
   
8. If the key is held down, repeatedly fire one keydown event and one or more keypress events at a rate determined by the system's repeat key interval setting
   

NOTE#1: follow Firefox or IE on what to do if keydown's preventDefault() was called? Test results:

• IE, Safari: fire keydown, keyup. No text insertion. (Above text standardises this)
  
• Firefox: fires keydown, keypress (does not insert text), keyup (and apparently web content relies on this)
  
• Opera: ignores preventDefault() on keydown, does not insert text if preventDefault() is called on keypress
  

NOTE#2: Microsoft virtual key codes attached as spreadsheet (Open Office format). Since step 1 and 2 of this algorithm handle alphanumerical keys, this step will mainly deal with punctuation characters. Their placements and virtual key codes vary greatly between keyboard layouts on Windows. The reason key codes for punctuation characters are so weird in a typical MSIE implementation are that they are usually placed on the keys Microsoft defines as "OEM keys". These are numbered OEM 1 - OEM 7 (with another OEM_102 on some layouts) plus OEM_COMMA, OEM_PERIOD, OEM_PLUS and OEM_MINUS.

However, different keyboard layouts change what OEM reference a certain key has! So the letter "ø" on a Norwegian keyboard is considered an OEM_3 key (code 192) but on EN-US layout the very same key is called OEM_1 and has code 186. Thus, even striking the *very same* key gives different virtual key codes on different keyboard layouts. Neither are those virtual key codes mapped to actual input: typing the *very same* character on different keyboard layouts can produce different key codes because these keys are associated with different "OEM" keys under the hood.

OEM_COMMA, OEM_PERIOD, OEM_PLUS, OEM_MINUS are obviously meant to map to the corresponding punctuation on most keyboards, but even this is not consistently applied. For example, on a Norwegian keyboard layout the + character is on the OEM_PLUS key and returns the virtual key code 187 but on the Icelandic layout + is on the OEM_2 key and returns its virtual key code 191.

There is no really consistent way to figure out how this should work for any OS and device. The assignment of keys to virtual key codes in Microsoft's OEM range is too chaotic to emulate, and not useful to authors. The way forward would be to specify explicit keyCode/charCode values for the following keys that never change with keyboard layouts:

• Backspace
  
• Tab
  
• Caps Lock
  
• Shift
  
• Control
  
• Alt
  
• Enter
  

And those punctuation characters that are *meant* to not change virtual codes between keyboard layouts:

• Period (.)
  
• Comma (,)
  
• Plus (+)
  
• Minus (-)
  
• Decimal key (on numpad)
  
• Space
  

Then leave other punctuation characters implementation dependent and inform script authors that they should listen to keypress or textInput events to reliably detect those.

TODO#1: note that IE does NOT take the upper-case value of certain non-English character (for example ø/Ø on Norwegian keyboards). I believe doing so makes the model cleaner and is unlikely to cause compatibility problems - this needs investigation though. Here we also probably need to specify some specific algorithm for upper/lower-casing characters?

TODO#2: Step 4 of this algorithm is incomplete, probably needs to specify how to get a virtual key from system? The issue step 4 is trying to solve is: If a given key, say the I key is mapped to something else, say "Hiragana I" keydown/keyup will still have the key code of an upper-case I in reference implementations but not according to this algorithm without some magic in step 4. Hence we need to fall back to reading virtual key codes from the system in step 4, but how to do this exactly is underspecified and will probably vary between operating systems.

TODO#3: If the IME is NOT in the middle of a conversion, keys that do not initiate a conversion in that IME (such as ESC and Enter) *do* fire keypress events. Is the text clear enough?

TODO: dead keys pressed twice fire two keypress events. Dead keys followed by space fire keydown space, keypress for the dead key's accent, keyup space (!). Dead keys are currently a bit underspecified in the above text.


USEFUL LINKS ON WINDOWS KEY HANDLING
http://www.microsoft.com/globaldev/handson/dev/inputinwin.mspx
http://www.woodmann.com/fravia/sources/WINUSER.H
http://tonesnotes.com/blog/2003/03/keycodes_defined_in_winuserh.html
http://www.microsoft.com/globaldev/tools/msklc.mspx

Let's keep this short and to the point: I have to leave Opera QA because of increasing allergy to IFRAMEs. My doctor says it's a new condition that is under-diagnosed in the web developer community, but a growing problem.

We all know that IFRAMEs have infected the web, at ever increasing speed since a certain announcement four years ago, and today I can hardly find a single bug to work on that won't give me allergic reactions.

So, goodbye. I plan to become a natural scientist to study spiders, just because I'm still sort of attached to that "Web" thing. Au revoir.

There is a new and very Spolskyesque article on browsers, web standards and IE8 just out from the one and only Joel Spolsky.

Some selected quotes.. On the IE8 team being stuck between a rock and a hard place:

"The IE team has to walk a fine line..." This is incorrect. It’s not a fine line. It’s a line of negative width. There is no place to walk. They are damned if they do and damned if they don’t.

On why "standards" generally aren't:

despite general agreement on what constitutes kosher food, .. you will not find a rabbi from one ultra-orthodox community who is willing to eat at the home of a rabbi from a different ultra-orthodox community. And the web designers are discovering what the Jews of Mea Shearim have known for decades: just because you all agree to follow one book doesn’t ensure compatibility, because the laws are so complex and complicated and convoluted that it’s almost impossible to understand them all

standards are sometimes misinterpreted, sometimes confusing and even ambiguous (...) it’s a platonic ideal and a set of misinterpretations

It's all so true. The web is badly under-specified (and quite likely anything as large and complex as the web and its underlying technologies would be, wouldn't it?).

However, while Joel has been busy selling MP3 players on Mars and watching Microsoft there are a couple of developments he's missed: maybe, maybe Microsoft has moved from what he calls a "pragmatist" position to an "idealist" position - but at the same time, the good old consortium has sort of moved in the opposite direction, from the merry idealism of un-implementable X-es to the grassroots pragmatism of HTML5.

Today the standard ITSELF is in beta, and planning to be in beta for a while among other things in order to figure out if it is compatible with the Web content. Standards Trotskyists are not cool anymore.

Joel asserts that we're dealing with "some mythical, platonic 'standard' that is not actually implemented anywhere". Actually, any modern standard has a reference implementation, it's called a test suite. (Note that I'm using circular logic here: if a standard doesn't I won't call it "modern" ) HTML5 will even have reference parsers. It's not quite the mind-reading validator which checks that your CSS rules are according to the spec and do what you had in mind. A site's compliance to the standard can't be tested beyond basic structural validation - but the user agent's compliance can! And the IE team understands this situation very well, as evident from the fact that one of the spin-offs from their CSS 2.1 implementation is a public test suite!

For all those websites Joel found that were broken in IE8, some of them may be broken due to bugs in the new implementation, some because the IE team fixed their CSS parser before implementing the specific issue a parser-based hack worked around, some because of the site violating standards generally and the rest - the whole lot of them - because of user agent sniffing. Every site and every detail needs analysis. We don't know yet to what extent the improved standards support causes these failures. Microsoft will have to figure that out over time and see on each count what there is to do about it. Even if it's down to the nasty quirks of client-side browser detection it may be possible to add workarounds. For example, if it turns out that most CSS served within IE conditional comments is now harmful, IE8 could stop parsing STYLE and LINK rel=stylesheet tags inside conditional comments unless the comment explicitly targets IE8.

Finally, Joel describes a world where the players don't cooperate. Today browsers are from Venus, we've spent more than 10 years exploring each others' quirks and collaborate on making compromises. Compromises that can be disappointing but useful. Joel is excused for not noticing, since we already know he's from Mars.