Input handling for TV interfaces/applications

Hello

I've been developing IPTV portal interfaces for some time now. My company is buying new set-top-boxes and they will come with a Opera browser.

My job is to programme the GUI-side of the portal solution we have, in HTML5 and all that good stuff.

Now, what bothers me is the input handling. I have two options to do it:

1) Just let the browser have focus and allow user to navigate whatever html content there is, this is the default way. Pros: Very simple to add new pages. Cons: Very hard to implement specialized UI like tv zapping environment.

2) Handle remote controller input events using a custom handler function (onkeypress), and react appropriately. Pros: You have full control of UI and remote control interaction. Cons: This requires heavy JavaScript programming to manage the GUI.


I've tried contemplating trying to do some sort of hybrid, that allows you to do 1) in some UI's like Video-On-Demand catalog but 2) in other scenarios, like live tv zapping. But I've not found any good way. I don't want to give the user too much control, but I also don't want to put too much effort into making a huge JavaScript based application for the GUI.

Doing 1) feels right, but also feels wrong because I have little control to do the UI properly. The TV+remote-controller cannot be used as a PC+mouse to navigate pages. There is no mouse to click on links, buttons, or whatever you have. There's only PROGRAM_UP, PROGRAM_DOWN, arrow buttons, OK, etc.


So, what does Opera think about this? Any suggestions? How does Opera do this?

Maybe CSS directional focus navigation will help you: http://www.w3.org/TR/css3-ui/#nav-dir

Originally posted by lbendig:

Maybe CSS directional focus navigation will help you: http://www.w3.org/TR/css3-ui/#nav-dir

Thank you, that's interesting. However, it is somewhat limited as it seems to only deal with arrow navigation via keyboard/remote controller, and requires all elements to be specified in the CSS, which doesn't work for dynamically populated elements.

Remote controller has more complex interaction with the screen, the CH+, CH-, info button, mute, volume up/down, menu button, number buttons (0-9) to specify directly the channel you want to zap to, blue/red/green/yellow buttons, etc. etc.

It would seem CSS/XHTML/HTML5 isn't mature enough to handle TV screen interaction, although it seems to somewhat support it for visual styles.


I've discussed this with my team and it would appear that we are required to grap all inputs, translate to something like "OK", "MENU", "PROGRAM_UP", etc. and give it to the view/screen that is currently active. That could be the tv zapping environment or the video-on-demand catalog. Those views/screens would receive those events, and have a corresponding action mapped as a function, which would change some model, and refresh the view.

This pretty much requires immense javascript work, which I feel like burdens any development and limits what we can do.

It is very surprising that Opera hasn't presented any solutions to this, because of it's widespread deployment on TV platforms, set-top-boxes, TV's, etc. Not saying this is Operas problem to solve, but it surely is within their domain to help solve since it is very beneficial to their customers.

Thanks.

Hi,

Indeed, navigation on TV is now problematic. I'll try to 'complete' the list:

1) CSS navigation
I don't recommend this for reasons ArniArent pointed out: it's limited and dull.
"doesn't work for dynamically populated elements" - not exactly true as you can always do: "el.style.nav*" (but who would!)

2) Built in navigation
"allow user to navigate whatever html content there is" - it's not exactly "whatever" but only "focusable" and "clickable" elements (e.g. form elements, links with 'href' attr, elements with 'tabindex' attr, elements with 'onclick' handler, etc). This is actually quite reasonable. And you get it for free. However you have no control over the 'navigation graph'

3) document.moveFocus* (Left, Right, Up, Down)
e.g. success = document.moveFocusRight() // true if the focus was moved
These are js methods implemented in Opera browser that invoke focus movement equivalent to the built-in nav (they use the same 'navigation graph'). However it's now easy to handle "special cases" (and modify the default movement).

For example:
SPECIAL_CASE = false
some_el.onfocus = function () {SPECIAL_CASE = true}
some_el.onblur = function () {SPECIAL_CASE = false}

document.onkeyup = function (evt) {
switch (evt.keyCode) {
...
case opera.tv.VK_UP:
if (SPECIAL_CASE) {/*special thing*/}
else {document.moveFocusUp()}
break;
case ...
}
evt.preventDefault() // don't use built-in stuff
}

This is probably the easiest method and Opera apps may take a big advantage of it.

4) js
JS gives you full control. If you're not satisfied with document.moveFocus* you will have to capture key events. However, I'd suggest you put it into some kind of a reusable library. Some old Opera TV apps do so (but the library is in dev state and not currently maintained)

Handling onkeypress() is the only option if you also want to use the other buttons (e.g. "MENU"). Opera shall provide JS api for TV, that includes properties for key codes (eg. opera.tv.VK_MENU, opera.tv.VK_LEFT, etc.). To use the api you need to include it as an external js file. Be sure to use the api CONSTANTS because actual keyCodes may vary across devices:
if (evt.keyCode == 100) { // wrong - may fail on some devices
if (evt.keyCode == opera.tv.VK_BLUE) { // ok

Thank you iYondaime.

1 and 2 are not viable for our UI's, and we're doing 4 currently which although we do have full control is does require a lot of coding and framework designing which has it's own problems.

3 is interesting, and does seem like a mix of all, I will take a further look at it and test it.

I've been testing 3, and it may help.

However, is there a way to stop the Opera browser from natively scrolling the window after doing document.moveFocus*? If I select a element that is outside of the viewable screen Opera automatically scrolls so it is in view. I don't want that.

I wish to do my own scrolling, e.g. smooth scrolling and by a certain amount of pixels. I have a mosaic of posters that is browsable horizontally, and Opera's scrolling behavior is bad.

Any ideas?

Hm...

I noticed that if you do "overflow: hidden" on an "inner" element (I mean: not on body), moving the focus will not scroll the content.

I tested in on a Desktop/Linux and don't know if it's "standardized" behavior. And I have no knowledge of "official" way to prevent scroll

In the simple cases when you know exactly which are the last-visible you can overwrite keypress handler to first do the scrolling and then call the focus (guess, there's no need to say that ;] )

I wonder, if it's good for a TV app not to fit the screen... Long [side-]lists (menus/categories) may require scrolling, but for the whole app i'd suggest it fits the screen. But I'm no UX guy, may be wrong

Originally posted by iYondaime:

Hm...

I noticed that if you do "overflow: hidden" on an "inner" element (I mean: not on body), moving the focus will not scroll the content.

I tested in on a Desktop/Linux and don't know if it's "standardized" behavior. And I have no knowledge of "official" way to prevent scroll

In the simple cases when you know exactly which are the last-visible you can overwrite keypress handler to first do the scrolling and then call the focus (guess, there's no need to say that ;] )

I wonder, if it's good for a TV app not to fit the screen... Long [side-]lists (menus/categories) may require scrolling, but for the whole app i'd suggest it fits the screen. But I'm no UX guy, may be wrong

Good points, thanks

some problem with flash and more at bank site

Not all devices ship with an Opera browser that have those key code constants defined. I'm working with two types of set-top-boxes from two different vendors, and the codes are different and not defined in API. Hence I have to define those constants myself. Mapping them is easy, just print out whatever code that occurs when pressing the remote control button. But this is no good when you're making an application for an unknown STB. But that's not the most daunting thing, each STB maker have their own API to do playback and other things, and they don't even follow the same RTSP specification.

IPTV is so very immature still, even after working in this since 2006 there's no common way to do things, and simple things don't work seamlessly between different STB vendors and video servers. Just look at a simple functionality like "resume". Each stb provider defines his own way of doing resume, each of them follows their rtsp standard, then the video server maker follows their standard. And nothing works seamlessly together.

I gotta give Opera some credit. GUI's work seamlessly between any type of STB's that have the same Opera browser, given the STB API and mappings are done correctly.



But I have a feeling that browsers aren't well suited to provide for a graphical TV UI experience. Sure, it's easy to get developers to make applications if they're web based, but TV is so different from an desktop environment.

Input handling is my worst enemy. Just consider this, desktop web pages offer the user hundreds, sometimes thousands of options, clickable links, images, everything clickable with a mouse etc. TV UI's have to be dumbed down because everything has to be big and easily selectable with simple infra-red arrow key remote control, so these UIs can only present handful of options to the user. YET, doing the dumbed down UI is many times more difficult to implement than the more complex desktop web page.

Let's take another issue, a simple one, like just pressing the 'OK' button on the remote control to verify that the user has chosen something, like a movie poster, clicked a button, or even a hyperlink. Now, CSS specifies a "a:active" which allows you to provide for a different look for clicked on elements. In Opera on TV devices this css selector isn't used, so whenever you want to provide for instant feedback on a button click, you have to manage that 'OK' press, change the style decoration by hand, and then after a short timeout (to allow for Opera to render the UI change) make the GUI activate the action that the user wants.

Then there's the simple thing of preloading images, so you don't have to make e.g. movie posters pop-up one by one when the user goes to the next page in your movie catalog. You have little control over the memory in the browser. Preloading something doesn't mean it will be rendered quickly. And if you preload one image, it may be cleaned out of the cache because of some FIFO order. But that may be a very important UI image, so the user keeps seeing flashing graphics of the browser redrawing everything because it needed to reload that important image into memory.

And doing any sort of animations is a killer for 99% of the devices out there.

As I said, web pages/browsers for TV are...questionable. Native rendering will provide you with a better experience, more optimized UI rendering rather than doing DOM manipulation and relying on ultra slow redraw/reflow.

If Opera browser has a future on TV devices they'll have to provide for a different kind of browser than for desktop pc's, a browser that solves these things, and standardizes things.

Hi,
I have a problem about autofocus in <button>. Please help me how to solve this issue.
When i press key RETURN will popup a dialog "Quit this application?" YES NO. I want to autofocus button "NO"