Web Applications

widgets.opera.com 2.0 - the best place for you to share and find Opera Widgets

By Jonathan Share. Thursday, 7. February 2008, 10:24:32

widgets.opera.com, python, Web Applications, Widgets, Programming

It's a been a while since we unleashed widgets.opera.com 2.0 on the world so we're long overdue a blog post from the devs regarding what we have done and where we intend to take things.

Why change?

With implementations of Opera Widgets for devices and mobile phones developing fast, we knew we needed to evolve the site in order to keep up with new requirements as well as improving the experience for both users and widget developers. Below I will explain what we've achieved so far, and what is still yet to come.

Progress so far

The code under the hood has been completely rewritten, plus the following improvements have been made, to make widgets.opera.com better for the users and developers of the Opera Widgets community.

Improved style and layout: The new site looks great in all major desktop browsers and the Internet Channel on the Nintendo Wii, we're also working on mobile specific optimisations at the moment.
Improved upload process: We now do basic sanity checks on all files that are uploaded.
Better versioning of widgets: This allows users to moderate widgets more securely, ensuring that end users only get quality widgets that are known to work.
Bug tracking system: If you find a bug in a widget on the widgets.opera.com site you can now file a bug for the author rather than just mentioning it in a comment, where it may get lost. When logged in to widgets.opera.com as a widget author you get a quick summary of all the bugs that are registered on your widgets.

The stats

For those more interested in the technical details here are a couple of quick snippets of information. I hope to blog about some of the programming topics in more detail over coming weeks.

Hardware: 2 blade servers both with two 2GHz Intel Xeons and 4Gb RAM
Workload: approximately 3,500,000 total requests per day (including media)
Language: Python
Application Framework: Pylons
Templating Language: Genshi
Database: MySQL

What's coming next?

Now that we have the new code base in a kind of stable state, what are we working on? From now on, you can expect to see us start on some of the more advanced features aimed at making widgets.opera.com a more tailored experience for the user.

Feedback

If you have any feedback regarding features or bugs with the site then please raise it on the widgets forum. We do check up there regularly and try to provide a response whenever we can.

Who can make the best travel widget?Search engine friendly multi-language support on widgets.opera.com

Comments

shadowk # 7. February 2008, 10:31

i always wondered what MyOpera was using as a backend language, and now i guess it's clear..it's Python or atleast it's clear for the widgets.opera.com site.

I still wonder though why you didn't go with PHP.O well..

Sharebear # 7. February 2008, 12:21

Actually, most of my.opera is currently Perl based. This was also the case for the previous version of widgets.opera.com. You can find some more information from the main my.opera team over at the Community Dev Blog

Why not PHP? That's a good question, it is used on some of the biggest sites in the world (Yahoo!, Flickr), so could also have been a good candidate. The short answer is probably part personal preference, part experience. Perhaps I can expand on this in a future blog post. I am a little hesitant though, I don't want to kick off a language war, all languages have their pros and cons, someone will always want it done a different way.

shadowk # 7. February 2008, 13:06

I didn't want to start a language war. Each language has it's powers and weaknesses, but it's good to know what's behind my opera and widgets2.0

Thanks for clearing it up.

asmodai # 7. February 2008, 17:44

Hey Jonathan,

as one of the people helping out on the Genshi project I am curious about your setup (obviously).

Are you guys using 0.4.4 or 0.5-dev?
Are you guys using anything like memcached for additional caching or just raw Genshi?

We would also be happy if you feel inclined to add yourself to http://genshi.edgewall.org/wiki/GenshiUsers

Thanks!

--Jeroen

Sharebear # 7. February 2008, 18:13

Jeroen,

Genshi 0.4.4

Yes we use memcache to cache full pages for logged out users, and some internal api calls.

I'll add us to the users page tomorrow (getting to the end of the work day here in Oslo) however, we may phase it out due to scaling concerns. In informal click testing with the profiler we saw that Genshi was taking the majority of the load time for a page.

If this is the case then switching to something faster would significantly affect the load on the servers. That, coupled with FormEncode's htmlfill and Genshi not playing very well together, is leading me think we should consider moving away.

Having said that, there are some areas in the templates that I think could be a cause of some of the slowness and should be refactored. If/when we prioritise looking at the templating I will try and get some proper, reproducable load tests done and publish the results if I can.

NoteMe # 7. February 2008, 18:51

Why are there never any job openings requiring Python if you do in fact us it

I guess everyone wants them

- ØØ -

Sharebear # 7. February 2008, 19:12

NoteMe,

Maybe you missed this one

NoteMe # 7. February 2008, 19:37

That slash must have been added since last time I checked. I thought it was all perl. I was so sure I checked once and saw that all the opera servers used mod_perl, but apparently I must be wrong, because it does not show up in the headers anymore:


[barqs]$ curl -I http://widgets.opera.com
HTTP/1.1 200 OK
Date: Thu, 07 Feb 2008 19:31:52 GMT
Server: Apache/2.2.3 (Debian) mod_ssl/2.2.3 OpenSSL/0.9.8c
X-Opera-Swingline: Opera Widgets: No banners!
Content-Length: 23703
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8

Thanks for pointing that out, Sharebear.
- ØØ -

asmodai # 7. February 2008, 19:58

Hi Jonathan,

some additional speedups were introduced in the current trunk, they might be interesting for you guys (and perhaps not).

Alec Thomas is working on additional performance optimizations in a separate branch at http://genshi.edgewall.org/browser/branches/experimental/optimizer

According to Christopher Lenz using htmlfill with Genshi is most likely an issue and he noted that Genshi features a better way to do so using genshi.filters.html.HTMLFormFiller (see http://genshi.edgewall.org/wiki/Documentation/filters.html#html-form-filler ).

Needless to say we are very happy you guys use Genshi and this kind of feedback is valuable for us (so if you have more, please feel free to let us know: drop in on Freenode's #python-genshi or on the mailinglists).

I hope this helps you guys a bit.

--Jeroen

Sharebear # 7. February 2008, 20:34

Jeroen,

Thanks for the pointer to the speed optimisations, We'll definitley include them if/when we get around to giving the templating some love (plenty of work to do before then).

I did look at HTMLFormFiller but, off the top of my head, I seem to remember that it would mean we couldn't use the FormEncode validator framework at all, which wasn't something I wanted to change at the time I realised Genshi stripped out the <form:error> tags before passing the result on to htmlfill.