By Cosimo Streppone.
Thursday, 29. October 2009, 21:11:58
scalability, lighttpd, httpd, static
...
It's been a while since we deployed the first
static server for My Opera.
It was a really necessary step, because we had, and we continue to have, lots of static resources to be served.
A while back, most of these resources were served by applications, causing much more load on servers than necessary.
Now most of the heaviest ones have been already moved to the static servers.
Some months ago we also added support for
partitioning of the user resources in our storage software layer. That means making sure that if we have the need to scale serving of resources on different machines, we can either replicate the entire content on different ones, or split the content over 2 or more machines. That part worked nicely so far. That's why at some point there was a
static.myopera.com and
static02.myopera.com.
Now there's also
static03.myopera.com 
We changed our setup again, transparently (for you users), to consolidate the previously recycled/temporary hardware into a new shiny machine with more disk space. On this machine, we installed
lighttpd instead of our usual Apache setup. We wanted to try out this software. For us it was the first time we tried it on a production setup.
Edoardo played with it for a while, prepared the setup and installed it on
static03. As of today, it has been running perfectly for nearly 1 month with a really low load, and peaks of 250 accesses per second. It's serving around 14M hits per day for avatars, user pictures, skin thumbnails, etc...
By Esteban Manchado Velázquez.
Tuesday, 20. October 2009, 08:51:03
dragonfruit, changelog, RELEASE
Yesterday we released a new version of My Opera, codenamed "Dragonfruit".
Each of the important new features in the release will be explained in detail in the community blog. This is the detailed changeset for Dragonfruit:
- Add new "phototagging" feature to the photo albums. See the post in the community blog for more details.
- Add new "Customize Design" functionality (see "My Page" -> "Design" -> "Customize your design"). It allows you to customize your page background and header without knowing CSS. Note that you can't customize your page if you use the default theme! See the post in the community blog for more details.
- New full-screen slideshow feature for photo albums.
- Improvements in social network integration: now you have the possibility of sharing your blog posts automatically on Facebook. We also fixed some bugs.
- New autocomplete feature when sending PMs: just start writing the username of a friend, and a drop-down will appear with possible completions.
- Add a new cool feature to the friend invite page and the messages page: you can now select a range of messages by clicking the checkbox for the first in the range, then Shift-clicking the checkbox for the last in the range.
- Add a "Accept all friend requests" button.
- Fixed a problem with "Save draft" that would change publication dates for blog posts.
- Add Twingly ping support for blog posts.
- Publish blog updates on other social networks when blogging via MMS.
- Photo ratings are now shown to the owner.
- Fixed an issue with multiple choice answers in polls and the preview function.
- Now attached images in forums link to the original.
- Other small fixes and tweaks to the forums.
- Small fixes for the Link interface, particularly for notes and Mini/Mobile.
- Add a date selection widget to the events/countdowns, no more typing dates by hand. You'll need a browser supporting the "date" input type, such as Opera, to see it.
- Fix some issues with avatar resizing. Some issues still pending, currently looking at them. If you have concrete issues that are not fixed, please add a comment with a link to the original image.
- Small bug fixes for the wallpapers section.
- OEmbed bug fix: property "provider_name" is now correctly named (it used to be "provider").
And of course, a bunch of small fixes, tweaks and improvements everywhere in the site.
EDIT: I forgot to mention improvements in social network integration. Added now.
By Cosimo Streppone.
Friday, 2. October 2009, 11:41:28
panic, servers, files, Downtime
...
This morning around 8 AM CEST we found out that no file from files.myopera.com was accessible anymore.
If you were using My Opera at that time, you certainly saw lots of Forbidden (403s) errors.
The server couldn't access any file on the filesystem.
All pictures gone, all user css gone, every user file missing.
Connecting to the machine, and trying to get directory contents, we only got back
I/O errors.
We tried to remount the folder, with no luck. We also tried to stop all services, unmount and remount. Didn't work. Then we rebooted the server, hoping for a clean mount of the filesystem.
It worked. Except that it was down again 2 hours later.
Our mighty sysadmins started working on the issue, and by lunch time, it was fixed.
Then we restarted the services, and everything looked good. At around 12:15 CEST, the files were back online. No data loss reported. Just panic for a while :-)
What happened is that a tiny little fibre channel connector, also known as
GBIC, decided to leave this cruel world this morning.
Right on time to celebrate our 3,000,000 users!
By Cosimo Streppone.
Friday, 25. September 2009, 16:48:08
My Opera, storage, pictures, albums
On September 21st, we finally reached 10 million pictures in our database!
Congratulations to all our users! Keep going!
By Cosimo Streppone.
Tuesday, 21. July 2009, 17:42:52
javascript, MyOpera, epic fail, caching
...
My Opera was on Slashdot again today!
This time is Hallvord's fault :-)
Yesterday he wrote a blog post about an epic Javascript fail that today was picked up by several major web sites in Norway and in the U.S. and mostly probably elsewhere.
We realized there was something strange going on because of the unusual load and number of requests on our front-end servers. Analyzing the logs, it was clear that Hallvord's post was getting lots of pageviews per second, mostly from Slashdot, but also from Reddit and from the norwegian online magazine Digi.
The blog post was immediately put into our list of "hot content" to be cached, and everything was fine again in a couple of minutes. The idea for the future is to be able to automatically detect such high number of requests, at least for blog posts, and automatically cache them. Another alternative could be to enable blog post caching by default, but this should be done very carefully, because blog pages contain dynamic content that usually depends on the visiting user and on the user who is the "owner" of the page.
We'll see...
By Cosimo Streppone.
Thursday, 16. July 2009, 11:22:22
test suite, continuous integration, development, cc.rb
...
On the My Opera project, and many other Opera projects, we try to follow the practices of continuous integration.
I'm sure our QA Ninja would be glad to talk about this a bit more. Anyway... back in the days, Nicolas wrote a widget to keep track of our build status. If the test suite failed, the widget would be red. When tests succeeded, it turned green. Great stuff.
Esteban recently picked it up and built from there with some improvements and made it fully generic. It's based on the continuous integration product that we use internally, Cruise Control.
For the curious, the Widget is publicly available on widgets.opera.com, and the current snapshot is:
:-)
By Cosimo Streppone.
Tuesday, 23. June 2009, 09:34:46
opera unite, development, servers, traffic
...
Tuesday 16th of June was a special day for My Opera too.
Opera Unite was released at 09:00 CEST, and just after that, the amount of traffic and visitors to My Opera
increased quickly, to reach the highest peak around 18:00 UTC.
This was also the highest peak ever reached for My Opera.
On that day, we finally broke the 2 million page views per day, that sums up to more than 100M hits and 600,000 unique visitors just that day.
We're very happy about this. Yes, the site was not completely usable, and sometimes it was serving pages very slowly, but still we managed to keep it running. We hacked together some quick and dirty fixes to achieve this. Some of those hacks have been removed now, and we're working on applying them permanently to the site.
For the curious/interested, I'm referring to what we call the "User" module, that provides most of the user-related information. When you open a page on My Opera, we call the user module at least for:
- the visiting user (you), and
- the user "owner" of the page (the my.opera.com/<this_bit_here>)
Not a case that this is the
most called module throughout the whole web site, so we want to optimize it and make it really fast, ideally around a few milliseconds.
Here's a few
munin charts, of the database connections (this is a weekly chart, so look around the "16"):
And of one of our backends, with a rough timeline of what happened:
Of course, there's still a lot of work to do, and we know which are the slower parts of the site. That might change while the community grows bigger and bigger every day. We will keep up. :-)
WIDGETIZE