My Opera News

Back to our primary master database

By Cosimo Streppone. Thursday, 19. November 2009, 18:09:25

innodb, development, slaves, servers ...

My Opera is now back to our more powerful primary master database.
In fact, since the switch we have been running on our secondary (as primary

Now our old primary, that became secondary, is the primary again.

We officially switched part of the My Opera database to InnoDB. This has been on the wishlist for such a long time... Now it's finally done. But it's not over. We need to proceed the conversion for all our slaves. This will happen "in the background"...

The failover procedure was really smooth. We're getting better at it, and we plan to automate it completely. I'm not sure if we will ever achieve this, but it doesn't seems to be really hard.
We already have a switch-master script that we can launch against each slave database, and it will safely switch the master db, checking that everything is in order, and it worked quite well today.

Someone the other day on this blog asked about Postgres. I really like Pg, I used it for so many years. MySQL can be good too. We'll see...

8 comments

My Opera now running on a master to master replication setup

By Cosimo Streppone. Monday, 16. November 2009, 21:25:11

servers, development, master-master, replication ...

The scheduled database maintenance was completed on Saturday. Well, it wasn't exactly maintenance, but rather an important change of setup. Now My Opera is running on a master to master replication setup, instead of the classic master-slave that has been running for some years now.

There's at least 2 good reasons why we chose that setup:

it allows for database master maintenance with practically no downtime, since you can have the site run on your secondary master, while the primary is taken offline. When maintenance is over, you can switch back to the primary.
in case of disaster or failures to the primary master, the secondary can kick in (manually for now) in a rather short time. Last time we had a primary master db crash, it took us something around 4-6 hours to be back online.

So, currently My Opera is running on the secondary master database, while the primary is being prepared. "Being prepared" in this case means that we're converting some of our heavy duty tables to the InnoDB storage engine. This was really needed because MyISAM table-level locking poses some hard limits on concurrency and scalability of your database, even if it's very fast. Add that we have more user activity, APIs and services than ever before, and you have the whole picture.

Of course, all this work should also improve raw performance of the site. Once we get back to our primary master server, which we will probably do in the next few days, the site will be definitely faster. Not ludicrous speed yet, but faster

13 comments

Scheduled downtime for database maintainance

By Cosimo Streppone. Friday, 13. November 2009, 11:01:42

servers, Planned downtime, Downtime, database

We are planning a database maintainance operation today from 19:00 UTC, 20:00 CET. Estimated downtime should be around 1 hour.

We tested this already on other db systems and we definitely saw good results, so cross your fingers.

We're looking forward to have a faster My Opera.

EDIT (2009/11/14 00:13:37): There has been no downtime so far.
We're continuing the maintainance operation tomorrow. Still 1 hour of estimated downtime. That will probably happen during the afternoon or early evening.

25 comments

Static resources on lighttpd

By Cosimo Streppone. Thursday, 29. October 2009, 21:11:58

scalability, lighttpd, httpd, static ...

It's been a while since we deployed the first static server for My Opera.
It was a really necessary step, because we had, and we continue to have, lots of static resources to be served.
A while back, most of these resources were served by applications, causing much more load on servers than necessary.
Now most of the heaviest ones have been already moved to the static servers.

Some months ago we also added support for partitioning of the user resources in our storage software layer. That means making sure that if we have the need to scale serving of resources on different machines, we can either replicate the entire content on different ones, or split the content over 2 or more machines. That part worked nicely so far. That's why at some point there was a static.myopera.com and static02.myopera.com.

Now there's also static03.myopera.com

We changed our setup again, transparently (for you users), to consolidate the previously recycled/temporary hardware into a new shiny machine with more disk space. On this machine, we installed lighttpd instead of our usual Apache setup. We wanted to try out this software. For us it was the first time we tried it on a production setup.

Edoardo played with it for a while, prepared the setup and installed it on static03. As of today, it has been running perfectly for nearly 1 month with a really low load, and peaks of 250 accesses per second. It's serving around 14M hits per day for avatars, user pictures, skin thumbnails, etc...

19 comments

Dragonfruit release

By Esteban Manchado Velázquez. Tuesday, 20. October 2009, 08:51:03

dragonfruit, changelog, RELEASE

Yesterday we released a new version of My Opera, codenamed "Dragonfruit".

Each of the important new features in the release will be explained in detail in the community blog. This is the detailed changeset for Dragonfruit:

Add new "phototagging" feature to the photo albums. See the post in the community blog for more details.
Add new "Customize Design" functionality (see "My Page" -> "Design" -> "Customize your design"). It allows you to customize your page background and header without knowing CSS. Note that you can't customize your page if you use the default theme! See the post in the community blog for more details.
New full-screen slideshow feature for photo albums.
Improvements in social network integration: now you have the possibility of sharing your blog posts automatically on Facebook. We also fixed some bugs.
New autocomplete feature when sending PMs: just start writing the username of a friend, and a drop-down will appear with possible completions.
Add a new cool feature to the friend invite page and the messages page: you can now select a range of messages by clicking the checkbox for the first in the range, then Shift-clicking the checkbox for the last in the range.
Add a "Accept all friend requests" button.
Fixed a problem with "Save draft" that would change publication dates for blog posts.
Add Twingly ping support for blog posts.
Publish blog updates on other social networks when blogging via MMS.
Photo ratings are now shown to the owner.
Fixed an issue with multiple choice answers in polls and the preview function.
Now attached images in forums link to the original.
Other small fixes and tweaks to the forums.
Small fixes for the Link interface, particularly for notes and Mini/Mobile.
Add a date selection widget to the events/countdowns, no more typing dates by hand. You'll need a browser supporting the "date" input type, such as Opera, to see it.
Fix some issues with avatar resizing. Some issues still pending, currently looking at them. If you have concrete issues that are not fixed, please add a comment with a link to the original image.
Small bug fixes for the wallpapers section.
OEmbed bug fix: property "provider_name" is now correctly named (it used to be "provider").

And of course, a bunch of small fixes, tweaks and improvements everywhere in the site.

EDIT: I forgot to mention improvements in social network integration. Added now.

19 comments

files.myopera.com outage this morning

By Cosimo Streppone. Friday, 2. October 2009, 11:41:28

panic, servers, files, Downtime ...

This morning around 8 AM CEST we found out that no file from files.myopera.com was accessible anymore.
If you were using My Opera at that time, you certainly saw lots of Forbidden (403s) errors.

The server couldn't access any file on the filesystem.
All pictures gone, all user css gone, every user file missing.

Connecting to the machine, and trying to get directory contents, we only got back I/O errors.
We tried to remount the folder, with no luck. We also tried to stop all services, unmount and remount. Didn't work. Then we rebooted the server, hoping for a clean mount of the filesystem.

It worked. Except that it was down again 2 hours later.

Our mighty sysadmins started working on the issue, and by lunch time, it was fixed.
Then we restarted the services, and everything looked good. At around 12:15 CEST, the files were back online. No data loss reported. Just panic for a while :-)

What happened is that a tiny little fibre channel connector, also known as GBIC, decided to leave this cruel world this morning.

Right on time to celebrate our 3,000,000 users!

12 comments