My Opera News

The scheduled database maintenance was completed on Saturday. Well, it wasn't exactly maintenance, but rather an important change of setup. Now My Opera is running on a master to master replication setup, instead of the classic master-slave that has been running for some years now.

There's at least 2 good reasons why we chose that setup:

• it allows for database master maintenance with practically no downtime, since you can have the site run on your secondary master, while the primary is taken offline. When maintenance is over, you can switch back to the primary.
  
• in case of disaster or failures to the primary master, the secondary can kick in (manually for now) in a rather short time. Last time we had a primary master db crash, it took us something around 4-6 hours to be back online.
  

So, currently My Opera is running on the secondary master database, while the primary is being prepared. "Being prepared" in this case means that we're converting some of our heavy duty tables to the InnoDB storage engine. This was really needed because MyISAM table-level locking poses some hard limits on concurrency and scalability of your database, even if it's very fast. Add that we have more user activity, APIs and services than ever before, and you have the whole picture.

Of course, all this work should also improve raw performance of the site. Once we get back to our primary master server, which we will probably do in the next few days, the site will be definitely faster. Not ludicrous speed yet, but faster

Tuesday 16th of June was a special day for My Opera too.

Opera Unite was released at 09:00 CEST, and just after that, the amount of traffic and visitors to My Opera increased quickly, to reach the highest peak around 18:00 UTC.

This was also the highest peak ever reached for My Opera. On that day, we finally broke the 2 million page views per day, that sums up to more than 100M hits and 600,000 unique visitors just that day.

We're very happy about this. Yes, the site was not completely usable, and sometimes it was serving pages very slowly, but still we managed to keep it running. We hacked together some quick and dirty fixes to achieve this. Some of those hacks have been removed now, and we're working on applying them permanently to the site.

For the curious/interested, I'm referring to what we call the "User" module, that provides most of the user-related information. When you open a page on My Opera, we call the user module at least for:

• the visiting user (you), and
  
• the user "owner" of the page (the my.opera.com/<this_bit_here>)
  

Not a case that this is the most called module throughout the whole web site, so we want to optimize it and make it really fast, ideally around a few milliseconds.

Here's a few munin charts, of the database connections (this is a weekly chart, so look around the "16"):



And of one of our backends, with a rough timeline of what happened:



Of course, there's still a lot of work to do, and we know which are the slower parts of the site. That might change while the community grows bigger and bigger every day. We will keep up. :-)

The Apricot release is now stabilized, after a few days of problems here and there.

The good thing is that we learned something more. The bad thing is that it took us a couple of days to realize what the real problem was.

I always like to say that My Opera becomes stronger and faster only after some "crisis" moments. It's those performance/stability/whatever problems that demand highest attention and you have to do something to fix them, no matter what. So, whatever doesn't kill you, it makes you stronger. :-)

Initially, we thought the problems were in the new friends system code. After careful inspection and partial rewrite of both Sidebar and Friends components, it became evident that the problem had another cause. As many other sites, we use memcached on several machines. It acts as a distributed cache store that makes the site faster and able to serve more requests with the same hardware resources.

Before we started the emergency rewrite of the Sidebar module, we decided to take a closer look at the class that was handling the memcached connections and queries. What we found out (part of what My Opera is today is still legacy stuff for us...) is that this class was issuing a new connection to our memcached server each and every time we queried some data. This is obviously not optimal, so we decided to keep persistent connections to the servers.

As of now, we restored all the functionality in Sidebar and Friends. Sidebar and Friends data is now present on much more pages than before, and that means more backend power needed to serve the same pageviews. We also have a lot more features and queries to the database, but still we're able to cope with the traffic, which is btw quickly increasing.

The Sidebar in particular was broken because when we introduced caching for it, it happened that the same (cached) version of the Sidebar content was used for the /albums/ and for the /blog/ page, which is completely wrong. Now we keep two separate copies because they can, and will, be different. The Sidebar content is also kept in the cache for 30 minutes at most, unless you change your sidebar settings. The recent visitors list didn't update because of this caching. Now everything should be back to normal.
We will probably make sure that your sidebar is updated when you write a new blog post or when there's a new comment, etc... For now this is just planned...

In the meanwhile, our monitoring team (LOL) noticed that at peak time and at some random moments in the day, the site slows down for a couple of minutes, and then everything is back to normal. We're working to solve this by optimizing the code even more. Without breaking anything, if possible... :-)

Big surprises ahead... And you don't know the super-secret feature yet... :-)