My Opera News

The scheduled database maintenance was completed on Saturday. Well, it wasn't exactly maintenance, but rather an important change of setup. Now My Opera is running on a master to master replication setup, instead of the classic master-slave that has been running for some years now.

There's at least 2 good reasons why we chose that setup:

• it allows for database master maintenance with practically no downtime, since you can have the site run on your secondary master, while the primary is taken offline. When maintenance is over, you can switch back to the primary.
  
• in case of disaster or failures to the primary master, the secondary can kick in (manually for now) in a rather short time. Last time we had a primary master db crash, it took us something around 4-6 hours to be back online.
  

So, currently My Opera is running on the secondary master database, while the primary is being prepared. "Being prepared" in this case means that we're converting some of our heavy duty tables to the InnoDB storage engine. This was really needed because MyISAM table-level locking poses some hard limits on concurrency and scalability of your database, even if it's very fast. Add that we have more user activity, APIs and services than ever before, and you have the whole picture.

Of course, all this work should also improve raw performance of the site. Once we get back to our primary master server, which we will probably do in the next few days, the site will be definitely faster. Not ludicrous speed yet, but faster

After we cut down the database size by an impressive 22 Gb yesterday (and we had our share of problems afterwards, as you probably noticed...), we managed to get some performance-related fixes out live.

According to our monitoring tools, all the servers are running faster and with lot less load than ever before. Of course, it's not going to last long, because traffic is slowly but continuously increasing. At least it should give us some more time to relax a bit (read: dedicate only to development stuff...).

For the curious among you, I've attached a screenshot of one of our own monitoring tools. A pity you can't see the realtime updates. This is taken at peak time, and it shows load, active threads and replication status for each machine. It shows most of the My Opera backend machines:

where green stuff is OK, while orange or red is not.

To add more fun, one of our main database servers has been repeatedly crashing during the weekends (mostly friday and saturday nights, super-fun!). Most of the time My Opera has been up and running without being affected. Today this server was updated with a brand new kernel and disk driver version. It shouldn't crash anymore. Now we're pushing it at full speed. Let's see...

For Christmas, we guys down here in the monkey dungeon wanted hardware. Even more hardware. A month or so ago we were rumoured to have a very prominent link to the Opera Community on the Nintendo Wii Browser, and we were obviously a tiny bit worried about all the lucky folks getting a Wii under their Christmas tree barging through our doors simultaneously. (I don't think we got a prominent link though. )

So, we already have Bigma and Bigpa. Hence, enter Fatboy:


Fatboy is two IBM x3950 servers virtualised into one brutish 8 Dual Core CPU, 64GB RAM monster. With this solution we'll be able to scale both horizontally (by adding more auxilliary servers) and vertically (by simply stacking more x3950 boxes on top of the ones we have resulting in one logical server with more CPUs).

Fatboy has been fed Debian Etch, and is now currently taking over the role of Bigpa as database serving visitors which are not logged in, while Bigpa is currently put on search query duty. Our trusty Bigma master database server underwent a slight downtime period yesterday to update the firmware of the RAID controller, which will hopefully make her run smoothly through the holidays.

Fatboy originally came delivered from IBM with a puny amount of RAM (3GB), because IBM having some warehouse in Poland being raided or something. So we had to insert all those 32 sticks of RAM ourselves! Manual labour! Below, we have Thomas holding the loot and Sverre feeding it to Fatboy.


We hope by this that we will be able to bring you the gift of gifts this Christmas: a stable Opera Community. Atleast we have tried to stress-test the system, and with significant extra load, it weathered the requests we threw at it well. If, against (according to?) all odds something breaks, we also have 24x7 on-call duty during the holidays in case of any severe problems.