My Opera News

March 2009

( Monthly archive )

Disappearing user pictures

By Cosimo Streppone. Tuesday, 24. March 2009, 17:21:03

Lately we have noticed two different problems, one with user pictures and another one with default avatars.

Sometimes it happens that user pictures just come up broken. For some reason, they don't show. This is a bit weird, and difficult to reproduce. Every time you try to reload the broken pictures, they misteriously show again.

We think we found out why. Analyzing static.myopera.com access logs, we found that some requests were being served as 503, which, according to the HTTP spec, is:

10.5.4 503 Service Unavailable

   The server is currently unable to handle the request due to a
   temporary overloading or maintenance of the server. The implication
   is that this is a temporary condition which will be alleviated after
   some delay. If known, the length of the delay MAY be indicated in a
   Retry-After header. If no Retry-After is given, the client SHOULD
   handle the response as it would for a 500 response.

      Note: The existence of the 503 status code does not imply that a
      server must use it when becoming overloaded. Some servers may wish
      to simply refuse the connection.

So, 503s are sent to clients when the server can't serve more requests, or when there are bandwidth or requests per second limits. In fact, some months ago we set some per-client bandwidth and connection limits, to allow fair usage of resources for all our users. Now we have raised the limits a bit, and we're keeping the access log monitored, trying to minimize "503" errors.

Hopefully this should eliminate all the "missing-pictures" problems. Shout if you still see broken pictures.
About avatars, it might be a bit trickier, since avatars are not served from static.myopera.com, so it cannot be the same problem. So, is anyone in the community able to reproduce the "default-avatar-missing" problem with any browser?

13 comments

Yet another performance optimization round

By Cosimo Streppone. Wednesday, 18. March 2009, 15:52:50

servers, development, apricot, optimization ...

After we cut down the database size by an impressive 22 Gb yesterday (and we had our share of problems afterwards, as you probably noticed...), we managed to get some performance-related fixes out live.

According to our monitoring tools, all the servers are running faster and with lot less load than ever before. Of course, it's not going to last long, because traffic is slowly but continuously increasing. At least it should give us some more time to relax a bit (read: dedicate only to development stuff...).

For the curious among you, I've attached a screenshot of one of our own monitoring tools. A pity you can't see the realtime updates. This is taken at peak time, and it shows load, active threads and replication status for each machine. It shows most of the My Opera backend machines:

where green stuff is OK, while orange or red is not.

To add more fun, one of our main database servers has been repeatedly crashing during the weekends (mostly friday and saturday nights, super-fun!). Most of the time My Opera has been up and running without being affected. Today this server was updated with a brand new kernel and disk driver version. It shouldn't crash anymore. Now we're pushing it at full speed. Let's see...

16 comments

Happy birthdays!

By Esteban Manchado Velázquez. Friday, 13. March 2009, 12:26:19

birthday, www, web, present

Hey there!

Our very own Kristin has a birthday tomorrow!

She has been with us for a bit more than 6 months already, doing a great job as part of the team

Today she received a small-present-in-advance from "the bestest bf in the world!" (in her own words

), and uploaded the pictures to My Opera. Go check them out!

It's also cool because today is the birthday of the World Wide Web!

So happy birthday to both, Kristin and the Web

11 comments

Status of sidebar, friends and other stuff...

By Cosimo Streppone. Thursday, 12. March 2009, 17:46:54

RELEASE, MyOpera, servers, development

The Apricot release is now stabilized, after a few days of problems here and there.

The good thing is that we learned something more. The bad thing is that it took us a couple of days to realize what the real problem was.

I always like to say that My Opera becomes stronger and faster only after some "crisis" moments. It's those performance/stability/whatever problems that demand highest attention and you have to do something to fix them, no matter what. So, whatever doesn't kill you, it makes you stronger. :-)

Initially, we thought the problems were in the new friends system code. After careful inspection and partial rewrite of both Sidebar and Friends components, it became evident that the problem had another cause. As many other sites, we use memcached on several machines. It acts as a distributed cache store that makes the site faster and able to serve more requests with the same hardware resources.

Before we started the emergency rewrite of the Sidebar module, we decided to take a closer look at the class that was handling the memcached connections and queries. What we found out (part of what My Opera is today is still legacy stuff for us...) is that this class was issuing a new connection to our memcached server each and every time we queried some data. This is obviously not optimal, so we decided to keep persistent connections to the servers.

As of now, we restored all the functionality in Sidebar and Friends. Sidebar and Friends data is now present on much more pages than before, and that means more backend power needed to serve the same pageviews. We also have a lot more features and queries to the database, but still we're able to cope with the traffic, which is btw quickly increasing.

The Sidebar in particular was broken because when we introduced caching for it, it happened that the same (cached) version of the Sidebar content was used for the /albums/ and for the /blog/ page, which is completely wrong. Now we keep two separate copies because they can, and will, be different. The Sidebar content is also kept in the cache for 30 minutes at most, unless you change your sidebar settings. The recent visitors list didn't update because of this caching. Now everything should be back to normal.
We will probably make sure that your sidebar is updated when you write a new blog post or when there's a new comment, etc... For now this is just planned...

In the meanwhile, our monitoring team (LOL) noticed that at peak time and at some random moments in the day, the site slows down for a couple of minutes, and then everything is back to normal. We're working to solve this by optimizing the code even more. Without breaking anything, if possible... :-)

Big surprises ahead... And you don't know the super-secret feature yet... :-)

18 comments

Sidebar and friends

By Cosimo Streppone. Saturday, 7. March 2009, 16:26:02

sidebar, RELEASE, MyOpera, apricot ...

Many of you noticed problems in the sidebar and in the friends pages (or friends in the sidebar) since "Apricot" was released.

This is the dev blog, so let's make a step backward, and then try to explain what happened. Most of the My Opera pages are "DML pages".

DML Pages

The DML concept was probably invented in the first big rewrite of My Opera, around 2002-2003. It basically consists in a XML template very similar to an HTML page. Here's a stripped down version of the My Opera homepage:

<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//Opera//DML//EN" "/DTD/dml.dtd">
<?xml-stylesheet type="text/xsl" href="/xsl/community/home.xsl"?>
<html>
<head>
<title<Share your photos, blog and meet friends</title>
...
<body>

  <definemenu>
    <selected>home</selected>
    <page>home</page>
  </definemenu>

  <include module="XWA::User"/>
  ...
  <include module="XWA::XMLProd::MemberCount" />
  <include module="MyOpera:: DML::LatestActivities" />
  <include module="MyOpera:: DML::CommunityNews" var="numposts=3" />

  </body>
</html>

This DML document is then processed by one of the core modules of My Opera, and all the include directives are expanded. The final document is then processed with the specified XSLT stylesheet.

This system has probably some drawbacks, but allows the front-end developers, if they know XSLT, to include backend modules and build pages with whatever dynamic content they want. We have a long list of DML modules (the included modules, like MyOpera:: DML::LatestActivities).
That means that you can put the latest activities data stream wherever you want, in any page on My Opera.

Heavy DML modules

Some of the heaviest modules, according to our performance profiling, are:

XWA::XMLProd::User::Sidebar (awful name, and awful code) it's the slowest and heaviest of all, and of course, provides each user the sidebar content
XWA::User::Friends, provides your friends data, names, pictures, etc...
XWA::User, contains your own data, or the current page owner (for blogs, etc) data.
XWA::User::Journal, builds a blog post page data.

This is why when we have server-side problems, sometimes you see messages like: "XWA::User::Friends died..." :-)

When Apricot went live, it was fine for 1/2 hours, then when we reached our peak traffic hour, everything was really slow and heavy on the servers. That's also a reason why it's difficult to avoid these problems by testing, because currently we have no way to reproduce the live systems traffic load. BTW, we're trying to get hardware to build a stress test cluster...

The Friends module

We traced this problem back to the Sidebar and Friends modules. As a first emergency measure, we disabled both modules and started analyzing their performance. Particularly, the Friends module was completely reworked for the new friends system that requires approval now. We found there was a new is_online() function that was found to be the really heavy part.
Even if the "online" check has been there for ages, it was reworked in a more modular way, and that caused also performance problems. Unfortunately, these problems only manifest clearly when thousands of concurrent requests are running...

Another source of problems was the fact that now the Sidebar and Friends modules have been added to many more pages than before, so they are invoked many more times on average. This is something we hadn't really realized before going live.

Current status

The Sidebar module has been partially rewritten to take advantage of caching. As many noticed, now your sidebar doesn't update immediately. Some users also reported that their latest visitors sidebar panel is not updating since the release. Of course, being Sidebar now cached, the visitors are not tracked for every request. We're working on this, and visitors will be restored soon.

About the Friends module, we will restore ASAP the "online" functionality as well. Just have a bit more patience. Other minor bugs will be addressed in the next days. In the meanwhile, we hope you appreciated all the new and improved features this Apricot release brought you.

28 comments

Links to picture comments! AKA #932

By Cosimo Streppone. Friday, 6. March 2009, 16:52:07

subscriptions, #932, watches, comments ...

We called this bug #932. It had been there for 6 months before I even joined Opera Software. Now it should be finally fixed.

In your subscriptions, when you had a comment to a single picture, you clicked on the link and you were sent to the album page. Then you had to look for the picture that had the comment.

This is now fixed, so you can click on the link and you will be sent to the exact comment on that specific picture. Hooray for a 20 months-old bug that is finally fixed.

I think at least a thousands of My Opera members have probably requested a fix for this.
And I hope the 365 members will be happy now. :-)

34 comments

We can has Apricots!

By Esteban Manchado Velázquez. Wednesday, 4. March 2009, 14:07:47

RELEASE, changelog, apricot

After a long wait, we have finally released "Apricot", the first My Opera release of 2009. In the original announcement you can see the more visible changes. A more complete list follows:

Brand new, beta, Flash-based photo uploader. You have to click on the "Switch to Flash uploader" button on the right to use it, and allows to select several pictures at once for upload. Please submit any problems you find.

New friend system, so that only people who you have "approved" can add you as friends.

Added as many as 9 new translations in this release. Now we have My Opera available in 17 languages!

Improved design and usability in photo albums.

New welcome page for people who has just signed up for a new account. Among others, it makes it easy and quick to specify your country, so you'll be able to find more people in the Members by country page.

My Opera buttons are now text based, which is better for accessibility and for translation.

Improvements in the skin section, especially for authors. Now you get suggestions for improvement when you upload a skin with commonly-made mistakes, and you can specify that your skin is for Opera 9.5 (ie. skin version 3).

Now blog posts and comments are checked for correct syntax before being sent. That will avoid some problems people have had with broken XML, which used to "break" their blogs.

We have also fixed some issues in Dev Opera, mostly related to profile editing.

We have started a documentation page for our public APIs. It's linked from the bottom of the page.

The order of the subscription items is now from newest to oldest, so it's easier to find your newest subscriptions.

When you make a comment on a blog, the page correctly "jumps" to it again.

Fixed a lot of bugs here and there, too many and too small to mention in detail.