Many of you noticed problems in the sidebar and in the friends pages (or friends in the sidebar) since "Apricot" was released.
This is the dev blog, so let's make a step backward, and then try to explain what happened.
Most of the My Opera pages are "DML pages".
DML Pages
The DML concept was probably invented in the first big rewrite of My Opera, around 2002-2003. It basically consists in a XML template very similar to an HTML page. Here's a stripped down version of the My Opera homepage:
<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//Opera//DML//EN" "/DTD/dml.dtd">
<?xml-stylesheet type="text/xsl" href="/xsl/community/home.xsl"?>
<html>
<head>
<title<Share your photos, blog and meet friends</title>
...
<body>
<definemenu>
<selected>home</selected>
<page>home</page>
</definemenu>
<include module="XWA::User"/>
...
<include module="XWA::XMLProd::MemberCount" />
<include module="MyOpera:: DML::LatestActivities" />
<include module="MyOpera:: DML::CommunityNews" var="numposts=3" />
</body>
</html>
This DML document is then processed by one of the core modules of My Opera, and all the include directives are expanded. The final document is then processed with the specified XSLT stylesheet.
This system has probably some drawbacks, but allows the front-end developers, if they know XSLT, to include backend modules and build pages with whatever dynamic content they want. We have a long list of DML modules (the included modules, like MyOpera:: DML::LatestActivities).
That means that you can put the latest activities data stream wherever you want, in any page on My Opera.
Heavy DML modules
Some of the heaviest modules, according to our performance profiling, are:
-
XWA::XMLProd::User::Sidebar (awful name, and awful code) it's the slowest and heaviest of all, and of course, provides each user the sidebar content
-
XWA::User::Friends, provides your friends data, names, pictures, etc...
-
XWA::User, contains your own data, or the current page owner (for blogs, etc) data.
-
XWA::User::Journal, builds a blog post page data.
This is why when we have server-side problems, sometimes you see messages like: "XWA::User::Friends died..." :-)
When Apricot went live, it was fine for 1/2 hours, then when we reached our peak traffic hour, everything was really slow and heavy on the servers. That's also a reason why it's difficult to avoid these problems by testing, because currently we have no way to reproduce the live systems traffic load. BTW, we're trying to get hardware to build a stress test cluster...
The Friends module
We traced this problem back to the Sidebar and Friends modules. As a first emergency measure, we disabled both modules and started analyzing their performance. Particularly, the Friends module was completely reworked for the new friends system that requires approval now. We found there was a new is_online() function that was found to be the really heavy part.
Even if the "online" check has been there for ages, it was reworked in a more modular way, and that caused also performance problems. Unfortunately, these problems only manifest clearly when thousands of concurrent requests are running...
Another source of problems was the fact that now the Sidebar and Friends modules have been added to many more pages than before, so they are invoked many more times on average. This is something we hadn't really realized before going live.
Current status
The Sidebar module has been partially rewritten to take advantage of caching. As many noticed, now your sidebar doesn't update immediately. Some users also reported that their latest visitors sidebar panel is not updating since the release. Of course, being Sidebar now cached, the visitors are not tracked for every request. We're working on this, and visitors will be restored soon.
About the Friends module, we will restore ASAP the "online" functionality as well. Just have a bit more patience. Other minor bugs will be addressed in the next days. In the meanwhile, we hope you appreciated all the new and improved features this Apricot release brought you.
We just sent out a couple of patches that should fix 2
of the most annoying (small?) Apricot bugs.
WIDGETIZE