files.myopera.com outage this morning
By Cosimo Streppone. Friday, 2. October 2009, 11:41:28
This morning around 8 AM CEST we found out that no file from files.myopera.com was accessible anymore.
If you were using My Opera at that time, you certainly saw lots of Forbidden (403s) errors.
The server couldn't access any file on the filesystem.
All pictures gone, all user css gone, every user file missing.
Connecting to the machine, and trying to get directory contents, we only got back I/O errors.
We tried to remount the folder, with no luck. We also tried to stop all services, unmount and remount. Didn't work. Then we rebooted the server, hoping for a clean mount of the filesystem.
It worked. Except that it was down again 2 hours later.
Our mighty sysadmins started working on the issue, and by lunch time, it was fixed.
Then we restarted the services, and everything looked good. At around 12:15 CEST, the files were back online. No data loss reported. Just panic for a while :-)
What happened is that a tiny little fibre channel connector, also known as GBIC, decided to leave this cruel world this morning.
Right on time to celebrate our 3,000,000 users!
If you were using My Opera at that time, you certainly saw lots of Forbidden (403s) errors.
The server couldn't access any file on the filesystem.
All pictures gone, all user css gone, every user file missing.
Connecting to the machine, and trying to get directory contents, we only got back I/O errors.
We tried to remount the folder, with no luck. We also tried to stop all services, unmount and remount. Didn't work. Then we rebooted the server, hoping for a clean mount of the filesystem.
It worked. Except that it was down again 2 hours later.
Our mighty sysadmins started working on the issue, and by lunch time, it was fixed.
Then we restarted the services, and everything looked good. At around 12:15 CEST, the files were back online. No data loss reported. Just panic for a while :-)
What happened is that a tiny little fibre channel connector, also known as GBIC, decided to leave this cruel world this morning.
Right on time to celebrate our 3,000,000 users!
WIDGETIZE