Extended Outage Details

submit to reddit

If you're a regular mock drafter, you're already aware that Fantasy Football Calculator has had an extended downtime of more than 36 hours.  Although we're far from the peak mock draft season of August, that type of outage is not acceptable.  I'm sorry that you haven't been able to do mock drafts during this time.

Here is an explanation of what happened, and what the plan is to prevent this from happening again in the future:

Cause of the downtime

Fantasy Football Calculator has been hosted on Amazon Web Services for the past 3 years.  Their track record has been great in that time, with only a few problems here and there.  If you have experienced any technical issues with the website in the past, it was most likely not Amazon's fault, and was an issue with scaling the website to handle the high traffic that it's been receiving in recent years.

On April 21st, Amazon started reporting problems, primarily with their Elastic Block Storage (EBS) product, which FFC uses for it's database storage.  Many other larger companies, such as Reddit and Netflix use this technology as well and reported that they were affected by it to varying degrees.  You can read Amazon's description of what happened on it's status page, and I won't bore you with the technical details here.  Anyways, Amazon finally resolved the issues that affected FFC's database, and the website came back online at 3:45PM EST April 23rd.  No data was lost and everything is back to normal.

Prevention and future plans

FFC has regular backups in place, but this outage prevented us from restoring from them because they relied on Amazon's EBS technology that was disabled during this period.  The last off-site backup was taken a month ago.   Just before the site was restored today, another off-site backup was made.

Obviously, more frequent off-site backups need to be taken.  If that had been in place, FFC could have been restored from one of those, and that would have reduced the downtime significantly.

Also, in the near future, FFC will move away from Amazon's EBS technology completely because it has proven to be unreliable.  It offers a lot of nice features, but with problems like this, it can't be trusted.

Lastly, FFC may stop using Amazon for hosting completely and switch to another company, such as Linode.  One of the features keeping FFC on Amazon was EBS.  If that is no longer being used, then alternative hosting companies may be better options.

Summary

  • FFC experienced about 2 days worth of downtime, which is unacceptable.
  • It was due to a large problem with Amazon's web hosting services that affected many other companies as well
  • The plan is to stop using the specific product that failed (EBS) and make more frequent off-site backups
  • FFC may switch away from Amazon hosting completely

To make up for the downtime, key rookies will be added to the database this weekend so that you can draft them before they're drafted in the real NFL draft next week!  The bye weeks will be updated to reflect the recently released NFL schedule as well.

Again, I'm sorry that this happened. Feel free to leave comments or contact me if you have any questions.

[Edit - August 1, 2011]: The site has been moved off of Amazon's EBS technology and off of Amazon's hosting completely.  It's now on Linode.  See the blog post about the migration here.

Never Reach For a Player Again!

Join the 18,000 others who stay up to date!

Get FREE articles now!



Categories: admin