Yesterday, we announced our plans to add some major upgrades to our server infrastructure for our LightCMS, WaterCMS, SkyCMS and Icebrrg products. Today, we initiated the upgrade process but, due to some complications, we decided to postpone the upgrades and revert to our prior hosting setup.

What happened?

We ran into some network routing complications that couldn't be adequately addressed within the confines of our maintenance window. As a result, we decided to postpone the upgrades. As soon as that decision was made, we reverted to our previous hosting setup as quickly as possible in order to minimize downtime.

How long was the system down?

Though we had scheduled a six hour maintenance window to allow for this contingency, the system was only down a total of one hour and fifteen minutes. This downtime was spread out over the period of 1:30am to 3:15am CDT. For approximately thirty minutes of what we are calling "downtime," the system was actually up and running with only a few pieces of functionality unresponsive.

What's the plan now?

For now, everything has been restored to full functionality exactly as it was before the upgrade attempt. It is still our plan to perform these upgrades, and we are tentatively rescheduling the upgrades for approximately one month from now, or sometime around mid-October. We will, of course, provide more specifics as the time approaches. 

Why wait so long?

One reason is so that we can fully investigate the issues we encountered before making a new attempt. The other reason is to avoid interfering with some additional upgrades we have planned to roll out on the system in the next few weeks. There are some exciting things in the works that we'll be sharing more about hopefully next week.

Is your current hosting infrastructure sufficient?

Yes. We have more than enough server headroom to accommodate our current traffic loads and even multiples of them. While we are very committed to accomplishing these new server upgrades, we do not consider them essential to our service at this point. That said, we are constantly monitoring and working to improve our system and we will respond as needed to any issues that may arise.

Thank you for your support

Thanks to all of you for your support. We received several positive emails in reply to yesterday's announcement of our upgrades and we appreciate that greatly. Rest assured, these scaling plans are still in place. We look forward to continuing to serve as your online partner.

As a part of our continuing effort to stay ahead of demand and provide our customers with the best possible experience, we are planning a major upgrade to our hosting infrastructure this weekend. This upgrade will add a handful of new web servers, file servers and caching servers to our already robust hosting environment. In addition, we will be enhancing our load balancing technology with the latest and most advanced equipment available. These changes are designed to increase our server redundancies and provide plenty of headroom for growth as we look to the future.

Unfortunately, massive improvements such as these require us to take our systems offline for a period of time while changes are made. We apologize for this inconvenience and we are scheduling our upgrades at the period of lowest possible system usage in order to minimize the impact of the upgrades.

Please be advised that our LightCMS, WaterCMS, SkyCMS and Icebrrg software systems will go offline for maintenance on Saturday morning, September 19th, 2009. Our maintenance window will be from 1:00am to 7:00am CDT (GMT -5). We are aiming for a total downtime of less than two hours, but we need to have the six hour window in place to plan for all contingencies. Throughout the upgrade period, we will be providing updates on our status page at status.elementfusion.com. 

Thank you for your patience and understanding. We are excited about these improvements which will not only improve things immediately but set us up for a host of performance improvements we have planned for the coming months. We look forward to continuing to serve as your online partner in the years ahead.

On Monday, March 30th, 2009, we have scheduled a maintenance window from 1:00am to 4:00am Central Daylight Time (GMT-5) for the servers powering WaterCMS, SkyCMS, LightCMS and Icebrrg. This window will be used to upgrade our server configuration to increase performance and accommodate the growing demand for our products.

Potential downtime

The system will go offline during the upgrade process which means that all of the products and the websites running on those products, will be inaccessible during the upgrade. While the window of potential downtime is three hours as outlined above, it is highly unlikely the downtime will last that long. In the past, similar upgrades have resulted in between 10-15 minutes of downtime and we anticipate similar results this time. The downtime could happen at any time during the maintenance window, but we anticipate it happening at the beginning of the window.

We apologize for the inconvenience caused by this downtime — especially to those of you outside of the Americas where the time of this upgrade is potentially more inconvenient. We have to make difficult decisions about when to schedule downtime based upon the usage patterns of all our customers, and we appreciate everyone's understanding in the process.

Thanks again and we look forward to continually providing you with the best possible performance and support.