Unscheduled downtime - power issue at primary data centre

Date: 
Thu, 12/01/2017 - 10:40

We are currently experiencing an issue with connectivity at our primary data centre resulting in the unavailability of customer servers.

UPDATE 10:52 - We have been informed that there is a power issue at our primary data centre which explains the unscheduled downtime. Engineers at the data centre are looking into this as a matter of urgency and we'll update this page as soon as we have further information.

UPDATE 11:12 - The power has just been restored at the data centre although our equipment hasn't yet come back online so we are continuing to work with our data centre provider to resolve the situation.

UPDATE 11:21 - The data centre provider is working to bring services back online and we have a support ticket open with them to track the progress of this. We will post further details on the issue and the cause in due course.

UPDATE 11:45 - Power has been restored to our equipment but the firewalls haven't yet come back online. We are working with remote hands at the data centre to diagnose the issue and attempt to bring services back online.

UPDATE 11:59 - We are setting up console access to the firewalls in order to diagnose the problem and attempt to bring them back online.

UPDATE 12:14 - We have console access to the firewalls and our network engineers are working to determine why the network is not functioning correctly after the earlier power issue.

UPDATE 12:31 - We have remote access to the firewalls and we are continuing to diagnose the networking issue.

UPDATE 12:41 - We are continuing to determine which parts of our infrastructure are running correctly thereby eliminating possible issues. The virtualisation platform is partially working and some virtual machines are running although still inaccessible.

UPDATE 12:48 - Customer servers are now being restarted and the infrastructure appears to be operating correctly. It may take a while for all machines to come back online and we will work through all servers to ensure they have booted correctly and are working. If you are still having issues after 13:30 please raise a ticket with us and we will investigate further.

UPDATE 13:08 - All customer servers are now back online with the exception of spam filtering which we are working on resolving now. Emails will be queued until this is resolved. Please accept our apologies for the extended downtime caused by the power issue at our data centre. We will investigate the cause and issue a full report in due course. If you are still having issues with your service please raise a support ticket or call us on 0333 3441270.

UPDATE 13:43 - Spam filtering is now working again so all customer services should be restored.