Unscheduled Downtime

We have an issue with one of our XenServer hosts which is affecting some customer machines. We are working on the issue and customer virtual machines should be back online shortly.

We apologise for any inconvenience caused.

Update 17:15
All customer virtual machines are now booting and we are looking into the cause of the issue.

Update 23:15

We are aware of an issue presently affecting connectivity to our Hosted Exchange services. We are looking into the issue and will post further updates shortly.

Update 11:21: Exchange services are now back online and we are looking into the cause of the issue.

Cisco have confirmed there is a hardware fault with our Cisco IronPort anti-spam appliance and they will be replacing this on Tuesday.

In the meantime we have set up a temporary spam filtering appliance using a virtual server. This will still filter spam but may not be as effective and there will be no quarantines (you will receive suspected spam emails with [*** SPAM ***] prepended to the subject).

Emails should now be being delivered correctly and any emails sent during the time the service was unavailable should be delivered eventually when the sending server retries.

We are currently experiencing an issue with out Cisco Ironport Anti-Spam appliance and as a result no email is being delivered that routes through mx01.lhc.uk.networkeq.net. No email will be lost, but some messages may get returned to sender with an undeliverable error.

We have raised a case with Cisco and are working towards returning to normal service as soon as possible.

We apologise for any inconveniece this may cause.

We have re-installed the Junos software on one of our firewalls and this is now back online. This took longer than expected due to the fact the partition table on the device was corrupted.

We have applied a backup configuration to the firewall so most customer services are now working but there are a handful of customers whose services will still be unavailable. We are working as quickly as possible to restore service to these customers.

We are currently experiencing an unexpected issue with one of our firewalls in Docklands, this has caused a loss of connectivity to all customer services. We are currently working on bringing the firewall back online and we will then investigate the root cause.

Please accept our apologies for any inconvenience.

Update 17:10: We have turned off the firewall in the cluster that we believe to be faulty and are currently working on bringing the other firewall back online. This issue is being dealt with at the highest priority, thank you for your patience.

The issue affecting our switches has now been resolved. Both switches are now back online (and have also been upgraded to the recommended software version).

Customer virtual machines will need to be rebooted before they will be available again (as connectivity to the storage was lost during the switch issue). This is happening now and all servers will be back online shortly.

Please accept our apologies for any inconvenience caused.

We are currently experiencing an issue with one switch in our virtual chassis which has caused our storage network to become unavailable, this is having a knock on effect on all services.

We are currently working to get the switch back online and resume normal service. We will post updates here as we know more.

Please accept our apologies for any inconvenience.

One of our XenServer hosts has become unresponsive and we need to restart the host as a result. This will cause a disruption to some customer virtual machines.

We apologise for any inconvenience caused.

Update 19:27

This issue is now resolved and all affected virtual machines are now coming back online.

We are still investigating the cause of the issue but please accept our apologies for any inconvenience caused.