We’ve had a xen host crash and a number of Brightboxes have been affected. We’re currently investigating.
UPDATE: The affected Brightboxes are now coming back online. All should be back within 15minutes.
UPDATE: All affected Brightboxes are now back online. We’re investigating the cause of the failure.
The shared MySQL cluster just had an interruption of service due to a backup server coming online and taking over an IP address. The interruption should have been very short, but some sites still appear to be having problems. We’re investigating.
UPDATE: The short outage caused a few clients to reconnect to the MySQL server, leaving their old connections idle. This increased the number of concurrent connections beyond the maximum allowed, preventing new connections. This has now been rectified and customer Rails apps should reconnect successfully on the next request. Apologies for this outage.
One of our Xen hardware nodes just experienced a failure and has been restarted. 9 customer virtual machines were affected and are now confirmed back online.
One of our Xen hardware nodes experienced a failure and has been restarted. 7 customer virtual machines were affected and are being booted now.
UPDATE: All machines are confirmed back online now.
Between 05:00 - 07:00 on Wednesday 19th November, one of our upstream providers will be upgrading their core switch network.
During this period, all routers and firewalls will be temporarily moved over to their corresponding backup device, and then returned back. This will cause one or more brief periods of Internet connectivity outage for Brightboxes on the 77.240.8.0/24 IP range, which should be less than one minute in each case.
One of our Xen machines failed in the last half an hour. 12 customer Brightboxes on it were affected but are now all confirmed back online.
Due to a fault, the Brightboxes were not immediately auto-booted which caused the outage to be unnecessarily extended. We’re investigating how to ensure this fault does not happen again and how to improve our monitoring systems to detect problems like this sooner in future.
We have some maintenance planned on the support sites on Tuesday 21st October between 21:00 and 22:00 BST. The support sites will be unavailable for short periods during this window.
The support ticket system, the wiki, the blog and the gems and apt repositories will be affected by this work.
UPDATE: This work was completed successfully with minimal downtime. All support sites are now stable again.
No current problems or planned maintenance
