FyfeWeb - Network issue Wolverhampton – Incident details

Network issue Wolverhampton

Resolved
Major outage
Started 4 days agoLasted about 4 hours

Affected

Data Centre Facilities

Partial outage from 3:16 PM to 3:25 PM, Operational from 3:16 PM to 3:25 PM, Major outage from 3:25 PM to 3:47 PM, Operational from 3:47 PM to 5:10 PM, Major outage from 5:10 PM to 7:18 PM

DC1, West Midlands

Operational from 3:16 PM to 3:25 PM, Major outage from 3:25 PM to 3:47 PM, Operational from 3:47 PM to 5:10 PM, Major outage from 5:10 PM to 7:18 PM

DC2, West Midlands

Partial outage from 3:16 PM to 3:25 PM, Major outage from 3:25 PM to 3:47 PM, Operational from 3:47 PM to 5:10 PM, Major outage from 5:10 PM to 7:18 PM

Hosted Virtual Machines

Operational from 3:16 PM to 5:10 PM, Partial outage from 5:10 PM to 7:18 PM

VPS Hosting

Operational from 3:16 PM to 5:10 PM, Major outage from 5:10 PM to 7:18 PM

Web Hosting

Operational from 3:16 PM to 5:10 PM, Partial outage from 5:10 PM to 7:18 PM

Updates
  • Resolved
    Resolved

    We have now seen full service recovery after swapping the affected device, this required a full reconfiguration due to various unexpected issues encountered along the way, which delayed the recovery process.

    I would like to take this opportunity to apologise to you for any inconvenience caused this evening by the outage, as well as thanking you for your patience and support. These issues fall well below the standard of high quality service we want to provide, and for that we are truly sorry. Rest assured, throughout the incident we were working as quickly as possible to restore service.

    We are continually investing in our network to increase reliability throughout, and will continue doing so. As always, if you have any issues please get in touch.

  • Update
    Update

    We have an ETA of around 60-90 minutes for full recovery, some services such as cPanel etc are recovered.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    Investigating

    The issue has re-occurred. We are going to perform a device swap, the new device is currently being prepared.

  • Update
    Update

    We are still monitoring the device for any further issues.

  • Monitoring
    Monitoring

    The device has been rebooted and is now back online, apologies for any inconvenience caused. We are still monitoring for further issues, a replacement device has been sourced and will be swapped out if the issue occurs again.

    We're currently checking the logs on the device but at first glance it appears as if the device crashed.

  • Update
    Update

    We have identified an issue with core1-dc2.wlv - one of our core routers. We are still working on a fix.

  • Identified
    Identified
    We are continuing to work on a fix for this incident.