FyfeWeb - Core Router Issue – Incident details

All systems operational

Core Router Issue

Resolved
Degraded performance
Started about 1 year agoLasted 41 hours 48 minutes

Affected

Data Centre Facilities

Degraded performance from 4:11 AM to 11:30 PM, Operational from 11:30 PM to 9:59 PM

DC1, West Midlands

Degraded performance from 4:11 AM to 11:30 PM, Operational from 11:30 PM to 9:59 PM

DC2, West Midlands

Degraded performance from 4:11 AM to 11:30 PM, Operational from 11:30 PM to 9:59 PM

Updates
  • Resolved
    UTC
    Resolved

    This incident has been resolved, we have observed stability since performing the swap.

  • Monitoring
    UTC
    Monitoring

    Services are now back online and we are monitoring for any further issues.

  • Update
    UTC
    Update

    The new device has been installed and is booting.

  • Update
    UTC
    Update

    The faulty device is now being replaced. Services will drop.

  • Update
    UTC
    Update

    We are now preparing the new device, please stand by for further updates.

  • Update
    UTC
    Update

    The work has been delayed due to issues with travel - the engineer is 30 minutes away from the data centre and will perform the device swap shortly. The device is currently still flapping up and down.

  • Update
    UTC
    Update

    A network engineer has been dispatched to the data centre and will perform remedial works between 10 PM and 11 PM this evening.

  • Update
    UTC
    Update

    We have located a replacement device in stores and will organise a device replacement this evening. This will cause a drop to all services connected to this router, a time will be confirmed within the next 6-7 hours.

  • Identified
    UTC
    Identified

    We have identified a potential hardware issue with one of our core routers in Wolverhampton, core0-dc2.wlv where the device is randomly rebooting due to an apparent hardware problem. We're currently organising the installation of a new replacement device and this will be swapped out over the weekend due to the reliability issues of the current device.

    We apologise for any inconvenience this is causing you.