FyfeWeb - Connectivity issues at Reliance House, Newcastle. – Incident details

All systems operational

Connectivity issues at Reliance House, Newcastle.

Resolved
Partial outage
Started about 2 years agoLasted 10 days

Affected

Data Centre Facilities

Major outage from 7:51 PM to 8:08 PM, Operational from 8:08 PM to 5:02 PM, Partial outage from 5:02 PM to 6:24 PM, Under maintenance from 6:24 PM to 7:27 PM

Reliance House, Newcastle

Major outage from 7:51 PM to 8:08 PM, Operational from 8:08 PM to 5:02 PM, Partial outage from 5:02 PM to 6:24 PM, Under maintenance from 6:24 PM to 7:27 PM

Dedicated Servers

Partial outage from 7:51 PM to 8:08 PM, Operational from 8:08 PM to 5:02 PM, Partial outage from 5:02 PM to 6:24 PM, Under maintenance from 6:24 PM to 7:27 PM

Network Services

Partial outage from 7:51 PM to 8:08 PM, Operational from 8:08 PM to 5:02 PM, Partial outage from 5:02 PM to 6:24 PM, Under maintenance from 6:24 PM to 7:27 PM

Internal Infrastructure

Operational from 7:51 PM to 7:27 PM

Updates
  • Resolved
    UTC
    Resolved

    We have completed the replacement and normal service has since resumed.

    Apologies for any inconvenience caused this evening.

  • Monitoring
    UTC
    Monitoring

    We have identified a faulty optic within the FPC of the device, instead of swapping the whole device we're going to swap the optic and monitor from there.

  • Update
    UTC
    Update

    We are continuing to work on a fix for this incident - we're on-site and building a replacement device, there will be another brief 5 min drop when we replace the device.

  • Identified
    UTC
    Identified

    We have seen a repeat of this incident - we are going to replace this device now. Only directly connected customers to this device will be impacted at the moment.

  • Resolved
    UTC
    Resolved

    We haven't seen any further issues since one of our core routers rebooted unexpectedly.

    We believe the issue was related to a bug within the software running on the device, triggering an unexpected reboot.

    We will be performing further emergency maintenance in the coming days in order to ensure stability across services connected to this device.

  • Monitoring
    UTC
    Monitoring

    It appears that an unexpected reboot of a core device is the cause of approximately 5 mins of downtime.

    We’re monitoring this closely and will plan maintenance imminently to ensure that further issues are prevented.

  • Investigating
    UTC
    Investigating

    We are aware of a incident affecting connectivity in our Newcastle Reliance House Facility.