FyfeWeb - Network Interruption - West Midlands DC1 – Incident details

All systems operational

Network Interruption - West Midlands DC1

Resolved
Major outage
Started 3 months agoLasted about 2 hours

Affected

Data Centres

Major outage from 2:24 PM to 2:40 PM, Operational from 2:40 PM to 4:41 PM

DC1, West Midlands

Major outage from 2:24 PM to 2:40 PM, Operational from 2:40 PM to 4:41 PM

Web Hosting

Major outage from 2:24 PM to 2:40 PM, Operational from 2:40 PM to 4:41 PM

cPanel Hosting

Major outage from 2:24 PM to 2:40 PM, Operational from 2:40 PM to 4:41 PM

Private Cloud & Dedicated Servers

Operational from 2:24 PM to 2:33 PM, Major outage from 2:33 PM to 2:40 PM, Operational from 2:40 PM to 4:41 PM

Bare Metal (Dedicated Servers)

Operational from 2:24 PM to 2:33 PM, Major outage from 2:33 PM to 2:40 PM, Operational from 2:40 PM to 4:41 PM

Updates
  • Resolved
    Resolved

    Services have remained stable since the previous update.

    Separate to this incident, this router is due to be upgraded with newer routing-engines. This work will now be prioritised and will be scheduled as a new maintenance window, to be announced in due course.

  • Monitoring
    Update

    Root Cause Analysis

    We identified an issue with the primary routing-engine in core1-dc1, the core router in DC1.

    We have manually switched mastership to the backup routing-engine without graceful-switchover which has caused all protocols to flap and re-converge on this router.

    Services have now recovered and the router is stable, however if you are still experiencing issues please get in touch.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Update

    We are still investigating the incident - the issue appears to be related to connectivity with an upstream provider within this facility from our core router. We are currently in contact with the provider in question. There is no ETA at this time, but we're trying to get this resovled as quickly as possible.

  • Investigating
    Investigating

    We are currently investigating a network issue in our West Midlands data centre - this is causing issues with some managed dedicated servers, colocation and cPanel Web Hosting.