FyfeWeb - Network Issue - West Midlands DC2 – Incident details

All systems operational

Network Issue - West Midlands DC2

Resolved
Operational
Started 8 months agoLasted 4 days

Affected

Network Services

Operational from 10:05 AM to 9:59 PM

IP Transit

Operational from 10:05 AM to 9:59 PM

Data Centres

Operational from 10:05 AM to 9:59 PM

DC2, West Midlands

Operational from 10:05 AM to 9:59 PM

VPS (Public Cloud)

Operational from 10:05 AM to 9:59 PM

Cloud01 Cluster

Operational from 10:05 AM to 9:59 PM

Updates
  • Resolved
    Resolved

    This incident is now resolved and the wavelengths in question have been reintroduced to our network.

  • Monitoring
    Update

    Service has remained stable for some time. Initial investigations indicate an issue with 2 wavelengths from Wolverhampton to London, causing a number of protocols to flap, interrupting traffic. We took action to disable those wavelengths which stabilised traffic. We will be reintroducing those wavelengths during a planned window late one evening to be able to monitor traffic as they're reintroduced.

  • Monitoring
    Monitoring

    After around 5 minutes of service interruption in total, we are seeing services return to normal. Please raise a ticket if you're experiencing any issues.

  • Identified
    Identified

    We are still seeing services flap, this has been raised to director-level with the upstream provider.

    They have confirmed they're currently investigating the root cause of the incident.

  • Investigating
    Investigating

    We're currently investigating an issue with packet loss in one of our data centres. At the moment, this appears to be isolated to an issue with an upstream provider and not on our network. We'll keep this incident updated throughout.

    This will cause issues with some VPS services, as well as some customer dedicated servers and colocation.