Resolved
Network Issue - West Midlands DC2

Started
February 12, 2024 at 10:05 AM
Status
Resolved after 4 days

Impact

Operational
Affected
Network Services
IP Transit
Data Centres
DC2, West Midlands
VPS (Public Cloud)
Cloud01 Cluster
Cloud02 Cluster
Cloud03 Cluster
Cloud04 Cluster
Private Cloud & Dedicated Servers
Bare Metal (Dedicated Servers)
Hosted Virtual Machines & VDS
  • Resolved
    Resolved

    This incident is now resolved and the wavelengths in question have been reintroduced to our network.

  • Monitoring
    Update

    Service has remained stable for some time. Initial investigations indicate an issue with 2 wavelengths from Wolverhampton to London, causing a number of protocols to flap, interrupting traffic. We took action to disable those wavelengths which stabilised traffic. We will be reintroducing those wavelengths during a planned window late one evening to be able to monitor traffic as they're reintroduced.

  • Monitoring
    Monitoring

    After around 5 minutes of service interruption in total, we are seeing services return to normal. Please raise a ticket if you're experiencing any issues.

  • Identified
    Identified

    We are still seeing services flap, this has been raised to director-level with the upstream provider.

    They have confirmed they're currently investigating the root cause of the incident.

  • Investigating
    Investigating

    We're currently investigating an issue with packet loss in one of our data centres. At the moment, this appears to be isolated to an issue with an upstream provider and not on our network. We'll keep this incident updated throughout.

    This will cause issues with some VPS services, as well as some customer dedicated servers and colocation.