FyfeWeb - Network Issue - Newcastle – Incident details

Network Issue - Newcastle

Resolved
Operational
Started about 1 year agoLasted about 3 hours

Affected

Network Services

Operational from 8:07 PM to 11:33 PM

IP Transit

Operational from 8:07 PM to 11:33 PM

Peering

Operational from 8:07 PM to 11:33 PM

Wavelengths & Data Centre Backhaul

Operational from 8:07 PM to 11:33 PM

DDoS Mitigation

Operational from 8:07 PM to 11:33 PM

Internal Systems

Operational from 8:07 PM to 11:33 PM

Updates
  • Resolved
    Resolved

    As you may be aware, FyfeWeb runs its own 100Gbps+ IP network, spanning 6 UK-based data centres. In March this year, we had a project on to bring in an additional DDoS mitigation service from Global Secure Layer, who operate a global IP Transit network with inline DDoS Protection. When the project was completed in March, everything ran solid for about 3 months until we started to see some issues in July/August. I've now escalated this to the CTO of GSL, and after a couple of hours discussing the issues, I know understand what's going on - and what's being done to fix it. Ultimately, GSL are building a new global network - with many sites now migrated over, London is ready to be migrated to the new network. I'm being told that we should see around ~1 minute of traffic impact whilst this is carried out, and after that, there should be no further issues. However, that being said, we are still looking to add further resilience to this service as soon as possible. Thank you for your patience so far.

  • Monitoring
    Monitoring

    The provider has stated that they've had a subsea cable failure which has caused the issue. Although the outage only lasted 11 minutes, we're wanting to improve reliability on this service. More information on this will follow in the coming days.

  • Identified
    Identified

    Service has been restored - we're finding out what happened with the provider at the moment.

  • Investigating
    Investigating

    We are currently investigating this incident. Updates to follow. It looks as if this is an issue with GSL, one of our DDoS-protected IP transit providers.