FyfeWeb - Upstream Carrier Issue (GTT) – Incident details

This incident has been resolved. We have provided an RFO below:

We experienced a brief network service disruption today between 10:20 and 10:50 UTC, resulting in intermittent connectivity issues for some customers. Our initial assessment indicates the event aligned with a routing irregularity affecting connectivity between GTT their Corero-based protection platform in Benelux/Germany, although depending on the routing, other regions could have been impacted. At around 10:35, BGP sessions were shutdown towards GTT but prefixes took some time to withdraw, still causing connectivity issues for customers. At around 13:00, we were told the issue was resolved - so turned up our BGP sessions with them once again.

FyfeWeb operates a diverse multi-homed network, with a diverse connectivity mix such as LINX, LONAP, NTT and Hurricane Electric. At all times, traffic flows via all carriers to ensure the best routing for our customers, and if there's maintenance / BGP session drop on a carrier, traffic automatically re-routes. In this case, the fault occurred within the providers network, resulting in any traffic traversing the GTT network for FyfeWeb would have experienced an issue.

Due to the complex nature of the fault, our initial time spent troubleshooting was increased.

We will be monitoring for any further issues.

BGP sessions with GTT remain shutdown whilst we await a resolution. Traffic is currently flowing via other carriers.

We implemented a fix and are currently monitoring the results. The issue was related to a fault with one of our upstream carriers, GTT, which has now been taken out of service temporarily until they resolve the issue.