Private Cloud & Dedicated Servers
This incident has been resolved.
We've implemented a fix and are currently monitoring the results.
We are currently experiencing a network issue affecting our Newcastle Data Centre, due to an upstream provider issue. Our team is actively investigating the situation, and further updates will follow as we work to resolve the issue with them.
Location: Newcastle Data Centre
Issue: Network disruption
Cause: Upstream provider connectivity issues
Please be aware that during this time, services at the Newcastle Data Centre may have intermittent connectivity problems. We apologise for any inconvenience and appreciate your patience as we work to fix the issue.
Thank you for your understanding.
- CompletedAugust 28, 2023 at 10:17 AMCompletedAugust 28, 2023 at 10:17 AM
The London network has now been fully rebuilt by the provider, and is now part of the new global network. Going forward, we don’t expect to see any further issues - but if we do, we’ll instigate our backup plans.
- UpdateAugust 26, 2023 at 4:16 PMIn progressAugust 26, 2023 at 4:16 PM
The provider has stated that they are still working on this, and are having to rebuild quite a lot of their network in London due to discovering some underlying issues. No impact is expected at the moment, but services are "at-risk" whilst the work is carried out.
- UpdateAugust 26, 2023 at 1:19 PMIn progressAugust 26, 2023 at 1:19 PM
Maintenance is still in progress with this. Traffic has been re-routed via alternative paths, so no impact has been seen. We're extending this window by another hour, just to ensure we have enough time to get the "all clear"
- In progressAugust 26, 2023 at 11:30 AMIn progressAugust 26, 2023 at 11:30 AM
Maintenance is now in progress
- PlannedAugust 26, 2023 at 11:30 AMPlannedAugust 26, 2023 at 11:30 AM
Our transit provider, Global Secure Layer, had network hardware shipped to London earlier this month which was found to be dead on arrival. The replacement units have arrived and completed all tests. These will be installed to complete the required upgrades for their EMEA network.
This maintenance window was created by them due to the issues they saw the other night - any occasional previous ~10-minute drops should be resolved going forward, once this work has been completed. FyfeWeb is in direct contact with the CEO/CTO at this time, ensuring we're constantly kept up to date with what's going on.
They expect outages to last less than 10 minutes during this maintenance window.
As you may be aware, FyfeWeb runs its own 100Gbps+ IP network, spanning 6 UK-based data centres. In March this year, we had a project on to bring in an additional DDoS mitigation service from Global Secure Layer, who operate a global IP Transit network with inline DDoS Protection. When the project was completed in March, everything ran solid for about 3 months until we started to see some issues in July/August. I've now escalated this to the CTO of GSL, and after a couple of hours discussing the issues, I know understand what's going on - and what's being done to fix it. Ultimately, GSL are building a new global network - with many sites now migrated over, London is ready to be migrated to the new network. I'm being told that we should see around ~1 minute of traffic impact whilst this is carried out, and after that, there should be no further issues. However, that being said, we are still looking to add further resilience to this service as soon as possible. Thank you for your patience so far.
The provider has stated that they've had a subsea cable failure which has caused the issue. Although the outage only lasted 11 minutes, we're wanting to improve reliability on this service. More information on this will follow in the coming days.
Service has been restored - we're finding out what happened with the provider at the moment.
We are currently investigating this incident. Updates to follow. It looks as if this is an issue with GSL, one of our DDoS-protected IP transit providers.
No notices reported this month