We've just seen BT and TT lines drop. We are investigating.
This is similar to yesterday's problem. We have lost BGP connectivity to both our carriers. Over the past fee minutes the BGP sessions have been going up and down, meaning customers are logging in and then out again. Updates to follow shortly.
Sessions are looking a bit more stable now... customer lines are still reconnecting
We have about half DSL lines logged in, but the remaining half are struggling due to what is looking like a Layer 2 issue on our network.
More DSL lines are now back up. If customers are still off, a reboot may help.
A number of TT and BT lines just dropped. They are starting to reconnect now though.
This is still on going - most DSL lines are back, some have been dropping but the network is still unstable. We are continuing to investigate this.
We managed to get a lot of lines up after around an hour, during which there was a lot of flapping. The majority of the rest (some of the talk talk back haul lines) came up, and flapped a bit, at 12:00 and came back properly around 12:40. However, we are still trying to understand the issue, and still have some problems, and we suspect there may be more outages. The problem appears to be something at layer 2 but impacting several sites at once.
We believe these outages are due to a partial failure of one of our core switches. We've moved most services away from this switch and are running diagnostics on it at the moment. We are not expecting these diagnostics to affect other services.
The network has been stable for a good few hours now. One of our 40G Telehouse-to-Harbour Exchange interconnects has been taken down and some devices have been moved off of the suspect switch. We have further work to do in investigating the root cause of this and what we plan to do try to stop this from happening again. We do apologise to our customers for the disruption these two outages have caused and we work on trying to prevent this from happening again.