14 Jul 2017 16:48:39
[Broadband] Broadband outage - Closed
Posted: 14 Jul 2017 10:33:01
We've just seen BT and TT lines drop. We are investigating.
14 Jul 2017 10:57:12
This is similar to yesterday's problem. We have lost BGP connectivity to both our carriers. Over the past fee minutes the BGP sessions have been going up and down, meaning customers are logging in and then out again. Updates to follow shortly.
14 Jul 2017 11:16:46
Sessions are looking a bit more stable now... customer lines are still reconnecting
14 Jul 2017 11:56:08
We have about half DSL lines logged in, but the remaining half are struggling due to what is looking like a Layer 2 issue on our network.
14 Jul 2017 12:23:19
More DSL lines are now back up. If customers are still off, a reboot may help.
14 Jul 2017 12:35:51
A number of TT and BT lines just dropped. They are starting to reconnect now though.
14 Jul 2017 12:52:31
This is still on going - most DSL lines are back, some have been dropping but the network is still unstable. We are continuing to investigate this.
14 Jul 2017 12:55:50
We managed to get a lot of lines up after around an hour, during which there was a lot of flapping. The majority of the rest (some of the talk talk back haul lines) came up, and flapped a bit, at 12:00 and came back properly around 12:40. However, we are still trying to understand the issue, and still have some problems, and we suspect there may be more outages. The problem appears to be something at layer 2 but impacting several sites at once.
14 Jul 2017 14:41:23
We believe these outages are due to a partial failure of one of our core switches. We've moved most services away from this switch and are running diagnostics on it at the moment. We are not expecting these diagnostics to affect other services.
14 Jul 2017 17:00:40
The network has been stable for a good few hours now. One of our 40G Telehouse-to-Harbour Exchange interconnects has been taken down and some devices have been moved off of the suspect switch. We have further work to do in investigating the root cause of this and what we plan to do try to stop this from happening again. We do apologise to our customers for the disruption these two outages have caused and we work on trying to prevent this from happening again.
Started 14 Jul 2017 10:30:00
Closed 14 Jul 2017 16:48:39