13 Jul 2017 21:38:01
[Broadband] Thursday evening major outage - Closed
Posted: 13 Jul 2017 18:56:13
Multiple circuits have disconnected and reconnected, staff are investigating
13 Jul 2017 19:00:34
Sessions seem to be repeatedly flapping rather than reconnecting - staff are investigating.
13 Jul 2017 20:05:47
We are still working on this, it's a rather nasty outage I'm afraid and is proving difficult to track down.
13 Jul 2017 20:17:42
Lines are re-connecting now...
13 Jul 2017 20:19:16
Apologies for the loss of graphs on a.gormless. We usually assume the worst and that out kit is the cause and tried a reboot of a FireBrick LNS. It did not help, but did clarify the real cause which was cisco switches. Sorry for the time it took to track this one down.
13 Jul 2017 20:19:27
Some line are being forced to reconnect so as to move them to the correct LNS, this will cause a logout/login for some customers...
13 Jul 2017 20:25:52
Lines are still connecting, not all are back, but the number of connected lines is increasing.
13 Jul 2017 20:29:55
We're doing some work which may cause some lines to go offline - we expect line to start reconnecting in 10 minutes time.
13 Jul 2017 20:34:35
We are rebooting stuff to try and find the issue. This is very unusual.
13 Jul 2017 20:42:56
Things are still not stable and lines are still dropping. We're needing to reboot some core network switches as part of our investigations and this is happening at the moment.
13 Jul 2017 20:52:29
Lines are reconnecting once more
13 Jul 2017 21:17:36
Looking stable.
13 Jul 2017 21:23:48
Most lines are back online now, if customers re still not online they a reboot of the router or modem may be required as the session may have got stuck inside the back haul network.
13 Jul 2017 21:41:38

We'll close this incident as lines have been stable for an hour. We'll update the post with further information as to the cause and any action we will be taking to help stop this type of incident from happening again.

We would like to thank our customers for their patience and support this evening. We had many customers in our IRC channel who were in good spirits and supportive to our staff whilst they worked on this incident.

14 Jul 2017 14:42:20
A similar problem occurred on Friday morning, this is covered on the following post: https://aastatus.net/2411
Closed 13 Jul 2017 21:38:01