13 Jul 21:38:01
[Broadband] Thursday evening major outage - Closed
Details
Posted: 13 Jul 18:56:13
Multiple circuits have disconnected and reconnected, staff are investigating
Update
13 Jul 19:00:34
Sessions seem to be repeatedly flapping rather than reconnecting - staff are investigating.
Update
13 Jul 20:05:47
We are still working on this, it's a rather nasty outage I'm afraid and is proving difficult to track down.
Update
13 Jul 20:17:42
Lines are re-connecting now...
Update
13 Jul 20:19:16
Apologies for the loss of graphs on a.gormless. We usually assume the worst and that out kit is the cause and tried a reboot of a FireBrick LNS. It did not help, but did clarify the real cause which was cisco switches. Sorry for the time it took to track this one down.
Update
13 Jul 20:19:27
Some line are being forced to reconnect so as to move them to the correct LNS, this will cause a logout/login for some customers...
Update
13 Jul 20:25:52
Lines are still connecting, not all are back, but the number of connected lines is increasing.
Update
13 Jul 20:29:55
We're doing some work which may cause some lines to go offline - we expect line to start reconnecting in 10 minutes time.
Update
13 Jul 20:34:35
We are rebooting stuff to try and find the issue. This is very unusual.
Update
13 Jul 20:42:56
Things are still not stable and lines are still dropping. We're needing to reboot some core network switches as part of our investigations and this is happening at the moment.
Update
13 Jul 20:52:29
Lines are reconnecting once more
Update
13 Jul 21:17:36
Looking stable.
Update
13 Jul 21:23:48
Most lines are back online now, if customers re still not online they a reboot of the router or modem may be required as the session may have got stuck inside the back haul network.
Update
13 Jul 21:41:38

We'll close this incident as lines have been stable for an hour. We'll update the post with further information as to the cause and any action we will be taking to help stop this type of incident from happening again.

We would like to thank our customers for their patience and support this evening. We had many customers in our IRC channel who were in good spirits and supportive to our staff whilst they worked on this incident.

Update
14 Jul 14:42:20
A similar problem occurred on Friday morning, this is covered on the following post: https://aastatus.net/2411
Closed 13 Jul 21:38:01