Multiple circuits have disconnected and reconnected, staff are investigating
Sessions seem to be repeatedly flapping rather than reconnecting - staff are investigating.
We are still working on this, it's a rather nasty outage I'm afraid and is proving difficult to track down.
Lines are re-connecting now...
Apologies for the loss of graphs on a.gormless. We usually assume the worst and that out kit is the cause and tried a reboot of a FireBrick LNS. It did not help, but did clarify the real cause which was cisco switches. Sorry for the time it took to track this one down.
Some line are being forced to reconnect so as to move them to the correct LNS, this will cause a logout/login for some customers...
Lines are still connecting, not all are back, but the number of connected lines is increasing.
We're doing some work which may cause some lines to go offline - we expect line to start reconnecting in 10 minutes time.
We are rebooting stuff to try and find the issue. This is very unusual.
Things are still not stable and lines are still dropping. We're needing to reboot some core network switches as part of our investigations and this is happening at the moment.
Lines are reconnecting once more
Looking stable.
Most lines are back online now, if customers re still not online they a reboot of the router or modem may be required as the session may have got stuck inside the back haul network.
We'll close this incident as lines have been stable for an hour. We'll update the post with further information as to the cause and any action we will be taking to help stop this type of incident from happening again.
We would like to thank our customers for their patience and support this evening. We had many customers in our IRC channel who were in good spirits and supportive to our staff whilst they worked on this incident.
A similar problem occurred on Friday morning, this is covered on the following post: https://aastatus.net/2411