Seems a major problem affecting all broadband lines - service si restoring automatically, but taking several minutes to get people back on line.
Lines are coming back but much more slowly than we would expect
Both LNSs are seeing people connect constantly and then disconnect, which makes little sense. We are still investigating the issue now.
We have reset equipment and switched LNSs, which has affected lines that were not previously affected. We are trying to see a pattern here.
Logs confirm that at 2am. suddenly things started working a lot better and many more lines coming back. Still investigating what is going on here.
We think, whatever it was, has magically fixed itself. We are going to clear all sessions to one LNS now, so a blip again for some people.
We are not entirely sure of the cause of the original problem, but it is such a long time since we have had a major issue like this it appears that we currently have a problem with the speed with which our systems can recover.
It seems our RADIUS server is struggling and once overloaded it is too slow to handle the connections and so the connections timeout and re-try causing more load and more delay. The result is that while everyone is trying to connect, nobody can, and it basically took well over an hour for lines to connect.
More investigation to follow.
I am closing the mahor incident on this now - but we are investigating the cause and the slow reconnect.