Some ADSL lines dropped at 12:52 - it does look like a BT problem, but we've not yet discovered any patters in the lines affected.
We'll update this shortly.
Update: We are sorry but it does actually apeear that this was all of our BT lines.
We will look into why our monitoing gave an inccorect percentage to begin with.
Most lines have logged back in again now. Lines are still logging in and we expect the remaining to log back in during the next few minutes.
Some lines (about half the number of the first drop) have just dropped again, we are seeing these reconnect though.
Some customers have logged into the wrong LNS, we are bouncing them back to the correct LNS now.
Title of this post has been changed to reflect the problem better
from: BT Blip affecting some Customers to: Blip Affecting BT Lines
All remaining lines were back online by 13:30
We're still looking on to what casued this, and will be reporting back.
Sorry for delay - we should have more details on this incident shortly.
Note that the outage duration may be much shorted than shown on the graph due to many lines switching to the backup LNS. The graphs for while on the backup LNS will not show (i.e. show as purple) even though on line.
Interestingly, having seen some BT issues before, this looks to be some sort of glitch in the LNS on the link to BT and not actually a BT fault. Whilst it is remotely possible this was caused by some external factor we think it is a random hardware glitch on this occasion.
The logs suggest this issue lasted 4 to 5 seconds at most but clearly it must have had a knock on effect that lasted the 10 seconds that are normally the timeout. This caused sessions to time out. The result was lines dropping and reconnecting to both main and backup LNS.
Action: From now on, all new sessions will have a 20 second timeout rather than 10 by default. This should help both the BT issues we have seen and anything like this happening.
Whilst lines started reconnecting immediately (within seconds) the time taken for some lines to connect we several minutes.
Action: We are working on multiple authentication servers not just for normal backup usage but for load sharing to ensure lines reconnect much faster in future. This should speed up recovery in the event of any issues like this.
While staff started to clear the sessions from the backup LNS in a controlled way an unexpected issue caused the backup LNS to fail. This affected a number of lines a second time and the reconnected to the main LNS.
Action: The exact cause is being investigated still and we hope to understand this better soon.
Further actions: We are very concerned at the apparent random issue on the BT link port, and we will be replacing the hardware some time on the next couple of weeks as part of routine maintenance.
Sorry for the inconvenience.