We're investigating line drops that started at around 4PM and is affecting various broadband lines.
We're seeing BT lines on three out of nineteen LNSs having high packetloss and PPP drops, we're still investigating.
We've just disabled one of our hostlinks to BT, this would have caused a PPP drop, and we're waiting to see if this improves service.
We're still seeing loss and are still investigating.
A large number of lines (Both BT and TalkTalk) would have just lost connection, and should reconnect shortly. Please to bear with us as we continue to investigate this fault.
We currently have some BT lines with packetloss and some TalkTalk lines which have PPP failures when trying to reconnect, we're still working on this.
Quick summary of where we are at the moment: We're having problems with our networking in our Equninx datacentre which has interconnects to BT and TalkTalk. We also BT and TalkTalk have interconnects in our Telehouse datacentre too - which are ok. The problem is, is that the Equinix side is half not working, which is causing these problems.
Having disabled our TalkTalk interconnect in Equinix, the affected TalkTalk lines are reconnecting now. We still have some BT lines with packet loss, which we are still investigating.
BT lines which were affected by packet loss are now looking better (ie normal!)
Service should be restored for everyone. We have moved all our BT and TalkTalk traffic away from our Equinix datacentre and service is all being routed via Telehouse (Usually traffic would be split roughly 50/50). This is a temporary fix whilst we investigate the actual cause of this evenings problems. The problem is to do with our network and not related to to datacentre or BT/TalkTalk problems specially, we will continue with our investigations.
We'll be investigating the actual cause of this incident - there were a number of different and separate events that occurred at nearly the same time (just after 4PM on Sunday) and we'll be investigating them to see if they can explain this incident. We'll post an update to this post with further explanation as and when we have one.