Some BT/TT lines dropping and packetloss since 4PM
MAJOR Closed Broadband
STATUS
Closed
AFFECTED
Broadband
STARTED
Sep 06, 04:00 PM (15½ days ago)
CLOSED
Sep 06, 08:00 PM (15½ days ago)
REFERENCE
37182 / AA37182
INFORMATION
  • INITIAL
    15½ days ago by Andrew

    We're investigating line drops that started at around 4PM and is affecting various broadband lines.

  • UPDATE
    15½ days ago by Andrew

    We're seeing BT lines on three out of nineteen LNSs having high packetloss and PPP drops, we're still investigating.

  • UPDATE
    15½ days ago by Andrew

    We've just disabled one of our hostlinks to BT, this would have caused a PPP drop, and we're waiting to see if this improves service.

  • UPDATE
    15½ days ago by Andrew

    We're still seeing loss and are still investigating.

  • UPDATE
    15½ days ago by Andrew

    A large number of lines (Both BT and TalkTalk) would have just lost connection, and should reconnect shortly. Please to bear with us as we continue to investigate this fault.

  • UPDATE
    15½ days ago by Andrew

    We currently have some BT lines with packetloss and some TalkTalk lines which have PPP failures when trying to reconnect, we're still working on this.

  • UPDATE
    15½ days ago by Andrew

    Quick summary of where we are at the moment: We're having problems with our networking in our Equninx datacentre which has interconnects to BT and TalkTalk. We also BT and TalkTalk have interconnects in our Telehouse datacentre too - which are ok. The problem is, is that the Equinix side is half not working, which is causing these problems.

  • UPDATE
    15½ days ago by Andrew

    Having disabled our TalkTalk interconnect in Equinix, the affected TalkTalk lines are reconnecting now. We still have some BT lines with packet loss, which we are still investigating.

  • UPDATE
    15½ days ago by Andrew

    BT lines which were affected by packet loss are now looking better (ie normal!)

  • UPDATE
    15½ days ago by Andrew

    Service should be restored for everyone. We have moved all our BT and TalkTalk traffic away from our Equinix datacentre and service is all being routed via Telehouse (Usually traffic would be split roughly 50/50). This is a temporary fix whilst we investigate the actual cause of this evenings problems. The problem is to do with our network and not related to to datacentre or BT/TalkTalk problems specially, we will continue with our investigations.

  • RESOLUTION
    15½ days ago by Andrew

    We'll be investigating the actual cause of this incident - there were a number of different and separate events that occurred at nearly the same time (just after 4PM on Sunday) and we'll be investigating them to see if they can explain this incident. We'll post an update to this post with further explanation as and when we have one.

  • Closed