Drop of TalkTalk lines
MAJOR Closed Broadband
STATUS
Closed
AFFECTED
Broadband
STARTED
Sep 15, 01:53 AM (1½ months ago)
CLOSED
Sep 15, 04:00 PM (1½ months ago)
REFERENCE
37299 / AA37299
INFORMATION
  • INITIAL
    1½ months ago by Andrew

    We're investigating the loss of a number of TalkTalk connected lines. Updates to follow shortly.

  • UPDATE
    1½ months ago by Andrew

    Lines are reconnecting...

  • UPDATE
    1½ months ago by Andrew

    Whilst many lines have reconnected successfully, some are failing to stay online, we're still investigating.

  • UPDATE
    1½ months ago by Andrew

    The initial cause of this outage was planned work on our TalkTalk interconnect in our Telehouse datacentre by TalkTalk. This would normally be OK, except that our second interconnect in a different datacentre, Equinix, was taken out of service by ourselves last week due to a separate incident: https://aastatus.net/37182 (aggggghhhhhhhhhhhhhhh)

  • UPDATE
    1½ months ago by Andrew

    Not all lines have reconnected, we're expecting those that are not back yet should reconnect soon.

  • UPDATE
    1½ months ago by Andrew

    A (smaller) number of lines are still failing to reconnect. We're investigating this - at this point we're not sure if this is related to the planned work TalkTalk are doing at the moment or if it's something separate.

  • UPDATE
    1½ months ago by Andrew

    There is still the smaller number of TalkTalk lines that have not been connect - we're still investigating the cause of this. The TalkTalk work is still happening and we'd expect these lines to reconnect once the work has completed (by 6AM) - but in the mean time we're still looking at what is causing these lines not to reconnect.

  • UPDATE
    1½ months ago by Andrew

    We're expecting TalkTalk to finish their work any moment, our interface to them is still down so they have not finished just yet. Currently most lines are working, there are about 100 TalkTalk lines which are still down.

  • UPDATE
    1½ months ago by Andrew

    TalkTalk's work is over running and has not completed. We're taking some steps to re-route the affected lines still down in an effort to restore their service.

  • UPDATE
    1½ months ago by Andrew

    The ~100 affected lines are not connecting successfully.

  • UPDATE
    1½ months ago by Andrew

    TalkTalk's work in Telehouse datacentre is still ongoing, so our traffic is going over our interconnect to TalkTalk in Equinix datacentre.

  • UPDATE
    1½ months ago by Andrew

    The work we carried out to re-route the remaining ~100 lines involved us making changes to the LACP configuration of some of our LNSs. Tonight's problem has highlighted a problem with one of our core switches which was also partly involved in the problems last week. We'll be planning some out of hours work ourselves in the coming days to diagnose this further.

  • UPDATE
    1½ months ago by Andrew

    The remaining ~100 customers are back online, but we'll keep this incident open for the time being. If customers are not online, then please try rebooting your router so as to force a fresh connection, and then get in touch with us.

  • UPDATE
    1½ months ago by Andrew

    TalkTalk have raised an incident regarding their work over running - https://aastatus.net/37301 however, our services are working over our second interconnect so this is not affecting our services.

  • UPDATE
    1½ months ago by Andrew

    There are a handful of customers who still have lines down, we're looking at these on an individual basis - do get in touch if this affects you.

  • UPDATE
    1½ months ago by Andrew

    The lines that are still offline are rather odd - most seem to not be relayed on to us by TalkTalk - so we never see the connection coming in to us. TalkTalk are still in the process of rolling back their planned work changes: https://aastatus.net/37301 Once that work has been completed we're expecting these remaining lines to reconnect - in the meantime we're still investigating why they are not connecting now!

  • UPDATE
    1½ months ago by Andrew

    TalkTalk have finished their planned work, and our Telehouse interconnect is back up and running. We still have a small number of lines that are failing to connect - in these cases, either TalkTalk are not seeing the end user router attempt to log in, or TalkTalk are not passing the connection on to us. We are compiling a list of these affected lines, so do get in touch if you have not already. We are then passing examples on to TalkTalk to investigate between is.

  • UPDATE
    1½ months ago by Andrew

    In some of these cases, powering off the router/modem for 20 minutes has resulted in the line reconnecting successfully.

  • UPDATE
    1½ months ago by Andrew

    We have seen some lines reconnect that have been off - maybe not all lines are back, but many have reconnected. There are probably only a small handful of lines that are off still - but do get in touch if you are still off as we are investigating these on an individual basis.

  • RESOLUTION
    1½ months ago by Andrew

    Summary and further work

    Here is a brief summary of this incident and what we are planning to do to improve matters.

    1. 01:12 Whilst we were running on a single interconnect to TalkTalk due to a previous incident, TalkTalk carried out planned work on the working interconnect causing all TalkTalk lines to drop. (Unfortunately is wasn't made clear to us that this work was happening, otherwise we would have been more prepared and this incident wouldn't have happened)
    2. 01:52 We bought our 2nd interconnect back in to service and about 70% of lines reconnected
    3. 01:52 about 30% of lines still had problems staying connected
    4. 05:37 After lots of investigation which led to use removing some internal LACP links and another 17.5% of lines connected.
    5. 06:16 Further work our side restored about 10% of lines, leaving a further 2.5% offline
    6. 12:00 TalkTalk's Planned work overran and their work was reverted and completed.
    7. Early afternoon: The remaining 2.5% of lines reconnected eventually - these were most problematic and seemed to have either stuck or blocked sessions within the TalkTalk network - a problem probably a result of TalkTalk's planned work and was resolved by TalkTalk clearing sessions manually after their planned work.
    We have scheduled the early hours of Thursday 17th September to investigate further one of our core network switches that has been the cause of some of these problems today and last week: https://aastatus.net/37312

    We do apologise to those affected by this incident, especially those who didn't reconnect successfully early on.

  • Closed