Major Talk Talk issue impacting broadband lines
MAJOR Closed Broadband and Ethernet
STATUS
Closed
CREATED
Mar 17, 06:02 AM (8 years ago)
AFFECTED
Broadband and Ethernet
STARTED
Mar 17, 06:00 AM (8 years ago)
CLOSED
Mar 17, 08:27 AM (8 years ago)
REFERENCE
2229 / AA2229
INFORMATION
  • INITIAL
    8 years ago by Adrian

    Planned work has finished but Talk Talk lines have not yet come back up. This is being worked on by engineers now. This seems to be coincidence and is a major issue impacting TT backhaul and stopping TT lines connect over all this wholesale customers, not just us.

  • UPDATE
    8 years ago by Adrian

    We are in contact with Talk Talk - it looks a lot like their RADIUS servers stopped talking to us during the work this morning and need some sort of a kick from their end.

  • UPDATE
    8 years ago by Adrian

    There will be a blip on half of our BT lines now - we are trying to eliminate anything in our network as the cause of the Talk Talk issue.

  • UPDATE
    8 years ago by Adrian

    As expected, that did not help the Talk Talk issue - we are waiting on them now.

  • UPDATE
    8 years ago by Adrian

    We have senior technical staff at Talk Talk working on this now.

  • UPDATE
    8 years ago by Adrian

    Talk Talk are expecting this to be sorted in under 20 minutes.

  • UPDATE
    8 years ago by Adrian

    During the planned work, both BT and TT lines went down at around 03:30 this morning. This was not quite as we had planned, as we expected lines to blip only for a few seconds. In fact the BT lines were down for over half an hour. However, the TT lines never came back. The planned work was actually fully reverted and that was completed by 6am but still TT lines did not come back. So this is looking very much like something in Talk Talk that is not right. We are hopeful that this will be resolved very soon but at this stage we are waiting for TT to fix their end.

  • UPDATE
    8 years ago by Andrew

    ...

  • UPDATE
    8 years ago by Adrian

    Talk Talk are working on this still. Just to be clear, this is only the TT back-haul lines. All BT back-haul lines should be fine. As ever, we have seen routers which can "sulk" so always worth "turning it off and back on again" before calling/irc us if you have a line on BT back-haul which is down.

  • UPDATE
    8 years ago by Adrian

    This is only *our* TT lines, not other ISPs, just to be clear. We think it is a matter of TT's RADIUS servers deciding that our RADIUS servers have stopped responding (due to the planned work) and sulking. We think that they need some manual intervention to start talking to us again. We will be trying to confirm this is the case with Talk Talk once this is resolved and see if this operating can be changed in future as this is clearly unacceptable.

  • UPDATE
    8 years ago by Adrian

    Thank you all for your patience on this - we are very much at the mercy of Talk Talk here, and have been in constant contact with their engineers to try and get this sorted. It should be simple, but seems to be taking rather a long time.

  • UPDATE
    8 years ago by Adrian

    Talk talk agree it is definitely something wrong there end and are still working on it.

  • UPDATE
    8 years ago by Adrian

    Talk talk have found the issue and working in a fix now. Should be very soon now.

  • UPDATE
    8 years ago by Adrian

    Latest update from TT is: "Got it. Working on fix now. Our issue"

  • UPDATE
    8 years ago by Adrian

    Latest update. This looks like a coincidence and is actually impacting all TT wholesale lines and stopping them connect. For most ISPs customers stay on line but as we had planned work it has meant all of our TT backhaul lines are down and simply stayed down. TT are working on it.

  • UPDATE
    8 years ago by Adrian

    TT have confirmed that this was not in any way caused by our planned work, just really bad timing.

  • UPDATE
    8 years ago by Adrian

    Unfortunately as we have backed out all of the planned work we will have to schedule it again and send engineers in to London in the middle of the night. We'll announce more on that once this is sorted.

  • UPDATE
    8 years ago by Adrian

    Lines are coming back on now

  • RESOLUTION
    8 years ago by Adrian

    TT have fixed the issue. We will be asking for more details of the cause. Thank you all for your patience.

  • Closed