Friday 13th: Major issue with broadband service
MAJOR Closed Broadband and Ethernet
STATUS
Closed
CREATED
Jan 13, 11:07 AM (12 years ago)
AFFECTED
Broadband and Ethernet
STARTED
Jan 13, 11:03 AM (12 years ago)
CLOSED
Jan 13, 11:37 AM (12 years ago)
REFERENCE
1347 / AA1347
INFORMATION
  • INITIAL
    12 years ago by Andrew

    We're looking in to a network problem affecting broadband customers - More details to follow...

    Lines are coming back and we arte trying to find what happened.

  • UPDATE
    12 years ago

    We have engineers onsite in telehouse looking in to this.

    However, lines and routing are recovering now.

  • UPDATE
    12 years ago

    Looks like some network issue in our core rack in telehouse - engineers on site investigating now. Lines mostly back up now.

  • UPDATE
    12 years ago

    We are still somewhat at a loss as to what happened, but it has meant all lines being dropped of the live LNS. We have switched the the backup anyway.

    However, and possibly unrelated, we are seeing issues with BT 20CN lines coming up - i.e. they are not. This was happening before this more major issue, and means 20CN lines are still down. We are chasing With BT.

  • UPDATE
    12 years ago

    20CN lines now on - looks like BT did a config change early which resulted in problems. This would have been a non issue if there was not also some sort of major blip which we have yet to identify.

  • RESOLUTION
    12 years ago by Andrew

    Two problems it seems.

    1. BT did a configuration change earlier than expected. No idea why, and normally this would have affected a couple of customers at most. We were looking in to this already when...

    2. Somehow there was a loss of contact between our live LNS and core network. We can see this must have happened and we are checking the switch logs to see if we can find why. Engineers were on site in the new rack (not connected, do not the cause!) and are checking this out now.

    The effect was all lines went off. We have changed the system to be less sensitive in such cases now and not drop all sessions due to a loss of connectivity to the core network.

    However, as all lines went off, the BT config change meant none of the 20CN lines came back until we discovered that what they had done.

    Almost all lines on-line, and all should now be able to connect as normal.

    All of the switches are being replaced as part of the upgrade anyway, but we obviously want to understand the underlying issue here.

    Update: This is looking increasingly like a switch issue - the switch is working now, and new switches are in the new rack ready to be deployed over the next week or two.

  • Closed