LNS upgrades Sunday morning
MAINTENANCE Closed Broadband and Ethernet
STATUS
Closed
CREATED
Mar 27, 09:17 PM (8 years ago)
AFFECTED
Broadband and Ethernet
STARTED
Apr 03, 02:00 AM (7¾ years ago)
CLOSED
Apr 03, 08:00 AM (7¾ years ago)
REFERENCE
2237 / AA2237
INFORMATION
  • INITIAL
    7¾ years ago by Adrian

    We are planning, at the very least, to upgrade the live LNSs all in one go. However, we are hopeful that we can also upgrade the routed links to both BT and Talk Talk at the same time. This should "just work" but if it does not we will roll back. The effect will be at least one, and possibly more PPP level restarts of all broadband lines and may be a few minutes of outage. If we are successful we will be able to deploy more active LNSs in the following week to spread load and allow more expansion. It will also allow us to more easily expand the number of links we have to carriers over the coming months. This is all part of a major ongoing network capacity upgrade.

  • UPDATE
    7¾ years ago by Adrian

    We have done a lot of work today - both in the data centre (deploying five more FB6000 series FireBricks) and from the office configuring systems ready for the weekend. Well done to all of the staff involved as I do not believe there was a single blip. I feel very confident that we are ready to the work on Sunday now.

  • UPDATE
    7¾ years ago by Adrian

    More preliminary testing yesterday and today but looking like we are on schedule for Sunday morning. We aim to upgrade all LNSs, change carrier interlink so we can use more LNSs and actually start using more LNSs.

  • UPDATE
    7¾ years ago by Andrew

    The work begins...

  • UPDATE
    7¾ years ago by Andrew

    LNSs are having their software updated, this will cause PPP blips for customers.

  • UPDATE
    7¾ years ago by Andrew

    Software has been updated on the LNSs. Stage two will start shortly, this is reconfiguring our links to BT and TT backhaul.

  • UPDATE
    7¾ years ago by Andrew

    We're about to start reconfiguring the TT interconnects. This will cause a PPP blip for customers on T backhaul.

  • UPDATE
    7¾ years ago by Andrew

    TT lines have just blipped as we reconfigured the backhaul links. Lines are logging back in now.

  • UPDATE
    7¾ years ago by Andrew

    TT is working but we are making sure the balancing works before we proceed to the BT side

  • UPDATE
    7¾ years ago by Andrew

    We're about to start reconfiguring the BT interconnects. This will cause a PPP blip for customers on BT backhaul.

  • UPDATE
    7¾ years ago by Andrew

    BT lines blipped their PPP as expected, and are logging back in now.

  • UPDATE
    7¾ years ago by Andrew

    BT lines looking good.

  • UPDATE
    7¾ years ago by Andrew

    All lines are about to have a short PPP blip as we load new software across the LNSs.

  • UPDATE
    7¾ years ago by Andrew

    We are not balancing the BT back haul links properly at the moment, this is being worked on. This is not affecting service at the moment but does mean we are only running on half our links to BT.

  • UPDATE
    7¾ years ago by Andrew

    We are close, but we do have to break things once more sadly, there will be some routing and PPP drop over the next 30 minutes.

  • UPDATE
    7¾ years ago by Andrew

    Lines are reconnecting and routes are being restored We'll then review where we are and will update again shortly.

  • UPDATE
    7¾ years ago by Andrew

    Customers are online, but we are still working on balancing the BT links.

  • UPDATE
    7¾ years ago by Adrian

    We are over running on the window for this planned work. However, customers are online, and we are working on some oddities in how the traffic over our BT links is being balanced.

  • UPDATE
    7¾ years ago by Adrian

    Things look stable and we have managed to balance the traffic properly we are still tracking down one issue this morning. There may be some re-balancing of traffic between LNSs later.

  • RESOLUTION
    7¾ years ago by Adrian

    It took a long time, but the work has been completed, including the deployment of 3 new LNSs to our pool. Thanks to the staff involved, and to our customers for their patience.

    Some further info about this work here: http://www.revk.uk/2016/04/growing-pains-pain-relief.html

  • Closed