We are planning, at the very least, to upgrade the live LNSs all in one go. However, we are hopeful that we can also upgrade the routed links to both BT and Talk Talk at the same time. This should "just work" but if it does not we will roll back. The effect will be at least one, and possibly more PPP level restarts of all broadband lines and may be a few minutes of outage. If we are successful we will be able to deploy more active LNSs in the following week to spread load and allow more expansion. It will also allow us to more easily expand the number of links we have to carriers over the coming months. This is all part of a major ongoing network capacity upgrade.
We have done a lot of work today - both in the data centre (deploying five more FB6000 series FireBricks) and from the office configuring systems ready for the weekend. Well done to all of the staff involved as I do not believe there was a single blip. I feel very confident that we are ready to the work on Sunday now.
More preliminary testing yesterday and today but looking like we are on schedule for Sunday morning. We aim to upgrade all LNSs, change carrier interlink so we can use more LNSs and actually start using more LNSs.
The work begins...
LNSs are having their software updated, this will cause PPP blips for customers.
Software has been updated on the LNSs. Stage two will start shortly, this is reconfiguring our links to BT and TT backhaul.
We're about to start reconfiguring the TT interconnects. This will cause a PPP blip for customers on T backhaul.
TT lines have just blipped as we reconfigured the backhaul links. Lines are logging back in now.
TT is working but we are making sure the balancing works before we proceed to the BT side
We're about to start reconfiguring the BT interconnects. This will cause a PPP blip for customers on BT backhaul.
BT lines blipped their PPP as expected, and are logging back in now.
BT lines looking good.
All lines are about to have a short PPP blip as we load new software across the LNSs.
We are not balancing the BT back haul links properly at the moment, this is being worked on. This is not affecting service at the moment but does mean we are only running on half our links to BT.
We are close, but we do have to break things once more sadly, there will be some routing and PPP drop over the next 30 minutes.
Lines are reconnecting and routes are being restored We'll then review where we are and will update again shortly.
Customers are online, but we are still working on balancing the BT links.
We are over running on the window for this planned work. However, customers are online, and we are working on some oddities in how the traffic over our BT links is being balanced.
Things look stable and we have managed to balance the traffic properly we are still tracking down one issue this morning. There may be some re-balancing of traffic between LNSs later.
It took a long time, but the work has been completed, including the deployment of 3 new LNSs to our pool. Thanks to the staff involved, and to our customers for their patience.
Some further info about this work here: http://www.revk.uk/2016/04/growing-pains-pain-relief.html