Overnight LNS Upgrades
MAINTENANCE Closed LNS Upgrades
STATUS
Closed
CREATED
Jan 02, 12:27 PM (4¼ months ago)
AFFECTED
LNS Upgrades
STARTED
Jan 03, 01:00 AM (4 months ago)
CLOSED
Jan 12, 02:00 PM (3¾ months ago)
SCOPE
12%
REFERENCE
42593 / AA42593
MASTODON
INFORMATION
  • INITIAL
    4 months ago by Andrew

    Brief summary:- This first stage involves around 12% of customers (those on the Y.Witless LNS) who will see their line reconnect in the early hours of Monday 8th January.

    We will be performing a series of overnight software upgrades on some of our LNSs (the ones named Witless) and also some of our BGP routers. This is to update their software in response to the crashes we experienced at the end of 2023. (https://aastatus.net/42577) As usual, this work involved us moving customers off an LNS so as to free it up to be upgraded. The process is then repeated on subsequent days until all LNSs in the pool are upgraded.

    Although we have confidence in the fix, we will upgrade the Witless routers at a slower rate than we usually would so as to minimise the impact to customers if there are problems. We have three to upgrade (X, Y, Z), but we'll only upgrade and put in to service one, as follows:

    • Wednesday 3rd January - Early hours: Move circuits off Z.Witless. (There are not many, as Z was taken out of service in December)
    • Wednesday 3th January - Daytime: Upgrade Z.Witless.
    • Wednesday 3th January - Daytime: Start to slowly move customers from Y.Witless to Z.Witless.
    • Monday 8th January - Early hours: Bulk move customers from Y.Witless to Z.Witless. (This would result in around 12% of customers being moved on to Y.Witless)

    If there are problems, we will revert to where we stood before this work started. If successful, then we will plan to upgrade Y and X during the week beginning 15th January.

  • UPDATE
    4 months ago by Andrew

    Z.Witless has been upgraded and put in to service. Customers on Y.Witless will be moved over to Z.Witless in the early hours of Monday 8th January.

  • UPDATE
    4 months ago by Andrew

    The bulk of customers on Y.Witless have now been moved on to the upgraded Z.Witless. We will continue to monitor and will schedule upgrades of X.Witless and Y.Witless later in the week.

  • UPDATE
    4 months ago by Andrew

    Due to the BT incident yesterday night we have a number of customers back on the Y.Witless LNS. We will be moving these lines off Y.Witless in the early hours of Wednesday 10th January to enable us to upgrade Y.Witless.

  • UPDATE
    4 months ago by Andrew

    In the early hours of Thursday 11th January we'll be moving customers from X.Witless on to Y.Witless.

  • RESOLUTION
    3¾ months ago by Andrew

    This round of upgrades is completed. There will be further work scheduled, these will be posted as separately.

  • Closed