Last 10 Maintenance posts, sorted by last updated
Order posts by limited to posts

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Mar 01, 03:00 AM (4 days ago)
CLOSED
Mar 01, 04:44 AM (3¾ days ago)
DESCRIPTION

We have work planned for the early hours of Friday morning that entails upgrading software on our LNSs and moving CityFibre and higher-speed BT/TalkTalk services on to separate pools of routers (LNSs) at our side.

In practice this will mean that most customers with speeds of 80Mb/s and above will experience a few PPP drops and reconnects between 3AM and 5AM as we carry out the work.

This is related to the hardware hangs we've been experiencing: https://aastatus.net/42608 and it will help us further investigate this ongoing issue.


Resolution: This work has been completed.

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Feb 17, 01:00 AM (17 days ago)
CLOSED
Feb 19, 01:52 AM (15 days ago)
DESCRIPTION

During the early hours of Saturday 16th and Sunday 17th we will be performing some software upgrades and rebalancing of customers that are currently on the A, B, C, G, Y, Z LNSs.

The aim here is upgrade the software on some of them and to spread the customers a bit more evenly over them. Customers on X will remain as is.
Resolution: This work has been completed - Some lines were moved to different LNSs in the early hours of Saturday and Monday.

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Feb 16, 12:30 PM (17½ days ago)
CLOSED
Feb 16, 12:40 PM (17½ days ago)
DESCRIPTION
The very small number of customers on G.gormless had their service drop and reconnect caused by the LNS locking up.
Resolution:

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Feb 11, 04:00 AM (23 days ago)
CLOSED
Feb 11, 05:00 AM (22¾ days ago)
DESCRIPTION
In the early hours of Sunday morning (4AM and 4:30AM) we will be performing a software upgrade on G.Gormless and Y.Witless. This will cause a drop for the small number of customers on these LNSs, but they will reconnect quickly. This software add a bit more logging to aid our diagnostics.
Resolution: This has been completed.

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Feb 10, 02:00 AM (24 days ago)
CLOSED
Feb 10, 10:00 AM (23¾ days ago)
DESCRIPTION

This week we have installed three additional FB9000 LNSs. These are: A.Gormless, B.Gormless, C.Gormless. These are in addition to the existing: G.Gormless, X.Witless, Y.Witless Z.Witless which are all used for customers on faster circuits (80M and above).

As of Friday 9th customers (with 80M and above connections) are mostly connected to G, X, and Y.

During the early hours of Saturday 10th and Sunday 11th we will rebalance customers that are currently on G and Y over to A, B, C, G, Y, Z - the aim here is to spread the load so fewer customers are on each LNS which means fewer customers will be affected if the LNS locks-up. Customer on X will remain as is.


Resolution: This work has been completed.

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Feb 03, 01:00 AM (1 month ago)
CLOSED
Feb 05, 06:20 PM (28¼ days ago)
DESCRIPTION
We are making plans to replace a few more of the "Gormless" LNSs with FireBrick FB9000 models. In order to free up these LNSs we will be moving the small number of customers that are currently on A, B & C Gormless to other LNS. This will happen in the early hours of Saturday and Sunday morning.
Resolution: a, b, c Gormless are free of customers. We will plan to swap these over to FB9000 on Wednesday.

MAINTENANCE Completed LNS
AFFECTING
LNS
STARTED
Feb 02, 10:00 AM (1 month ago)
CLOSED
Feb 02, 10:32 AM (1 month ago)
DESCRIPTION

We will be making live a new LNS called "G.Gormless" and adding it to the pool of LNS for our faster lines. This will complement the existing X, Y & Z Witless LNSs.


Resolution: G.Gormless is now live.

MAINTENANCE Completed Z.Witless
AFFECTING
Z.Witless
STARTED
Jan 27, 10:30 AM (1¼ months ago)
CLOSED
Jan 31, 04:43 PM (1 month ago)
DESCRIPTION

Z.Witless has locked up a few times this week. We will be performing the following work on it to help us investigate the problem further:

  1. Move customers off Z.Witless (3AM on Sunday 28th January)*
  2. Install an NVMe drive - (Reasons for this in https://aastatus.net/42608)
  3. Install JTAG connection to the main PCB so that future lockups of the CPU can be investigated further
* Circuits will be moved on to Y.Witless which already has NVMe and JTAG.
Resolution: This work has been competed.

MAINTENANCE Completed Y.Witless
AFFECTING
Y.Witless
STARTED
Jan 22, 05:00 PM (1¼ months ago)
CLOSED
Jan 23, 02:00 PM (1¼ months ago)
DESCRIPTION

This post is an "action point" related to the previously posted update on our LNS situation: https://aastatus.net/42608

There are a small number of customer on the Y.Witless LNS, these will be moved off in the early hours of Tuesday 23rd January.

Later that day, the hardware of Y.Witless, A.Gormless and B.Gormless will be replaced. These three units will then be updated hardware with an NVMe drives and with it's low-level debugging connector (JTAG) connected so that we can gain further information upon any future lockup.

The work to replace the hardware is not expected to impact the service.


Resolution: Y.Witless has been replaced. The Gormless LNSs work has been postponed and will be rescheduled for later in the week.

MAINTENANCE Completed Control Pages
AFFECTING
Control Pages
STARTED
Jan 21, 10:00 AM (1¼ months ago)
CLOSED
Jan 21, 07:17 PM (1¼ months ago)
DESCRIPTION
We will be carrying out maintenance work on our accounts system and the accounts control pages on Sunday the 21st of January. The accounts pages will be unavailable for a period of time during this work, but we'll aim to make any disruption as brief as possible.
Resolution: