Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.
Over the past few weeks we have been carrying out planned work to 'shuffle' customers between some of the 'Witless' LNS routers at our side. This work was carried out to apply both a software update and configuration change. From a customer view, this work cause a line drop overnight which is usually a very short outage in the early hours of the morning, and not usually much of an inconvenience. However, in addition to this planned work we have had a few crashes over the past week. (10th at 8:30AM, 13th at 1:40PM and 14th at 2:50AM.) These have caused a few minutes interruption to some customers during the day time, which is very inconvenient and we do apologise for this.
The cause of these restarts is known to us. It is in relation to a low level processor issue that requires a complicated workaround. We are working on a longer term software fix for this. However, one of the configuration changes we made has exacerbated this bug and has made it more prevalent, we therefore plan to revert the configuration change ASAP. The longer term software will then be applied at a later date once it is ready
The configuration change does need us to reboot our routers, and so this will involve us carrying out another round of moving customers between LNSs again. This work will happen in the early hours of the morning starting on Wednesday 15th November.
We have multiple interlinks with BT in separate datacentres, however, BT need to make a change to their Core Network that will unfortunately impact all BT services. (Etherway/Etherflow services are not affected)
BT are simultaneously upgrading some 21C network software across multiple sites. The window is 8th November 00:01 until 07:00, and will start at 00:01 with non-disruptive pre-checks. From 01:00 BT will start to re-boot the devices to the new version of code. This will cause an outage to all customers of up to 15 minutes whilst the device restarts. However, if roll-back is required, the outage could exceed 1hr. All work will be complete by 07:00.