Over the past few weeks we have been carrying out planned work to 'shuffle' customers between some of the 'Witless' LNS routers at our side. This work was carried out to apply both a software update and configuration change. From a customer view, this work cause a line drop overnight which is usually a very short outage in the early hours of the morning, and not usually much of an inconvenience. However, in addition to this planned work we have had a few crashes over the past week. (10th at 8:30AM, 13th at 1:40PM and 14th at 2:50AM.) These have caused a few minutes interruption to some customers during the day time, which is very inconvenient and we do apologise for this.
The cause of these restarts is known to us. It is in relation to a low level processor issue that requires a complicated workaround. We are working on a longer term software fix for this. However, one of the configuration changes we made has exacerbated this bug and has made it more prevalent, we therefore plan to revert the configuration change ASAP. The longer term software will then be applied at a later date once it is ready
The configuration change does need us to reboot our routers, and so this will involve us carrying out another round of moving customers between LNSs again. This work will happen in the early hours of the morning starting on Wednesday 15th November.
Customers on "Y.Witless" will be moved during the early hours of Wednesday 15th November. Y.witless will then have it's config change and rebooted the following day.
Tuesday evening (21:03) Sadly, X.Witless crashed this evening causing disruption for those customers connected to it. X.Witless would have been scheduled to have the mentioned config change and reboot tomorrow night - that won't be needed now, as the restart means it has now applied the config change.
All our 'Witless' LNS routers have now had the configuration change applied, which should hopefully reduce the chance of further crashes. We'll keep this post open and will post further updates regarding the software upgrade which we hope will happen early next week.
Some more changes and testing of the software are still required before we update our routers.
A software upgrade is being applied this week: https://aastatus.net/42582
Due in 1¾ days