Work to help resolve recent LNS problems
MAINTENANCE Open LNS
STATUS
Open
CREATED
Nov 14, 04:50 PM (14 days ago)
AFFECTING
LNS
STARTED
Nov 14, 05:00 PM (14 days ago)
REFERENCE
42577 / AA42577
MASTODON
INFORMATION
  • INITIAL
    14 days ago by Andrew

    Over the past few weeks we have been carrying out planned work to 'shuffle' customers between some of the 'Witless' LNS routers at our side. This work was carried out to apply both a software update and configuration change. From a customer view, this work cause a line drop overnight which is usually a very short outage in the early hours of the morning, and not usually much of an inconvenience. However, in addition to this planned work we have had a few crashes over the past week. (10th at 8:30AM, 13th at 1:40PM and 14th at 2:50AM.) These have caused a few minutes interruption to some customers during the day time, which is very inconvenient and we do apologise for this.

    The cause of these restarts is known to us. It is in relation to a low level processor issue that requires a complicated workaround. We are working on a longer term software fix for this. However, one of the configuration changes we made has exacerbated this bug and has made it more prevalent, we therefore plan to revert the configuration change ASAP. The longer term software will then be applied at a later date once it is ready

    The configuration change does need us to reboot our routers, and so this will involve us carrying out another round of moving customers between LNSs again. This work will happen in the early hours of the morning starting on Wednesday 15th November.

  • UPDATE
    14 days ago by Andrew

    Customers on "Y.Witless" will be moved during the early hours of Wednesday 15th November. Y.witless will then have it's config change and rebooted the following day.

  • UPDATE
    13¾ days ago by Andrew

    Tuesday evening (21:03) Sadly, X.Witless crashed this evening causing disruption for those customers connected to it. X.Witless would have been scheduled to have the mentioned config change and reboot tomorrow night - that won't be needed now, as the restart means it has now applied the config change.

  • UPDATE
    13¼ days ago by Andrew

    All our 'Witless' LNS routers have now had the configuration change applied, which should hopefully reduce the chance of further crashes. We'll keep this post open and will post further updates regarding the software upgrade which we hope will happen early next week.

  • UPDATE
    8 days ago by Andrew

    Some more changes and testing of the software are still required before we update our routers.

  • UPDATE
    1 day ago by Andrew

    A software upgrade is being applied this week: https://aastatus.net/42582

  • NEXT UPDATE...

    Due in 1¾ days