Outages
Maintenance
Our Network:
Major Outages

0

Minor Outages

0

Future Planned

1

Open Events:
STATUS
Open
CREATED
Jan 13, 10:16 AM (4½ days ago)
AFFECTING
Email
STARTED
Jan 14, 09:00 AM (3½ days ago)
REFERENCE
42733 / AA42733
MASTODON
INFORMATION
  • INITIAL
    3½ days ago by Andrew

    We will be doing work on our email servers which should not affect customers. The work involves rebuilding one of the two back end file stores and will be taking place across multiple days.

  • UPDATE
    15¼ hours ago by Andrew

    This work has been moved to next week - week beginning 20th January.

  • NEXT UPDATE...

    Due in 3½ days

EXPECTED CLOSE
Jan 21, 04:00 PM ( In 3½ days )
STATUS
Open
CREATED
Nov 19, 09:46 AM (2 months ago)
AFFECTING
Broadband
STARTED
Nov 19, 09:30 AM (2 months ago)
REFERENCE
42728 / AA42728
MASTODON
INFORMATION
  • INITIAL
    2 months ago by Andrew

    The work outlined below will start from Saturday 23rd November.

    Background:

    Our FireBrick team has been working on the 'hang' problem that we faced with the LNSs earlier in the year.

    The nature of the problem has made investigating the problem very time consuming as it is extremely difficult to reproduce. However, we do believe that a plausible cause has been identified, and code changes have been made to mitigate the problem.

    We have been testing this new code, both in our test lab and on a few select A&A routers, for over two months. During this time the new code has not caused the hardware to hang, where older versions of the code did.

    Our next step is to run the new code on our LNSs, the ones our customers connect to for their broadband connections.

    We plan to do this slowly, out of hours and in a couple of phases.

    We believe the cause of the hang is related to how memory is initially allocated for the tasks the FireBrick will be performing, this means that if the hardware is going to hang then this will most likely happen over the first couple of days (or first couple of hours).

    Stage one: (Completed)

    We plan to upgrade only one of our LNSs at first. We will move broadband connections on to it in the early hours of the morning and then move them back off a few hours later. This means that during the day, customers will be on the normal set of LNSs.

    Then, each night, over the course of two weeks, the LNS will be power cycled and we will move an increasing number of connections over, until it is at the point of taking twice the amount of connections that we'd normally run on an LNS. (We normally run LNSs at around 40% capacity, so twice the number of connections is not a problem.)

    Stage two: (Completed)

    Once we have confirmed that the hang is not happening, the second phase would be to run customer connections on the upgraded for a few days at a time.

    We will go through a cycle of: move connections off, reboot the LNS, move connections on, wait a few days. Repeat. We will do this with an increasing number of connections until it's at the point of taking a normal amount of connections.

    Stage Three:

    As of the end of 2024, half our LNSs have been running the new software without any problems. From January 7th we will e doing overnight upgrades of the remaining LNSs.

    More information:

    So as to minimise impact to customers, the work of moving connections off and on will happen overnight between 1AM and 5AM.

  • UPDATE
    1¾ months ago by Andrew

    Overnight test over the weekend have so far been successful. We'll be continue to test during his week.

  • UPDATE
    1½ months ago by Andrew

    No problems thus far. We are now at phase 2, where we are running customers on the upgraded LNS for longer periods of time. We currently have eight LNSs in use, and one (i.gormless) is running the new software. This stage will involve lines on G, H and I.Gormless.

  • UPDATE
    1¼ months ago by Andrew

    Still no problems. We will upgrade two more LNS (G.gormless and H.Gormless), and will move some customers on to these over the weekend. This will happen overnight.

  • UPDATE
    1 month ago by Andrew

    All LNSs remain stable. Over this weekend (14-15th December) we will upgrade further LNSs and move customers between LNSs: E, F, G, H, I. This will be our last set of LNS upgrades until the new year.

  • UPDATE
    1 month ago by Andrew

    These LNSs are now running newer software: F, G, H, I. We'll continue this work in the new year.

  • UPDATE
    10½ days ago by Andrew

    Overnight works are planned to upgrade the remaining LNSs. This will happen from 8th January onwards.

  • UPDATE
    4½ days ago by Andrew

    All of our LNS routers have been upgraded and all are working well. We will continue to upgrade our remaining FB9000 BGP routers over the next 10 nights.

  • NEXT UPDATE...

    Due in 5½ days

STATUS
Open
CREATED
Jun 03, 03:13 PM (7½ months ago)
AFFECTING
authoritative DNS
STARTED
Jun 03, 10:35 AM (7½ months ago)
REFERENCE
42684 / AA42684
MASTODON
INFORMATION
  • INITIAL
    7½ months ago by Andrew

    This is only relevant to customers who run their own authoritative DNS servers and use our secondary-dns.co.uk as an additional nameserver.

    Overview: We run a "secondary" DNS service for customers where they run the master DNS server and we are secondary slaves. We have a project underway that involves migrating all our authoritative DNS services to a new platform. As part of this we are needing to disable some of the automation we do for adding and updating the customer's master IP address automatically.

    The change: From June 17th, If you run your own master DNS server for your domain(s) and secondary-dns.co.uk is a slave, if you change the IP address of your master you will need to contact support@aa.net.uk to request us to update our side.

    We have more information about our Authoritative DNS project on our Support Site: https://support.aa.net.uk/New_Authoritive_DNS

  • UPDATE
    7 months ago by James

    This change has been made.

  • NEXT UPDATE...

    Due 7 months ago (overdue)

Broadband blip graph
About our status page

This is the status page of Andrews & Arnold Ltd.

Our status page shows outages (problems) and maintenance (planned work) that happen on our own network and systems and also that of our suppliers networks and systems. We try and ensure this site is updated as soon as possible with incidents as they happen. Live discussion of issues is usually available on IRC.

The last update was Today 07:57:57

Contacting us
Our support number is 033 33 400 999, or you can email support@aa.net.uk or text 01344 400 999 to raise a support ticket.

Spotted a Major Service Outage? (MSO)
A Major Service Outage disrupts the service of multiple customers simultaneously. If you believe that a problem affects multiple customers, and is not mentioned here already, text the number above. Begin the text with "MSO". This alerts multiple staff immediately, waking them if necessary. False alarms (i.e. raising MSO for a single line being down) may result in your number being prevented from raising MSO alerts in future. More info.

Regular Maintenance
Thursday evenings, from 10pm, are designated as a general maintenance window where we will perform non-service affecting updates.