Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Feb 14, 12:00 PM (7¾ days ago)
CLOSED
Feb 18, 03:00 AM (4 days ago)
DESCRIPTION

Happening Monday and Tuesday early mornings:

Most of our LNSs have extra diagnostic and debugging hardware installed which has been used to help us track down the problems we were having with them in early 2024. We are now in a position where we can remove this. We have planned to do this work over two nights: the early hours of February 17th and 18th.

This work will involve us moving customers off the LNSs as we do the work. In practice this means customer connections will experience a drop and reconnect at around 1AM and 4AM on the 17th and 18th February.


Resolution: This work was completed at around 3AM pn Tuesday morning.

MAINTENANCE Completed BT
AFFECTING
BT
STARTED
Feb 12, 01:00 AM (10¼ days ago)
CLOSED
Jan 29, 11:35 AM (23¾ days ago)
DESCRIPTION

BT are performing "Simultaneous Upgrades" which will affect all BT ADSL, VDSL and FTTP services during the early hours of February 12th. It is disappointing that this work is a multi-site activity as we have multiple links to BT in multiple datacentres so as to prevent outages when BT have work to do.

They say:

Start: 2025-02-12 00:01. We're simultaneously upgrading some 21C network software across multiple sites. The PW window is from 00:01 until 07:00, and we will start at 00:01 with non-disruptive pre-checks. From 01:00 we will start to re-boot the devices to the new version of code. This will cause an outage to all customers of up to 15 minutes whilst the device restarts. However, if roll-back is required, the outage could exceed 1hr. All work will be complete by 07:00.

In practice, we expect customers with BT provided connections to drop and reconnect a couple of times in the early hours of February 12th.


Resolution: CANCELLED! This notification fas sent to us by BT in error - their work is not expected to affect our services at all.

MAINTENANCE Assumed Completed DNS
AFFECTING
DNS
STARTED
Feb 11, 03:30 PM (10½ days ago)
DESCRIPTION

This notification is for customers who run their own authoritative DNS servers and use our secondary-dns.co.uk as secondary (slave).

As part of our DNS infrastructure project we will start to initiate Zone Transfers from an additional set of IP addresses.

You will still send your NOTIFY to secondary-dns.co.uk but you will start to see AXFR requests from an additional set of IP addresses.

Therefore, please update your ACLs to allow the following addresses in addition to what you have at the moment:

  • 194.4.172.3
  • 194.4.172.4
  • 194.4.173.3
  • 194.4.173.4
  • 2001:8b6:1:0:194:4:172:3
  • 2001:8b6:1:0:194:4:172:4
  • 2001:8b6:2:0:194:4:173:3
  • 2001:8b6:2:0:194:4:173:4

Please update your ACLs by March 11th.

We will have an overlap of using the existing DNS servers and the new DNS servers. As part of our testing we will start to use the new IP addresses from February 17th


MAINTENANCE Completed Email
AFFECTING
Email
STARTED
Feb 10, 09:00 AM (11¾ days ago)
CLOSED
Feb 18, 02:11 PM (3¾ days ago)
DESCRIPTION
We will be doing work on our email servers which should not affect customers. The work involves rebuilding one of the two back end file stores and will be taking place across multiple days.
Resolution: The initial set of work has been completed successfully and without any customer impact. We'll be doing further work in a couple of weeks time, and will post a new status post nearer the time.

MINOR Closed General
AFFECTED
General
STARTED
Feb 05, 11:20 PM (16¼ days ago)
CLOSED
Feb 05, 11:30 PM (16¼ days ago)
DESCRIPTION
We had a network incident lasting a couple of minutes from 23:20. This caused one of our transit routers to fail and it is currently offline. This was one of our routers to 'Lumen'. We have 5 other transit routers, including one that is also connected to Lumen from a different location so this is not causing any further problems.
Resolution:

MINOR Open Email
AFFECTING
Email
STARTED
Feb 03, 09:00 AM (18¾ days ago)
DESCRIPTION

Over the past week we have seen a huge number of 'bots' trying to guess customer email credentials in order to try to send email through our outbound email servers: smtp.aa.net.uk.

The attempts were being blocked due to wrong passwords being used, but this caused significant load on our severs due to all the database lookups involved. To address this, we are blocking IP addresses that are listed on the Spamhaus 'Exploits Blocklist (XBL)' and the Spamhaus 'Combined Spam Sources (CSS)' lists. -These are typically IP address known to have hijacked in some way or known spam senders.

This has reduced the load on the email servers significantly, however we are are still blocking around 1.5 million unique IP addresses each day.

We have had a small number of legitimate customers affected by this as their IP address is on these blocklists. (IPs can be looked up on https://check.spamhaus.org). In these cases, please do contact support and we can discuss workarounds.


MINOR Closed Routing
AFFECTED
Routing
STARTED
Jan 27, 08:28 PM (25½ days ago)
CLOSED
Jan 27, 09:38 PM (25¼ days ago)
DESCRIPTION
We're investigating a large number of drops and reconnects of our L2TP service (l2tp.aa.net.uk)
Resolution:

MINOR Closed Database
AFFECTED
Database
STARTED
Jan 24, 07:02 AM (29 days ago)
CLOSED
Jan 24, 07:22 AM (29 days ago)
DESCRIPTION
Some VoIP calls and the control pages were affected by a Database issue this morning between 7AM and 7:20AM.
Resolution: Our database problem seems to be caused by a recent Linux kernel update. We're tracking the upstream bug and have reverted to a previous kernel.

MINOR Closed Éowyn
AFFECTED
Éowyn
STARTED
Jan 23, 07:00 PM (29½ days ago)
CLOSED
Feb 12, 09:00 AM (9¾ days ago)
DESCRIPTION
Openreach have notified us of the precautions they are taking in light of storm Éowyn. This involves them not sending Engineers on Friday to areas with 'red' warnings. This may may missed appointments for install or repair activities whilst these warnings are in place.
Resolution:

MINOR Closed SIP2SIM
AFFECTED
SIP2SIM
STARTED
Jan 23, 02:00 PM (29¾ days ago)
CLOSED
Jan 23, 03:30 PM (29½ days ago)
DESCRIPTION
We are investigating a problem with our SIP2SIM servers registering to customer's own PBXs. This isn't affecting SIP2SIM customers where we are also providing the VoIP service/number.
Resolution:

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Jan 23, 10:03 AM (29¾ days ago)
CLOSED
Jan 23, 10:14 AM (29¾ days ago)
DESCRIPTION
We're investigating a problem with VoIP registrations having problems.
Resolution: Early indications looks to point the cause at our REDIS cluster having failing over when the master had a problem. Further investigations will continue.

MAINTENANCE Completed Broadband
AFFECTING
Broadband
STARTED
Nov 19, 09:30 AM (3 months ago)
CLOSED
Jan 23, 07:00 PM (29½ days ago)
DESCRIPTION

The work outlined below will start from Saturday 23rd November.

Background:

Our FireBrick team has been working on the 'hang' problem that we faced with the LNSs earlier in the year.

The nature of the problem has made investigating the problem very time consuming as it is extremely difficult to reproduce. However, we do believe that a plausible cause has been identified, and code changes have been made to mitigate the problem.

We have been testing this new code, both in our test lab and on a few select A&A routers, for over two months. During this time the new code has not caused the hardware to hang, where older versions of the code did.

Our next step is to run the new code on our LNSs, the ones our customers connect to for their broadband connections.

We plan to do this slowly, out of hours and in a couple of phases.

We believe the cause of the hang is related to how memory is initially allocated for the tasks the FireBrick will be performing, this means that if the hardware is going to hang then this will most likely happen over the first couple of days (or first couple of hours).

Stage one: (Completed)

We plan to upgrade only one of our LNSs at first. We will move broadband connections on to it in the early hours of the morning and then move them back off a few hours later. This means that during the day, customers will be on the normal set of LNSs.

Then, each night, over the course of two weeks, the LNS will be power cycled and we will move an increasing number of connections over, until it is at the point of taking twice the amount of connections that we'd normally run on an LNS. (We normally run LNSs at around 40% capacity, so twice the number of connections is not a problem.)

Stage two: (Completed)

Once we have confirmed that the hang is not happening, the second phase would be to run customer connections on the upgraded for a few days at a time.

We will go through a cycle of: move connections off, reboot the LNS, move connections on, wait a few days. Repeat. We will do this with an increasing number of connections until it's at the point of taking a normal amount of connections.

Stage Three:

As of the end of 2024, half our LNSs have been running the new software without any problems. From January 7th we will e doing overnight upgrades of the remaining LNSs.

More information:

So as to minimise impact to customers, the work of moving connections off and on will happen overnight between 1AM and 5AM.


Resolution: All of our FireBrick FB900 LNSs and BGP routers have been upgraded successfully.