Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

MAINTENANCE Assumed Completed Incoming Email
AFFECTING
Incoming Email
STARTED
Oct 17, 12:00 PM (1¼ months ago)
CLOSED
Nov 29, 02:00 PM (17¼ hours ago)
DESCRIPTION
Our email services have slightly changed the way we that we spam score email. We are now using two spam checkers (Rspamd and Spamassassin) and we will use the highest score to classify email as spam. The purpose is to catch more spam. Please do let us know if you notice any problems or have any questions.
Resolution:

MAINTENANCE Assumed Completed LNS
AFFECTING
LNS
STARTED
Nov 14, 05:00 PM (14 days ago)
DESCRIPTION

Over the past few weeks we have been carrying out planned work to 'shuffle' customers between some of the 'Witless' LNS routers at our side. This work was carried out to apply both a software update and configuration change. From a customer view, this work cause a line drop overnight which is usually a very short outage in the early hours of the morning, and not usually much of an inconvenience. However, in addition to this planned work we have had a few crashes over the past week. (10th at 8:30AM, 13th at 1:40PM and 14th at 2:50AM.) These have caused a few minutes interruption to some customers during the day time, which is very inconvenient and we do apologise for this.

The cause of these restarts is known to us. It is in relation to a low level processor issue that requires a complicated workaround. We are working on a longer term software fix for this. However, one of the configuration changes we made has exacerbated this bug and has made it more prevalent, we therefore plan to revert the configuration change ASAP. The longer term software will then be applied at a later date once it is ready

The configuration change does need us to reboot our routers, and so this will involve us carrying out another round of moving customers between LNSs again. This work will happen in the early hours of the morning starting on Wednesday 15th November.


MAINTENANCE Assumed Completed Witless LNS
AFFECTING
Witless LNS
STARTED
Nov 28, 01:00 AM (19½ hours ago)
DESCRIPTION
We'll be performing overnight software upgrades on our 'Witless' pool of LNSs this week, starting Tuesday. This involves moving lines between the LNSs and upgrading when lines have been moved off. Customers may see a PPP drop and reconnect in the early hours of the morning this week as this work is carried out. This work is related to https://aastatus.net/42577

MINOR Closed DATA SIMs
AFFECTED
DATA SIMs
STARTED
Nov 21, 10:42 AM (7¼ days ago)
CLOSED
Nov 21, 11:00 AM (7¼ days ago)
DESCRIPTION
At 10:42 we saw a large number of Data SIMs drop and reconnect. This looks like a problem upstream of us. Service was restored quickly as the SIMs reconnected quickly. We'll update when we have more information.
Resolution:

MINOR Closed Broadband
AFFECTED
Broadband
STARTED
Nov 17, 10:23 AM (11¼ days ago)
CLOSED
Nov 17, 05:18 PM (11 days ago)
DESCRIPTION
Update from supplier; We've identified an issue impacting customer connectivity. Our initial analysis shows the issue is impacting service in and around London regions. We're currently working through the impact and issue and will look to provide more detail as soon as we have it. We'll keep you informed and updated on developments. Our priority is to restore service to normal operations with minimal disruption as soon as possible.
Resolution: This was caused by a fibre break in the London area - traffic was rerouted, and has remained stable.

MINOR Closed TalkTalk
AFFECTED
TalkTalk
STARTED
Nov 15, 08:30 AM (13½ days ago)
CLOSED
Nov 17, 11:30 AM (11¼ days ago)
DESCRIPTION
TalkTalk have had some core network problems and we've see a small number of lines reconnects today at 8:30, 11:15 and 11:19 and 11:38 (and further ones). NOTE: This will be affecting other ISPs who use TalkTalk Wholesale services.
Resolution: TalkTalk believe this fault is fixed, and are monitoring.

MINOR Closed BT
AFFECTED
BT
STARTED
Nov 16, 09:00 AM (12¼ days ago)
CLOSED
Nov 16, 01:00 AM (12¾ days ago)
DESCRIPTION
Just before 1AM and again at 2AM we had a large number of BT connected lines drop and reconnect. This looks related to an incident affecting one of our hostlinks to BT. The timing of this suggests BT were carrying out planned work, but so far we are unaware of any notifications for this work. We'll investigate further.
Resolution:

MAINTENANCE Completed Witless LNS
AFFECTING
Witless LNS
STARTED
Nov 02, 12:05 AM (26¾ days ago)
CLOSED
Nov 13, 06:00 AM (15½ days ago)
DESCRIPTION
Customers with higher speed services (ie above 100Mb/s) typically get connected to our Witless set of LNSs. We will be doing some overnight 'shuffling' to move circuits between the LNSs this week and next week. This will happen overnight and involves forcing lines to reconnect.
Resolution: This work has finished.

MAINTENANCE Completed Drops
AFFECTING
Drops
STARTED
Nov 13, 01:41 PM (15¼ days ago)
CLOSED
Nov 13, 01:45 PM (15¼ days ago)
DESCRIPTION
A number of customer circuits have dropped and are in the process of reconnecting, we're investigating.
Resolution: All services restored by 13:48. There have been a couple of drops like today's recently. The underlying cause is being investigated. We've applied a short term fix that should hopefully prevent further issues.

MINOR Closed Logouts
AFFECTED
Logouts
STARTED
Nov 10, 08:35 AM (18¼ days ago)
CLOSED
Nov 10, 09:00 AM (18¼ days ago)
DESCRIPTION
At around 8:35 one of our LNSs restarted unexpectedly and caused lines to drop and reconnect.
Resolution: Service is now back to normal. We do apologise to those who had their service interrupted.

MAINTENANCE Completed General Internet Access
AFFECTING
General Internet Access
STARTED
Nov 02, 12:01 AM (26¾ days ago)
CLOSED
Nov 02, 09:16 AM (26¼ days ago)
DESCRIPTION
What we are doing: Router Upgrade Description of the works taking place: Upgrading router firmware as per vendor recommendations to enable new features and required levels for support.
Resolution:

MAINTENANCE Completed All BT Circuits
AFFECTING
All BT Circuits
STARTED
Nov 08, 12:01 AM (20¾ days ago)
CLOSED
Nov 02, 02:45 PM (26 days ago)
DESCRIPTION

We have multiple interlinks with BT in separate datacentres, however, BT need to make a change to their Core Network that will unfortunately impact all BT services. (Etherway/Etherflow services are not affected)

BT are simultaneously upgrading some 21C network software across multiple sites. The window is 8th November 00:01 until 07:00, and will start at 00:01 with non-disruptive pre-checks. From 01:00 BT will start to re-boot the devices to the new version of code. This will cause an outage to all customers of up to 15 minutes whilst the device restarts. However, if roll-back is required, the outage could exceed 1hr. All work will be complete by 07:00.


Resolution: BT will be rescheduling this, they say "This is to advise you that this Change on the BT UK Core Network is being rescheduled and will NOT take place on 08/11/2023 00:01."

MINOR Closed SIP2SIM
AFFECTED
SIP2SIM
STARTED
Nov 01, 04:00 PM (27 days ago)
CLOSED
Nov 01, 04:26 PM (27 days ago)
DESCRIPTION
Our mobile carrier for our SIP2SIM service is experiencing problems which is affecting the service. (Making calls may not be working at the moment)
Resolution:

MAINTENANCE Completed TalkTalk
AFFECTING
TalkTalk
STARTED
Nov 01, 12:05 AM (27¾ days ago)
CLOSED
Nov 01, 07:36 AM (27½ days ago)
DESCRIPTION
TalkTalk have emergency work to reboot some of their equipment. This is happening at short notice, between midnight and 5AM on 1st November.
Resolution: Traffic has been restored, and there was no impact to customers.