Recent posts
Timeline view of events on our network and systems

Events from the AAISP network from the last few months on a scrollable timeline. Mouseover for brief details, click incident to view the full post.

MAINTENANCE Assumed Completed BT
AFFECTING
BT
STARTING
Feb 11, 12:01 AM (2½ hours )
DESCRIPTION

BT have a series of planned work that involves them replacing cable bundles that include our hostlinks - our interconnects into BT's network. We're not sure when these will take place, but will happen between 00:30 and 02:00 and will happen over three nights. This will cause 'blips' for many customers - lines will drop and reconnect within seconds or a couple of minutes.

We would usually be able mitigate any disruption by cleaning moving our traffic off the affected links. However, at the moment BT have not specified which link and the date that this work will happen. Thanks BT!


MINOR Closed BT
AFFECTED
BT
STARTED
Feb 10, 12:32 AM (20¾ hours ago)
CLOSED
Feb 10, 12:42 AM (20½ hours ago)
DESCRIPTION
at around 00:32 a number of BT circuits dropped and reconnected within a couple of minutes.
Resolution:

MAINTENANCE Completed L2TP
AFFECTING
L2TP
STARTED
Feb 04, 03:15 AM (6¾ days ago)
CLOSED
Feb 04, 01:30 PM (6¼ days ago)
DESCRIPTION
On Wednesday 4th February, at 3:15AM we will be upgrading the software on the other L2TP server (l2tp.aa.net.uk). We upgraded the first one last week. This will cause some L2TP customers to drop and reconnect twice as this happens. (Twice, as whilst the router is upgraded connections will reconnect to the other L2TP server, and then will moved back once the upgrade has completed.)
Resolution: This was completed.

MAINTENANCE Completed BT
AFFECTING
BT
STARTED
Jan 29, 12:01 AM (12¾ days ago)
CLOSED
Jan 29, 07:25 AM (12½ days ago)
DESCRIPTION
BT have planned work scheduled on one of our 'hostlinks' - the cable service that connects us to BT. We run multiple hostlinks so we will take the one that is being worked on out of service for the period. This should mean that the service is unaffected.
Resolution: This has been completed with no effect on customers.

MAINTENANCE Completed L2TP
AFFECTING
L2TP
STARTED
Jan 28, 03:15 AM (13¾ days ago)
CLOSED
Jan 28, 08:00 AM (13½ days ago)
DESCRIPTION
On Wednesday 28th January, at 3:15AM we will be upgrading the software on one of our L2TP servers (l2tp.aa.net.uk) This will cause some L2TP customers to drop and reconnect twice as this happens. (Twice, as whilst the router is upgraded connections will reconnect to the other L2TP server, and then will moved back once the upgrade has completed.)
Resolution: This work was competed. We do see a very small number of L2TP clients which look like they are unable to connect. This may not be a new problem though, so please do get in touch if this has affected you.

MINOR Closed 07 Numbers
AFFECTED
07 Numbers
STARTED
Jan 27, 12:30 PM (14¼ days ago)
CLOSED
Jan 29, 10:29 AM (12¼ days ago)
DESCRIPTION
One of our carriers is currently experiencing connectivity issues with their service (again). Their engineering team is investigating this issue.
Resolution: Closing for now, as calls have been working ok for the past 22 hours.

MINOR Closed 07 Numbers
AFFECTED
07 Numbers
STARTED
Jan 24, 06:00 PM (17 days ago)
CLOSED
Jan 24, 09:25 PM (16¾ days ago)
DESCRIPTION
One of our carriers is currently experiencing connectivity issues with their service. Their engineering team is investigating this issue.
Resolution: Carrier say services restored at 21:25, 24th January. We'll be asking for their RFO.

MAINTENANCE Completed City Fibre
AFFECTING
City Fibre
STARTED
Jan 24, 12:01 AM (17¾ days ago)
CLOSED
Jan 24, 11:00 AM (17¼ days ago)
DESCRIPTION
Capacity Upgrade Cambridge - Customers will experience an outage of up to 60 minutes during the maintenance window.
Resolution:

MAINTENANCE Completed BT
AFFECTING
BT
STARTED
Jan 20, 12:01 AM (21¾ days ago)
CLOSED
Jan 20, 06:46 AM (21½ days ago)
DESCRIPTION
BT have planned work scheduled on one of our 'hostlinks' - the cable service that connects us to BT. We run multiple hostlinks so we will take the one that is being worked on out of service for the period. This should mean that the service is unaffected.
Resolution: BTs work was carried out, and our customer traffic and connections were not affected. (There is similar work on Jan 29th on our other pair of hostlinks https://aastatus.net/42817)

MINOR Closed Data SIMs
AFFECTED
Data SIMs
STARTED
Jan 19, 10:18 PM (21¾ days ago)
CLOSED
Jan 20, 06:47 AM (21½ days ago)
DESCRIPTION
At 22:18 we had a large number of data SIMs drop their connection. They are reconnecting at the moment, but it looks like some are struggling with their connection. This looks like an upstream problem - ie within the mobile network or the network operator.
Resolution: Just to provide more clarification on this, our carrier's network team was carrying out works on one of their routers. This was not expected to affect Data SIM routing, but had an unforeseen knock-on. They've identified the root cause and are taking steps to rectify this.

MINOR Closed BT
AFFECTED
BT
STARTED
Jan 19, 10:25 AM (22¼ days ago)
CLOSED
Jan 19, 10:20 AM (22¼ days ago)
DESCRIPTION
At 10:25 we say a small number of BT lines drop and reconnect - we're not sure of the cause yet, but are investigating.
Resolution:

MINOR Closed 07
AFFECTED
07
STARTED
Jan 16, 06:00 PM (25 days ago)
CLOSED
Jan 30, 05:00 PM (11 days ago)
DESCRIPTION
SMSs from EE to our 07 range seem not to be working. Our tests show the fail the be accepted by EE when sending, and the phone gives an error. This has been reported to both EE and our mobile carrier (who have also passed this on to EE).
Resolution: EE and our customers confirmed that this was fixed on Friday afternoon.

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Jan 15, 10:13 AM (26¼ days ago)
CLOSED
Jan 15, 10:29 AM (26¼ days ago)
DESCRIPTION
At around 10:13 there was a few minutes where out VoIP service was interrupted. This would have upset making of new calls and registrations.
Resolution: The problem was resolved by 10:18.

MINOR Closed VoIP
AFFECTED
VoIP
STARTED
Jan 14, 05:10 PM (27 days ago)
CLOSED
Jan 16, 10:24 AM (25¼ days ago)
DESCRIPTION
We've had reports that calls from Vodaphone to some of VoIP numbers failing. We are contacting the carrier involved to investigate this issue with Vodafone.
Resolution: This problem seems to have gone away, but with no actual cause of the problem being identified.

MAINTENANCE Completed BT
AFFECTING
BT
STARTED
Jan 12, 12:01 AM (29¾ days ago)
CLOSED
Jan 12, 09:00 AM (29½ days ago)
DESCRIPTION
BT have notified us of planned work scheduled on ALL of our 'hostlinks'. However, we do not believe this work will be service affecting. This post is here on the chance that it will affect services.
Resolution: This was a miss-communication and no work was carried out.

MAINTENANCE Completed Amazon
AFFECTING
Amazon
STARTED
Dec 02, 09:30 AM (2¼ months ago)
CLOSED
Jan 16, 03:04 PM (25¼ days ago)
DESCRIPTION
We are in a position where we are now able to re-enable one of our interconnects to Amazon/AWS. This will move most Amazon traffic away from our peering with them over LINX to the direct interconnect we have. This is planned for Tuesday morning at around 10AM. (This was scheduled for Friday 28th, but as it was Black Friday, we've moved to Tuesday)
Resolution: Traffic has been flowing over one of our new Amazon links today ok. We consider this upgrade completed.

MINOR Closed Email
AFFECTED
Email
STARTED
Feb 03, 09:00 AM (1 year ago)
CLOSED
Feb 06, 01:00 PM (1 year ago)
DESCRIPTION

Over the past week we have seen a huge number of 'bots' trying to guess customer email credentials in order to try to send email through our outbound email servers: smtp.aa.net.uk.

The attempts were being blocked due to wrong passwords being used, but this caused significant load on our severs due to all the database lookups involved. To address this, we are blocking IP addresses that are listed on the Spamhaus 'Exploits Blocklist (XBL)' and the Spamhaus 'Combined Spam Sources (CSS)' lists. -These are typically IP address known to have hijacked in some way or known spam senders.

This has reduced the load on the email servers significantly, however we are are still blocking around 1.5 million unique IP addresses each day.

We have had a small number of legitimate customers affected by this as their IP address is on these blocklists. (IPs can be looked up on https://check.spamhaus.org). In these cases, please do contact support and we can discuss workarounds.


Resolution: