Order posts by limited to posts

30 Nov 2017 19:08:34
Posted: 22 Nov 2017 11:43:08
We are seeing a denial of service attack, which is causing more problems that usual, and this is disrupting traffic to some customers - but it is moving, and so different customers at different times. Obviously we are working on this, and unfortunately cannot say a lot more of the details.
22 Nov 2017 12:15:03
At present, customers using relayed L2TP services (including data SIMs) to external IP addresses, and those accessing VoIP from external IP addresses, may be experiencing issues. IPv4 only - if you are able to use IPv6, please do. Access from customer lines should be unaffected.
22 Nov 2017 12:31:19
Services are returning to normal for now. Thank you for your patience.
22 Nov 2017 12:40:35
During the attack, external access to our LNSs (L2Tp relayed services) and VoIP may be limited. It is back at the moment.
22 Nov 2017 12:59:48
As a side effect of today's DoS attacks, we're also experiencing intermittent failures of the telephones in the office - customers may experience difficulties placing calls into our office. We are aware and working on restoring this as soon as possible.
22 Nov 2017 13:04:32
I am sorry we cannot go in to huge amount of detail on this, but we essentially having to go in to a sort of lockdown on external access to key services during an attack. We will be doing more planned work over the coming days to make this less and less disruptive, obviously, but at the end of the day a denial of service attack can impact any network of any size.
22 Nov 2017 13:14:42
We're still monitoring carefully, obviously...
22 Nov 2017 14:23:38
They are back, shields up!
22 Nov 2017 15:08:32

VOIP customers accessing from non A&A addresses may want to restart asterisk if that is what you are using or restart your client.

L2TP relay customers should now be working

25 Nov 2017 19:56:42
Brief issues on one LNS this evening, but sorted now.
Resolution For now, we are treating this as resolved.
Started 22 Nov 2017 11:30:00
Closed 30 Nov 2017 19:08:34

22 Nov 2017 12:08:14
Posted: 21 Nov 2017 12:57:30
We are seeing another Denial of Service attack, currently this is affecting A, B, C, D and E gormless intermittently and will manifest itself as packet loss and/or dropped sessions for customers on these LNSes.
We are investigating.
21 Nov 2017 13:34:06
We are going to update some routers today as well, this should not cause any additional impact.
21 Nov 2017 14:21:34
All seems quiet over the last hour.
22 Nov 2017 11:44:02
It appears that the attackers are back. Broadband lines and Ethernet customers may experience difficulty accessing the internet .
Resolution Covered by another status posted
Started 21 Nov 2017 12:50:00 by AA Staff
Closed 22 Nov 2017 12:08:14

20 Jul 2017 12:34:33
Posted: 20 Jul 2017 12:20:48
We're investigating a routing problem that started a few minutes ago affecting broadband and general routing across our network.
20 Jul 2017 12:24:33
This is affecting our services between London and maidenhead, so is affecting some Ethernet circuits too.
20 Jul 2017 12:33:34
Things are getting back to normal now.
Resolution Unfortunately, this was caused by human error in a configuration on one of our core switches. The work was being done as part of our investigations to the problems we had last week, and this change was really not meant to cause any issue, except a mistake was made in the configuration. The change was rolled back and the process is being reviewed.
Started 20 Jul 2017 12:15:00
Closed 20 Jul 2017 12:34:33

14 Jul 2017 17:00:47
Posted: 14 Jul 2017 13:16:20
The issue affecting broadand services has also been affecting hosted servers and our own services in Maidenhead. This has also has some issues with Ethernet customers.
14 Jul 2017 14:43:15
Updates are being posted in https://aastatus.net/2411
14 Jul 2017 17:00:53
The network has been stable for a good few hours now. One of our 40G Telehouse-to-Harbour Exchange interconnects has been taken down and some devices have been moved off of the suspect switch. We have further work to do in investigating the root cause of this and what we plan to do try to stop this from happening again. We do apologise to our customers for the disruption these two outages have caused and we work on trying to prevent this from happening again.
Started 14 Jul 2017 10:00:00
Closed 14 Jul 2017 17:00:47

09 Feb 2017 13:00:00
Posted: 09 Feb 2017 12:54:39
There appears to be a power outage in maidenhead. This is affecting all our VOIP and Email services. More information to come.
09 Feb 2017 12:59:03
Power is back. VOIP and Email services have been restored. Postmortem to come.
Resolution Power was restored within a few minutes - we do apologise for this unexpected power outage - the cause is suspected to be a faulty network switch which tripped the circuit breaker when it was lightly touched by one of our staff in the datacentre! The switch has been powered off and will be removed from the datacentre out of hours.
Started 09 Feb 2017 12:52:45
Closed 09 Feb 2017 13:00:00

20 Jul 2016 12:41:10
Posted: 20 Jul 2016 08:18:13
We can see a major routing issue, and we are investigating. Not entirely clear the extent of the problem at this stage.
20 Jul 2016 08:21:55
At the moment we are seeing what appears to be an issue with the links between London and Maidenhead. This can impact some Ethernet customers, VoIP, hosting, etc. However, a number of systems have a fallback for this, so many are still working, e.g our VoIP services are OK.
20 Jul 2016 08:22:23
The link is partially back, and we are investigating the cause.
20 Jul 2016 10:19:15
This is looking like a major fault within BT's core network. This is affecting many ISPs in the UK. We'll post more details when we can.
20 Jul 2016 10:19:48
also see http://www.bbc.co.uk/news/technology-36844712 and http://www.ukinternetreport.co.uk/?p=d&id=215 Which are reporting the problem within BT's network.
20 Jul 2016 10:30:51
This fault is affecting our communications with BT's systems. This means that our sales and support staff have very limited access to BT's ordering, diagnostic and fault reporting problems. so please to bear with us as staff will be struggling to do their work at the moment. (TalkTalk lines are ok though)
20 Jul 2016 10:35:40
This is looking related to a power issue at a major london datacentre
20 Jul 2016 10:48:05
We've had an update from BT regarding this issue, they say: There is an intermittent incident affecting the Openreach B2B and XML Gateways, as a result of this issue you may be experiencing a delay in the receipt of KCI's for both Fulfilment and Assurance requests as well as failures in Dialogue Services.
20 Jul 2016 12:40:00
It looks like everything affected by this outage has returned to normal now.
20 Jul 2016 12:40:06
It looks like everything affected by this outage has returned to normal now.
20 Jul 2016 12:40:23
It looks like everything affected by this outage has returned to normal now.
Started 20 Jul 2016 08:04:00
Closed 20 Jul 2016 12:41:10

19 Nov 2015 11:49:24
Posted: 18 Nov 2015 21:13:36
We are investigating a network issue affecting most customers with packet loss - this impacts broadband, ethernet, VoIP, and all services.
18 Nov 2015 21:50:44
We're still working on this....
18 Nov 2015 22:23:32
We are being severely DOS'd, our usual anti-DOS systems have not helped this time due to the nature of the attack. Needless to say, we are working to mitigate this.
18 Nov 2015 22:40:25
Things are not perfect, but we have mitigated most of the problems
19 Nov 2015 00:11:00
The network has been stable for over an hour now. There are are a very small number of customers who are offline. We will be contacting them in the morning, However do call our Support Team from 8am.
19 Nov 2015 07:20:56
Whilst service is mostly stable, we are going to have to do more work on this during the day. We'll try and keep disruption to a minimum.
19 Nov 2015 09:03:54
The DDOS has moved to a new target block which has caused more problems again - we are working on this still, but service is improving again.
19 Nov 2015 09:04:57
Looking good now - a handful of customers off line totally but we are working on that now.
19 Nov 2015 09:19:29
Side Note: This wasn't an attacks against our network or servers specifically, it wasn't a 'hacking' attempt like we've seen in the news recently. This was a denial of service attack, probably, against a particular customer. We may not get to the bottom of why this customer was attacked but we are investigating.
19 Nov 2015 10:02:26
Some customer are having their WAN IP address changed to get them on-line. Some customers with blocks of IPs are being contacted regarding a change of IP address block. Obviously if you need the IP to be reinstated once this issue is resolved, we are happy to do that.
19 Nov 2015 11:31:50
Looking a lot better now - monitoring carefully. Impacted customers should now be on line.
Resolution Whilst the attack has now stopped, we have a number of things we still need to do. Thank you for your patience and understanding. More details http://aa.net.uk/news-20151119-dos.html
Started 18 Nov 2015 20:50:00
Closed 19 Nov 2015 11:49:24

06 May 2015 14:03:19
Posted: 05 May 2015 16:29:43
Our office internet is currently offline. We're arranging our backup connection at the moment.
05 May 2015 16:55:40
We're making progress on our backup link as well as investigating the cause of the main fault.
05 May 2015 18:57:51
We do apologise about our connectivity problems this afternoon. We have been running on our backup FTTC line, and we're investigating why the direct fibre is down.
06 May 2015 10:34:33
Our main internet connection is still down this morning. We have a number of BT engineers working on this at the moment. This does mean that we are running on our backup FTTC links and most things are working OK! Some telephone calls are a little temperamental, we'd appreciate customers using IRC or email where possible, see: http://aa.net.uk/kb-irc.html for details of connecting to IRC. We hope to be back to normal later this morning.
06 May 2015 11:57:53
BT have identified 2 faults with our fibre. They have fixed one of them (a kink in a fibre patch lead in Bracknell exchange) and are investigating the second (low light level on one leg).
Resolution We're all back to normal now!
Started 05 May 2015 16:25:00
Closed 06 May 2015 14:03:19

25 Sep 2014 12:07:57
Posted: 25 Sep 2014 11:48:00
We are investigating a network problem affecting our offices.
25 Sep 2014 11:51:31
This is affecting our telephones.
25 Sep 2014 12:08:45
The office is back online. We had lost IPv4, we're looking in to the cause of this.
Started 25 Sep 2014 11:47:00
Closed 25 Sep 2014 12:07:57

25 Aug 2014 23:49:30
Posted: 25 Aug 2014 22:15:51
We are seeing what looks to be routing problems within our network with traffic to/from our Maidenhead datacentre. Routes seem to be flapping and disrupting connectivity with increased latency and packet loss. This would be affecting Ethernet services from Maidenhead as well as customers accessing web and email services that we host in Maidenhead. Customers are also reporting DNS problems.
25 Aug 2014 22:19:09
Engineers are investigating...
25 Aug 2014 23:33:53
Staff are still working on this. The cause of the problem has been identified and is being worked on.
25 Aug 2014 23:50:13
The problem has been resolved, traffic is now back to normal, we apologise for this inconvenience.
Started 25 Aug 2014 21:45:00
Closed 25 Aug 2014 23:49:30

25 Jun 2014 06:28:18
Posted: 25 Jun 2014 05:39:05
The minor change last night is still having some issues with some servers being slow, and this is currently impacting VoIP. This is still being working on.
25 Jun 2014 06:07:24
VoIP services starting to work properly again, still not 100% right.
25 Jun 2014 06:15:04
VoIP looking a lot healthier now. Still working on tracking down the poor performance issues.
25 Jun 2014 08:35:02
For more details on what happened, please see https://www.facebook.com/AAISP/posts/673421826086051
Resolution We'll be monitoring this closely during the day.
Started 25 Jun 2014 05:38:24
Closed 25 Jun 2014 06:28:18

22 Feb 2014 13:00:00
Posted: 22 Feb 2014 08:03:44
A disk server has failed, it is impacting all web sites we host and email. Engineers are working on this now.
22 Feb 2014 10:08:38
There is a major issue with one of the disk servers, and we are planning to switch to a backup, but that is likely to involve an engineer visit to the data centre.
22 Feb 2014 10:16:44
Engineer is on his way to the data centre now.
22 Feb 2014 11:15:40
This is looking more complex than expected - we have switched the secondary controller, but there are issues with one of the disk arrays as well. Engineer still on site.
22 Feb 2014 11:17:48
Disk array is rebuilding now. We should have email working shortly and then web pages once the disk array rebuilds.
22 Feb 2014 11:57:48
Web space up, and mail servers being reconnected to disk array now.
22 Feb 2014 12:11:56
Issues with web pages again, investigating.
22 Feb 2014 12:14:17
The secondary disk server is now showing problems too. We are working on it.
22 Feb 2014 12:32:03
This is proving to be quite a serious issue - we appear to have issues with two separate disk controllers and with some of the RAID disks and with the file system on one of the disks. This is a very odd multiple failure, especially given that all of this is monitored constantly and was not showing any issues yesterday. We do have daily backups, so if all else fails there are ways to get service restored with backups and some loss of recent emails or changes. At this stage we are working to repair the failed file systems before considering that move.
22 Feb 2014 12:35:22
It looks like we have the mail file store repaired and mail should be back on line shortly.
22 Feb 2014 12:38:27
Web pages back.
22 Feb 2014 12:40:20
Incoming email should now be working again.
22 Feb 2014 12:44:44
We are checking all mail and web servers now to confirm all is well again.
Resolution Obviously this sort of multiple failure is somewhat unexpected. We do have plans for new disk servers anyway, and this type of failure will be considered as part of that system design.
Started 22 Feb 2014 00:39:00
Closed 22 Feb 2014 13:00:00
Previously expected 22 Feb 2014 13:00:00

25 Sep 2013 16:20:00
Posted: 25 Sep 2013 16:09:44
We're currently investigating a problem with our disk storage server that runs our email and web space. We are investigating.
25 Sep 2013 16:18:26
The server is being restarted at the moment.
25 Sep 2013 16:20:41
Ser server is back on line, we are doing post-boot up checks etc to being the services back online.
25 Sep 2013 16:20:58
Web services are now back online.
Started 25 Sep 2013 16:00:00
Closed 25 Sep 2013 16:20:00

17 Feb 2013 07:57:04
Posted: 17 Feb 2013 06:51:33

Our web site, and hosted customer web sites are not responding. We are looking in to this.

17 Feb 2013 07:32:53

This issue is the same as the one affecting email, and is related to a disk server.

17 Feb 2013 07:43:06

We are working on disk server now


We sites now working.

Started 17 Feb 2013 06:22:30
Closed 17 Feb 2013 07:57:04

11 Feb 2013 12:53:12
Posted: 11 Feb 2013 12:40:11


We've just had a blip causing ADSL lines to drop of and reconnect, this also affected routing for about a minute.

More info to follow shortly.


11 Feb 2013 12:53:12

The issue affected both BGP links and LNS traffic, cauing lines to reconnect. Not all lines and services were affected. The lines have reconnected quickly, and the problem seems to have cleared without any need for any intervention from staff.

We are investigating the cause - this looks to be something external that has triggered this. We are trying to find exactly how this has happened so we can avoid the issue in future.

General Users Affected 33%
Closed 11 Feb 2013 12:53:12

26 Jan 2013 04:59:58
Posted: 26 Jan 2013 04:56:46

We have lost access to everything in HEX 6/7 which means L2TP only customers are off line. This also affects some of our general servers. This is being investigated now.

26 Jan 2013 04:57:41

The issue is a result of a port/link failure earlier. We are re-configuring access via an alternative route now.


Access has been restored

Started 26 Jan 2013 04:31:00
Closed 26 Jan 2013 04:59:58

09 Apr 2012 17:00:43
Posted: 09 Apr 2012 16:23:00

Our main web pages and customer web pages are currently inaccessable or very slow due to a denial of service attack. We are taking steps to address this. Sorry for any inconvenience.

09 Apr 2012 16:26:34

This appears to be an attach on one web site, which is being moved.

09 Apr 2012 16:31:17

Load is starting to move to the backup server now.

09 Apr 2012 16:44:32

We have moved a lot of the load over, and adjusted TCP settings to try and ensure we sites are working albeit slightly slow.

09 Apr 2012 16:59:12

Web access is being much more responsive now.

Started 09 Apr 2012 14:36:19
Closed 09 Apr 2012 17:00:43

14 Feb 2012 13:06:19
Posted: 13 Feb 2012 12:37:17
We currently have a DNS problem with our main domains aa.net.uk and aaisp.net.uk - this will cause a problem with various services that we run.
13 Feb 2012 12:45:36

This was a simple planned change - it seems however we have been caught out by bind refusing to reload if one zone has one syntax error. Seems one of our customer zones had a typo in it which our tools had not managed to check and that caused it to refuse to load.

Unfortunately it took quite a few minutes to find what was wrong as this was nothing to do with the changes we actually made.

13 Feb 2012 12:55:05

Most things are OK.

13 Feb 2012 13:13:38

Both authorative serves are serving the aaisp.net.uk and aa.net.uk zones correctly.

13 Feb 2012 13:39:09

There are still problems with the aaisp.net.uk domain - we're working on this.

VoIP is currently affected too, this is being worked on now.

13 Feb 2012 13:56:00
We are running in to slightly unexpected errors as well, and working through them. Some things that we have not touched are not working, which kind of makes no sense.
13 Feb 2012 14:43:27

Whilst most things are working, at least from customer lines, there was an issue which caught us out, but was way to far in to the process to sensibly back up. The top level delegation for aaisp.net.uk was going to a special DNS server, which meant when we moved everything it stopped working properly.

13 Feb 2012 14:46:17

Some incoming VoIP is not working, we are still working on this.

13 Feb 2012 15:09:53

Still lots of progress mopping up things. Mostly non customer affecting.

Just to clarify - you should be able to use aaisp.net.uk or aa.net.uk as they interchangable. However, right now, some places are having some issues seeing some of the aaisp.net.uk sub domains.

The preferred version is now aa.net.uk and we should be quoting that everywhere now.

Using aa.net.uk is also a work around for the issues some people are seeing right now, which are down to DNS caches.

13 Feb 2012 16:18:42

At the moment we still have some services affected, these are: some incoming VoIP and accessing our services from outside our network (ie customer using DNS resolvers ither than ours)

13 Feb 2012 17:38:56

We really think this should be sorted now - but monitoring carefully.

13 Feb 2012 18:25:30

Mor details on http://aa.net.uk/news-2012-02-dns.html

14 Feb 2012 13:06:34

We think all is OK now, so closing this incident.

Started 13 Feb 2012 12:15:49
Closed 14 Feb 2012 13:06:19

22 Jan 2012 08:32:57
Posted: 22 Jan 2012 07:57:48

It looks like we have a major issue with mail and web services.

This is being investigated now.

22 Jan 2012 08:04:01

This looks like an issue with the main storage array used by web and email. Being worked on now.

22 Jan 2012 08:31:56

Looks like we have it working again - web pages are fine - email being a bit sluggish catching up.


The disk server has been restarted to clear the problem. The underlying cause of the problem is being investigated.

Started 22 Jan 2012 06:45:00
Closed 22 Jan 2012 08:32:57

11 Nov 2010 11:59:24
Posted: 11 Nov 2010 11:53:30

Routing blipped briefly - we're investigating.

11 Nov 2010 12:00:39

Routing is back now.

Total downtime was from about 11:45 to 11:53

We're investigating the cause of this problem.

Started 11 Nov 2010 11:50:28
Closed 11 Nov 2010 11:59:24

26 Oct 2010 23:06:25
Posted: 26 Oct 2010 20:57:35

We are seeing problems with equipment located in our HEX datacentre, most likely power related.

This will be affecting A&A webspace, customer L2TP connections, beta tester data SIM connections and some customer equipment which is hosted there.

We have been in contact with the data centre staff, who are investigating it.

26 Oct 2010 21:32:34

The problem doesn't look like power, the core routers we have there appear to be rebooting.

Power cycling has restored the service to one of the routers, so access to servers hosted there is working again now. However the second router remains offline and is still causing problems with data SIM testers.

We are working to resolve that problem too.

26 Oct 2010 21:33:04

The main issues have been resolved now - it does not look like power, but trying to find why two separate routers developed problems at the same time in HEX. Data SIMs will be affected still.

26 Oct 2010 22:09:33

We have re-routed the data SIMs and set up backup for future use anyway.


Both routers working now.

Started 26 Oct 2010 20:38:00
Closed 26 Oct 2010 23:06:25

22 Sep 2010 09:30:00
Posted: 22 Sep 2010 08:50:08

Investigating now - if this is the same as we had at the weekend we should be able to sort it quite quickly.

22 Sep 2010 08:58:53

We hope to have this sorted in a few minutes.

22 Sep 2010 09:03:14

This is impacting some VoIP services but not all.

22 Sep 2010 09:11:09

There will be a slight blip on broadband while we sort this.

22 Sep 2010 09:13:04

This looks like some issue with routing through LINX. We may take down the route collector peering until we are happy we have identified the cause.

22 Sep 2010 09:16:25

Still seeing some issues.

22 Sep 2010 09:21:03

Equipment reboot worked briefly and then the problem re-occured. It seems clear this is a routing issue with a peer that is causing a black hole. We do not understand exactly where or how yet and this is being addressed.

22 Sep 2010 09:35:38

We have taken town LINX route server peering and things are looking a lot better - checking things now.

22 Sep 2010 09:46:39

It may be worth explaining this a little. We have dual redundent equipment to allow for failures. If something fails completely, or can be turned off, then the systems re-route to use other equipment. Depending on where such issues are this can mean no outage, a few seconds or a few minutes.

However, if there is a partial failure, such as a single black-hole route for the link to Maidenhead, then this is not an equipment failure. The other routers get that route and expect it to be valid. This can create complex problems that are hard to diagnose, also and mean we have to use various alternative means to access systems which causes delays.


I would stress, just because taking down the LINX route server seems to have addressed the issue does not mean there is an issue with LINX. This could be something odd with our routers, or the LINX router server or a peer via that route server feeding something odd to us as a route. We're trying to identify what has happened but for now we'll leave the route server shutdown until we know.

Started 22 Sep 2010 08:37:00
Closed 22 Sep 2010 09:30:00

21 Jul 2010 15:06:47
Posted: 21 Jul 2010 14:26:21

There is a general issue with packet loss at the moment.

It looks like it might be related to transit, and we're investigating.

21 Jul 2010 14:42:46

Still investigating. Looks like a problem with peering.

Something's recovering - traffic levels are returning to normal.

21 Jul 2010 14:43:52

Looks like a power failure in Telehouse North, which would have affected a lot of peering, and some transit.

21 Jul 2010 15:07:17

It looks like everything is back to normal for us.

Started 21 Jul 2010 14:23:47
Closed 21 Jul 2010 15:06:47

27 May 2010 20:20:26
Posted: 27 May 2010 19:29:03

We have been trying to resolve this properly and run in to a further snag which means again we have no routing to servers in HEX. Broadband unaffected as before but many services like control pages, hosted servers, our web site, accounts, and so on are all off-line while we reoslve this.

27 May 2010 19:44:13

We are waiting for someone on-site at present.

27 May 2010 20:20:26

Ok, working again.


Trying to find the exact problem for a permanent fix.

Started 27 May 2010 19:27:00
Closed 27 May 2010 20:20:26

27 May 2010 17:26:33
Posted: 27 May 2010 17:13:39

We have a major issue with a firewall/router in HEX at present which will affect access to web pages, control pages, accounts system, and variosu systems. We hope to have this resolved ASAP.


Routing has been reworked - some smaller issues remain.

Started 27 May 2010 17:16:37
Closed 27 May 2010 17:26:33

08 Apr 2010 08:30:52
Posted: 08 Apr 2010 01:35:23

We seem to have lost access to machines in Harbour Exchange Square.

This includes one of our key machines for RADIUS authentication for broadband lines (clueless) and our accounts server (priceless), customer web servers (Limitless) and a few hosted machines.

We are trying to identify the cause of the problem now.

08 Apr 2010 01:46:48

We have managed to confirm it is not a power issue

08 Apr 2010 01:56:38

We are gettign someone to check the rack in HEX now.

The backup RADIUS and DNS servers are working as they should.

08 Apr 2010 02:05:02

This appears to be an issue with our main firewall in HEX. It is firewalling rather too well all of a sudden.

08 Apr 2010 02:12:13

We are just waiting on a reboot now. We believe we have found the cause of the issue though so can take some preventative measures once the reboot is complete.

08 Apr 2010 02:37:29

Reboot complete - all working

08 Apr 2010 08:30:52

One slight side effect - session tracking timed out as the RADIUS server could not see the LNS. This meant PPP restarts for most lines at some point during the night and a few may do so during the day (on the hour). Until then usage is not being meter for those lines.


We will be applying an update to the firewall shortly

Started 08 Apr 2010 00:59:10
Closed 08 Apr 2010 08:30:52