Order posts by limited to posts

Thursday 02:00:00
Details
Posted: 23 Mar 11:04:21
On the 26/04/18 between 2am and 6am BTW have planned works to migrate customers connected via the Bristol POSI (Point of service Interconnects) nodes to new hardware. The work is expected to take 25 - 45 minutes.
Planned start Thursday 02:00:00 by Carrier
Update expected Thursday 10:00:00
Expected close Thursday 06:00:00 (Estimated Resolution Time from Carrier)

Yesterday 10:00:00
Details
Posted: 18 Apr 11:04:49

We are finally able to move all our TalkTalk circuits on the the new TalkTalk network. We've had customers testing the new network since November last year; the new network brings increased capacity and increased resilience.

From Monday 23rd April, we will start moving small batches of lines over to the new network by forcing a PPP reconnect. This will happen overnight and we expect this to take 11 nights to complete.

Update
Yesterday 10:10:08
TalkTalk have completed their work and new PPP logins from now on will connect over their new network. Existing connections will carry on working, but a PPP reconnect will connect over the new network. Starting from tonight we will be moving over batches of lines as part of our LNS upgrades which are happening. https://aastatus.net/2518
Started Yesterday 10:00:00
Expected close 2 May 09:00:00

13 Apr
[Broadband and Ethernet] Router upgrades - Open
Details
Posted: 12 Apr 18:26:04
Some more router upgrades are planned, and then we expect to do a rolling update or LNSs soon. As usually router upgrades should be no disruption and LNS upgrades being a PPP restart over night.
Update
Yesterday 10:40:40
We are running a rolling LNS upgrade from tonight.
Started 13 Apr
Expected close 6 May

09 Mar 2017 20:00:00
Details
Posted: 08 Mar 2017 12:29:14

We continue to work with TalkTalk to get to the bottom of the slow throughput issue as described on https://aastatus.net/2358

We will be performing some routing changes and tests this afternoon and this evening, we are not expecting this to cause any drops for customers, but this evening there will be times when throughput for 'single thread' downloads will be slow. Sorry for the short notice, please bear with us, this is being a tricky fault to track down.

Update
08 Mar 2017 22:39:39
Sorry, due to TalkTalk needing extra time to prepare for their changes this work has been moved to Thursday 9th evening.
Started 09 Mar 2017 20:00:00
Update was expected 09 Mar 2017 23:00:00

19 Apr 08:00:00
Details
Posted: 28 Mar 09:14:11
On the 19/04/18 between 2am and 6am BTW have planned works to migrate customers connected via the Newcastle POSI (Point of service Interconnects) nodes to new hardware. The work is expected to take 25 - 45 minutes.
Started 19 Apr 02:00:00 by Carrier
Closed 19 Apr 08:00:00
Previously expected 19 Apr 06:00:00 (Last Estimated Resolution Time from Carrier)

17 Apr 23:00:00
Details
Posted: 16 Apr 19:13:47
Calls are currently failing to/from SIP2SIM products. Reports also indicate SMS failures.
Our upstream carrier is aware of the issue and currently has an open issue (ref 8273) regarding this issue.
Customers with a O2/EU SIP2SIM product may be able to place calls by manually selecting the roaming profile, but note that the handset will roam back to the O2 profile after a short time. No such workaround is currently known for regular SIP2SIMs.

Information from our upstream carriers reports that "USSD requests on the O2 profile are failing" and they are reporting with each update that the "02 MVNO team are investigating". Further updates are expected at 21:00
Update
16 Apr 20:47:36
As of 20:42, our upstream supplier has advised the following:
BT have a major transmission outage in Frankfurt which has taken down their links with Telefonica. Engineers are currently on site investigating
Update
16 Apr 21:46:49
Our upstream carriers have advised the following at 21:39 :
BT have confirmed that service has been restored Engineers are continuing to monitor
Update
16 Apr 23:46:46
As of approximately 22:35 we received customer reports that SIM2SIM is failing again.
An update has been received from our upstream carrier as of 23:42 and reads as follows:
Service had restored for a short while between roughly 20:30 - 22:30 BST, however engineer has confirmed that the service is down again. Engineers are continuing to investigate
Update
17 Apr 08:17:17
Overnight updates from Carrier:

06:46 BT are still seeing major signalling issues for 02 traffic. 02 and their carrier BT are working to resolve as soon as possible.

05:41 O2 continue their investigations with the support of their signalling provider BT.

04:37 O2 are continuing to investigate with the support of their signalling provider BT.

03:37 O2 are continuing to investigate with the support of their signalling provider BT.

02:30 O2 are continuing to investigate with the support of their signalling provider BT.

01:33 O2 continue to investigate with the support of their signalling provider BT.

00:36 O2 are continuing to investigate with the support of their signalling provider BT.

Update
17 Apr 08:55:54
08:27 (Update from Carrier) BT are still seeing major signalling issues for 02 traffic. 02 and their carrier BT are working to resolve as soon as possible.
Update
17 Apr 09:34:21
(Update from Carrier) There are some reports that the service is working again - this is unconfirmed my the carrier at the moment.
Update
17 Apr 10:34:37
10:18 (Update from Carrier) Service remains unstable and BT engineers are continuing to work to restore network stability.
Update
17 Apr 11:44:09
11:12 (Update from Carrier) Service remains unstable and BT engineers are continuing to work to restore network stability.
Update
17 Apr 14:02:43
13:48 (Update from Carrier) BT & O2 continue to work towards a resolution, service remains unstable at this time
Update
17 Apr 15:02:20
14:52 (Update from Carrier): Since late yesterday afternoon, O2 have been experiencing signalling issues, and due to the way O2 route their signalling traffic, this has been impacting the signalling traffic back to Manx Telecom. Manx Telecom have been in regular contact with O2 since the issue was first identified, and we have escalated our ticket within O2 to the highest possible level. O2 have in turn escalated their ticket within Telefonica. We continue to chase in order to obtain a timescale for a permanent fix to be implemented. As soon as we have a further update, we will let you know.
Update
17 Apr 19:58:41
18:13 (Update from carrier) BT & O2 continue to work towards a resolution, service remains unstable at this time
Update
17 Apr 23:00:57
22:10 (update from carrier) O2 have diverted their signalling traffic to another service provider to try resolve the issue. Engineers continue to monitor.
Update
17 Apr 23:45:14
23:42 (Update from Carrier) Service has remained stable since O2 diverted traffic. Engineers are continuing to monitor
VoIP and SIMs Users Affected 100%
Started 16 Apr 19:00:00 by Customer report
Closed 17 Apr 23:00:00
Cause Carrier

18 Apr 08:00:00
Details
Posted: 09 Nov 2017 15:47:52
"TalkTalk have built a new Juniper LTS solution in order to:
  • Increase the total subscribers per TalkTalk LTS
  • Increase of the throughput capacity per chassis
  • Assist with load balancing
  • Increase network resilience"

In other words, they have been working on a new network for some time which should see the end of the various minor congestion issues that we've been seeing and will mean the have ample capacity for the future.

Before we migrate all our customers over we are asking for some customers to test this new network. Testing simply means us making some changes and the customer changing their PPP username slightly. Everything else works the same.

If you would like to test, please send an email with your login to: trial@aa.net.uk and we'll get back to you with further instructions.

We are one of the first ISPs that will be using the new network and we expect to move all our customers over before the end of November but we have not set a date yet.

Update
22 Nov 2017 16:04:33
Customers on the TalkTalk trial will be moved to a new LTS this afternoon. The LTS is the router at TalkTalks end of our links to them. This involves a PPP re-connect and this will be carried out by TalkTalk. Your router shoudl log back in within a few seconds. This new LTS has newer software which we're testing to see if some of the bugs we've reported are fixed. We'll post more about this shortly.
Update
12 Dec 2017 09:08:54
Update: The new network on the whole is performing well. We have been working with TalkTalk engineers and Juniper developers to help fix some problems. The main problem is that some lines fail to establish PPP, we have implemented a workaround on our LNSs and Juniper working towards a permanent fix for this.
We don't yet have a date when we'll move all customers across to the new network. We do hope this will be before Christmas bust due to the holidays it may be early January.
Update
26 Jan 14:02:59
Due to TalkTalk needing to wait for new software releases from their vendor (Juniper), moving our TalkTalk circuits over to their new network won't happen until mid-March 2018. We are still happy to have customers on the trial.
Update
18 Apr 08:46:39
We will be migrating all TalkTalk circuits over to the new network from 23rd April. We'd like to thank the customers who have been testing the new network, it has been useful. We had identified a couple of issues and have worked with TalkTalk who in turn have worked with their vendor, Juniper, to fix these. Customers still using @a.8 and @a.9 logins should get in touch with our Support team to get moved back to the normal @a.1 and @a.2 once the migration has fully completed. (there will be a separate status post about this)
Started 09 Nov 2017 15:30:00
Closed 18 Apr 08:00:00

12 Apr 08:00:00
Details
Posted: 28 Mar 09:15:28
On the 12/04/18 between 2am and 6am BTW have planned works to migrate customers connected via the Edinburgh POSI (Point of service Interconnects) nodes to new hardware. The work is expected to take 25 - 45 minutes.
Started 12 Apr 02:00:00 by Carrier
Closed 12 Apr 08:00:00
Previously expected 12 Apr 06:00:00 (Last Estimated Resolution Time from Carrier)

10 Apr 14:30:00
[Broadband and Ethernet] Blip at 14:20 today - Closed
Details
Posted: 10 Apr 14:33:32
Many customers saw an outage of a few minutes at 14:20. This looks to have been caused by some network configuration we were doing at the time. It was not meant to cause any problems, we are reviewing what actually happened. We do apologise to those customers who were affected.
Update
10 Apr 14:34:15
Line are reconnecting, and services is restored for most users.
Update
10 Apr 14:38:30
The blip affected around half of TalkTalk connected circuits and would have affected some routing to the rest of the internet.
Started 10 Apr 14:20:00
Closed 10 Apr 14:30:00

4 Apr 18:00:00
Details
Posted: 29 Mar 19:59:48
Over the last few days we have seen several short attacks on parts of our network. Staff are monitoring and taking action. We don't usually put a lot of detail regarding DoS attacks. In some cases services may be briefly unavailable and routing may change. Sorry for any inconvenience.
Update
29 Mar 20:02:40
Overall traffic for broadband customers does not seem to be that impacted, but clearly something not quite right, and we are still looking in to it.
Update
29 Mar 20:04:01
Things are not quite perfect but mostly looking back to normal now.
Update
29 Mar 20:13:04
It has died down a lot, with some action from the ops team, but not entirely sorted you. Bear with us...
Update
30 Mar 15:38:10
Sorry for delay posting, we are seeing the problem again this afternoon.
Update
30 Mar 15:39:15
Mostly broadband traffic not impacted - but services in Maidenhead are being hit, including VoIP and web servers.
Update
30 Mar 18:38:41
Again some issues, and again a slight delay posting a status as effort on ensuring service working first, sorry about that. Some really odd things this last day or so...
Update
30 Mar 19:17:20
Once again, it is worth explaining that the majority of traffic levels are sensible but this is causing some brief annoyances - it almost seems to be randomly targeting customers. We are keeping a close eye on the issues.
Update
31 Mar 14:24:43
Obviously we are a bit limited with what we can say regarding attacks. At present the web servers are having issues, sorry for any inconvenience.
Update
31 Mar 18:58:27
OK, something more severe right now, looking in to it
Update
31 Mar 19:03:10
This seems to be impacting all broadband.
Update
31 Mar 19:16:32
That was weird, something very odd there and may have been one of our processes that caused an issue this evening, we are reviewing now.
Update
1 Apr 16:55:18
Problems are happening again this afternoon - staff have been on call all weekend working behind the scenes on this.
Update
1 Apr 17:26:18
It took slightly longer chasing the attack this evening. Some transit disruption.
Update
3 Apr 19:18:38
Sorry, reporting every blip, even if only a few seconds or even if once in a day, is probably not helping and may feed the attacker. I can say it is being works on at many levels and seems to be diminishing. Sorry for inconvenience.
Update
4 Apr 08:02:58
We are seeing routing issues to Maidenhead at present (web sites, voip). This seems to be a result of work we are doing now to address some of the issues of the last few days.
Update
4 Apr 08:13:35
This looks stable now.
Update
4 Apr 18:08:08
Still ongoing this evening, and we are improving the responses and have even added some pro-active measures. We are working on it, and having to take much more severe measures than ever before, but we are not really able to go in to a huge amount of detail, sorry. Please bear with us.
Update
5 Apr 08:34:52
No issues late last night - we're continuing to work on this, obviously.
Update
6 Apr 16:50:40
We're keeping this ticket open a little longer - nothing for nearly 48 hours, but staff are still monitoring closely.
Resolution Ongoing work is underway following our experience from these events.
Started 29 Mar 19:00:00
Closed 4 Apr 18:00:00

4 Apr 16:35:28
Details
Posted: 4 Apr 14:48:30
Magrathea (one of our voice carriers) is currently investigating network issues, further updates to follow.
Update
4 Apr 15:04:02
Looks like a major outage for them. This will be affecting inbound calls but some customers are still able to make calls outbound. I'm afraid Magrathea have no more information to provide on the issue at hand.
Update
4 Apr 15:11:48
Partial service has been restored now. Magrathea are still working on this. More updates to follow...
Update
4 Apr 15:49:19
Magrathea believe service is almost fully restored now. We'll let you know once we get the all OK.
Update
4 Apr 16:35:46
From Magrathea: We can confirm that we have the all clear and service is fully restored. Early analysis shows the peak of the outage was between 14:20 and 15:00 with service improving rapidly after that. As soon as we have completed our analysis and have a reason for outage we will send an email with details.
Started 4 Apr 14:28:00
Closed 4 Apr 16:35:28

5 Apr
[Broadband and Ethernet] Router upgrades - Completed
Details
Posted: 3 Apr 15:07:35
We are planning some router upgrades - as usually these should have little or no impact on normal services. This should be spread out over a number of days.
Update
4 Apr 08:39:06
Most router upgrades were done over night, with some config issues this morning. We hope not to have to do any work during the day, but it will depend how things go.
Started 3 Apr 18:00:00
Closed 5 Apr
Previously expected 7 Apr

1 Apr 09:04:35
[DNS, Email and Web Hosting] Problems viewing email - Closed
Details
Posted: 1 Apr 08:28:42
We are investigating problems with some users having problems receiving email by POP3/IMAP. Webmail is working, so for the moment that can be used: https://webmail.aa.net.uk
Resolution Mail connectivity has been restored, sorry for the inconvenience.
Started 1 Apr 08:00:00
Closed 1 Apr 09:04:35

29 Mar 10:40:49
Details
Posted: 23 Mar 10:33:28
On the 29th March BT are planning to update some of their core routers. The work is expected to take roughly an hour sometime between 02:00-06:00. We expect this to affect most of our customers on BT backhaul.
Update
29 Mar 10:40:54
This work was completed successfully.
Started 29 Mar 02:00:00 by Carrier
Closed 29 Mar 10:40:49
Previously expected 29 Mar 06:00:00 (Last Estimated Resolution Time from Carrier)

29 Mar 10:40:17
Details
Posted: 23 Mar 10:48:03
On the 29/03/18 between 2am and 6pm BTW have planned works to migrate customers connected via the Peterborough POSI (Point of service Interconnects) nodes to new hardware. The work is expected to take 25 - 45 minutes.
Update
29 Mar 10:40:40
This work was completed successfully.
Started 29 Mar 02:00:00 by Carrier
Closed 29 Mar 10:40:17
Previously expected 29 Mar 02:00:00 (Last Estimated Resolution Time from Carrier)

26 Mar 16:01:06
Details
Posted: 26 Mar 16:01:06

Our Home::1 tariff has always had a fixed single IPv4 address and a block of IPv6 addresses, however, we are pleased to announce that we will allocate a /30 or a /29 block of IPv4 addresses to anyone who needs them for free.

A /30 will give you two usable addresses and a /29 will give you 6 usable addresses. (The block of IPv6 addresses gives you around 18 million trillion usable addresses!)

Contact sales@aa.net.uk if you require a block.

Started 26 Mar 16:00:00

23 Mar 15:05:04
Details
Posted: 23 Mar 14:56:59

We are pleased to announce new installation prices, selectable contract lengths and a new router option which has dual band WiFi.

When ordering, you can choose 1, 6 or 12 month minimum terms for ADSL and FTTC services. If you choose 6 months then the install cost is free, if you choose 12 months then the install cost and a router with 2.4Ghz and 5Ghz WiFi are both free.

These new options and prices are separate to the "copper pair" which has no minimum term.

These options are live on our website and the Quote/Order form will guide you through the available options.

Started 23 Mar 14:00:00

12 Mar 22:30:00
Details
Posted: 12 Mar 10:32:05
We are investigating an intermittent problem sending email via our webmail service and directly from their normal email programs
Update
12 Mar 12:02:04
We are seeing low levels of packetloss, but only on IPv6 - we are disabling IPv6 on smtp.aa.net.uk for the moment to see if that makes a difference whilst we investigate further.
Update
12 Mar 12:10:00
IPv6 was a red herring - we're still investigating this intermittent problem.
Update
12 Mar 13:54:30

This problem is occurring when trying to send messages to email accounts that are hosted on Office 365 / outlook.com servers. This is related to https://aastatus.net/2502

Our mail servers should handle the type of failure better, we are looking at improving this.

Resolution Routing is now working again, we believe the problem is due to to routing through one of our routers which we have disabled this evening. We are still investigating the actual root cause of this. We would like to thank the other customers who reported this fault to the Office365 help desk, and do apologise for the frustration and inconvenience this has caused.
Started 12 Mar 09:00:00
Closed 12 Mar 22:30:00

12 Mar 22:29:43
Details
Posted: 12 Mar 10:30:13
We have had some reports of problems routing to Offic365. We have tried re-routing the traffic via various routes and traffic still fails deep within the Microsoft network. We are trying to contact Microsoft regarding this. This may affect sending of email to Office365/Outlook/Microsft/Hotmail/etc accounts as well as general access of Office 365 services.
Update
12 Mar 10:56:28
The problem we're seeing is certain *.protection.outlook.com mail servers not accepting connections on port 25 from us when we try send email to them. Luckily the hosts for microsoft-com.mail.protection.outlook.com do work, and we have emailed Microsoft regarding this.
Update
12 Mar 12:06:21

We have not yet heard back from Microsoft, but we encourage Office 365 customers to report this as a fault to Microsft too. The problem can be seen by attempting connections to the mail servers listed as the MX records for your domain, this will be in the form of *.mail.protection.outlook.com. This can be seen doing:

   telnet DOMAIN.mail.protection.outlook.com 25 

Where 'DOMAIN' is your domain name with .'s turned in to -'s.

If it fails, it will timeout, with a message such as:
  Unable to connect to remote host: Connection timed out
If it works, then the message is similar to:
   220 BL2NAM06FT009.mail.protection.outlook.com Microsoft ESMTP MAIL Service ready
Update
12 Mar 13:08:20
Separate to our emails to the Microsoft NOC we've had a 40 minute call with the Office356 enduser helpdesk, and they are investigating further.
Update
12 Mar 13:16:39
It seems Office365 servers in Dublin are the ones which are not accepting connections from us. These look to have IP addresses starting 213.199.154.x. Where as Office365 servers in the US such as 104.47.46.x and possibly 23.103.156.x are accepting connections from us OK.
Update
12 Mar 18:27:38
We have been pursuing various channels to speak to staff at Microsoft who are in a position to be able to help - so far it's proving difficult.
We have been talking to the general Office365 technical support team and team leaders, who after a couple of hours are starting to realise that the problem is one which they can't help with, and are attempting to move the ticket over to a network team.
Resolution Routing is now working again, we believe the problem is due to to routing through one of our routers which we have disabled this evening. We are still investigating the actual root cause of this. We would like to thank the other customers who reported this fault to the Office365 help desk, and do apologise for the frustration and inconvenience this has caused.
Started 12 Mar 09:00:00
Closed 12 Mar 22:29:43

12 Mar 00:50:00
[Broadband and Ethernet] Major service outage - Closed
Details
Posted: 12 Mar 00:45:25
We noticed (and received reports) that multiple customers were having issues at approximately 00:15 this morning, and investigation work commenced immediately. At present it appears we have a routing problem, affecting traffic to/from some destinations. This is actively being worked on right now. Sincere apologies for the inconvenience this is causing.
Update
12 Mar 00:50:12
Improvement : Work on this problem has improved matters. Traces in from external networks appear to now be working in instances where they were not before. Customers who initially reported the problem are now reporting a restoration of routing. At this point in time we do not want to consider this "fixed". But will continue to monitor and work on this until we are able to consider it so.
Update
12 Mar 00:54:51
Technical update : At the moment the cause seems to have been our route servers, getting stuck holding routes from a peer (m.aimless) that didn't actually have routes due to its connection with one carrier (NTT) being down due to routine planned work at their side. Work continues on this. Apologies again.
Update
12 Mar 00:57:57
This problem has been resolved.
Broadband and Ethernet Users Affected 75%
Started 12 Mar 00:15:00
Closed 12 Mar 00:50:00

8 Mar 15:15:34
Details
Posted: 8 Mar 15:15:34
There was a routing blip that affected some connectivity and caused some L2TP and data SIMs to drop and reconnect at around 15:02 today. The fault was resolved at around 15:08. We are investigating the cause.
Started 8 Mar 15:02:00

5 Mar 09:21:46
Details
Posted: 10 Jul 2017 22:52:30

OVERVIEW:

Using encryption is strongly recommended wherever possible to as to ensure private communication between you and the server and to protect your username and passwords from being intercepted by other people.

Historically, accessing our email servers using SSL/TLS has been available but problematic in that the certificates we use is issued by 'CA CERT' which although is 'secure', email programs often popped up a warning about the security settings.

SOLUTION:

We have added a new set of servers which customers can use to access their email. These use a Let's Encrypt certificate which will be accepted by email clients without any warnings.

IN TESTING:

At the moment (July 2017) we are still testing this out, and this should be viewed as 'beta'. There are a few rough edges which we are still working on fixing before making this an official service. that said, customers may try this out and we will make any maintenance as seamless as possible - meaning that it should work fine but there may be short periods of time when it stops working.

USAGE:

A simple change needs to be made to your email account settings within your email program (eg account settings in Thunderbird, Outlook, Mail etc). Go to the account settings where the IMAP or POP3 server details are, and ensure the following is set:

For IMAP:

Server hostname: mail.aa.net.uk

Port: 993

Security options: SSL/TLS

For POP3:

Server hostname: mail.aa.net.uk

Port: 995

Security options: SSL/TLS

You must use SSL/TLS as we do not support insecure access on these servers.

CHANGING BACK:

Customers are free to change the settings back to what they had before. This will not affect the emails stored on the server in any way.
Update
28 Jul 2017 20:46:50
Note: mail.aa.net.uk can also be used to manage your sieve filters (TLS on port 4190)
Update
11 Jan 12:36:02
Using the incoming mail server as: mail.aa.net.uk is now the recommended name to use. We ask that all customers check and update their settings.
Started 10 Jul 2017 22:00:00

5 Mar 09:21:13
[DNS, Email and Web Hosting] Issues with outbound email - Closed
Details
Posted: 01 Jun 2017 10:27:24
One of the key IP reputation services appears to have blacklisted both of our outgoing servers which is causing issues for some email customers. We have noticed this occurring for any emails destined for icloud.com or mac.com email addresses, however some other servers may be affected. We have contacted the service in question requesting that we are unblocked. We will update this status page as any updates become available.
Update
01 Jun 2017 11:01:17
We have adjusted our outgoing servers to use different IPs to circumvent the block. We would now suggest that mail customers attempt to resend any failed messages.
Update
12 Jun 2017 09:41:39
We are seeing this again unfortunately. We are investigating the cause of this, however we may have to wait for any blocks to expire.
Update
12 Jun 2017 10:01:38
We've applied a work around for the time being and sending email should be fine. We'll investigate the cause of why we were blocked in the first place and why our normal methods for blocking junk email in the first place did not work this time.
Started 01 Jun 2017 06:55:00 by AA Staff
Closed 5 Mar 09:21:13

5 Mar 09:15:27
Details
Posted: 9 Jan 13:42:48
One of our BT links dropped its connection - this caused some DSL circuits dropped - they should reconnect soon if not already
Update
9 Jan 13:47:13
The fibre is back up, and most DSL circuits have not reconnected.
Update
9 Jan 13:50:58
The link dropped again (whilst we were talking to BT!). We'll take the link out of service for the time being. Affected lines will connect on our other links.
Update
9 Jan 13:58:32
Again, lines have mostly all reconnected. We have taken the affected fibre out of service whilst we investigate the cause with BT.
Update
9 Jan 14:33:09
The network is stable and BT are investigating the cause. The fibre is still out of service whilst we investigate the cause with BT.
Update
11 Jan 12:55:17
BT were unable to find a fault when the investigated this on Tuesday. However, the circuit is dropping out again today. This has caused PPP reconnects for some customers. We are back on the case with BT.
Update
11 Jan 13:10:35

For the record and to help diagnostics of broadband faults, this incident has caused PPP drops at the following times:

2018-01-09 13:39
2018-01-09 13:48
2018-01-11 12:39
2018-01-11 12:33
2018-01-11 11:55
Update
11 Jan 14:22:04
BT are investigating.
Started 9 Jan 13:38:00
Closed 5 Mar 09:15:27

5 Mar 09:01:16
[Maidenhead Colocation] Web and email outage - Closed
Details
Posted: 04 Apr 2013 15:47:25

This is ongoing. We're investigating.

Update
04 Apr 2013 16:07:41

This should now be fixed. Please let support know if you see any problems, or have any questions.

Update
05 Apr 2013 09:15:04

This was resolved yesterday afternoon.

Update
20 Jul 2016 09:58:43
This may be related to a wider issue on the internet with a power outage at a major london data centre. Thos routing problems are still ongoing.
Started 04 Apr 2013 15:46:11
Closed 5 Mar 09:01:16

5 Mar 08:58:20
[DNS, Email and Web Hosting] Email Authentication Problem - Closed
Details
Posted: 31 Jan 19:03:43
A number of customers were affected by an email authentication problem this afternoon. The problem was not widespread. It appears to have started around 14:45, and gradually affected more customers until it was fixed at around 18:45. It affected authentication at the beginning of a mail session, so it was possible to have a device that connected before the problem started continue to work throughout even though new logins were broken for the same account. Sorry for any inconvenience.
Started 31 Jan 14:45:00 by Customer report
Closed 5 Mar 08:58:20

5 Mar 08:53:13
Details
Posted: 20 Oct 2017 15:01:04

Multiple vulnerabilities have been reported in dnsmasq, the service on the ZyXEL routers which provides a DNSresolver, DHCP functions and router advertisement for IPv6.
A list of these can be found here.

The scope of these is relatively broad, and most attack vectors are local. causing the router to fail DNS/DHCP/RA if exploited (DoS condition), one additional vector includes execution of arbitrary code.
As dnsmasq is tightly integrated into the router, it can't just be turned off, however as a workaround you can change the DNS servers the routers serves in it's DHCP server to minimise potential impact from a local attacker should they perform a DNS request in a way which exploits the DoS conditions. This can be performed by going to the Router Settings page on your control pages, and entering our nameservers 217.169.20.20 and 217.169.20.21 into the IPv4 DNS fields, clicking save, then clicking on the DHCP button at the bottom (for B10Ds only) or clicking on Send ZyXEL configuration (B10As or B10Ds, will wipe existing settings on the router)

An official patch from ZyXEL is estimated to be provided during December 2017 for the VMG1312-B10D routers (Firmware V5.13(AAXA.7)), and January 2018 for the the VMG1312-B10A routers.
An update will be made to this post once firmware updates have been released.

Update
30 Jan 13:49:52
Updated software available, version 1.00(AAJZ.14)C0: https://www.zyxel.com/euosearch/dl-search.aspx?mci_country=uk&mci_lang=en&keyword=VMG1312-B10A&submit=Search
Previously expected 1 Feb

5 Mar 08:52:43
Details
Posted: 04 Sep 2017 17:22:46

We have a number of tariff changes planned, after a lot of interesting comments from my blog post - thank you all.

Some things are simple, and we are able to do sooner rather than later, like the extra 50GB already announced. Some will not be until mid to late October as they depend on other factors. Some may take longer still.

To try and ensure we get improvements as quickly as possible for customers I am updating a news item on our web site with details as we go.

http://aa.net.uk/news-2017-tariffs.html

As you will see, we are testing a change to make top-up on Home::1/SoHo::1 not expire. We have the end of a period (full moon) in two days where we can see if code changes work as expected on a live customer line. If all goes well then later this week we can change the description on the web site and officially launch this change.

Do check that page for updates and new features we are adding as we go.

Update
06 Sep 2017 13:57:18
We have made top-up on Home::1 and SoHo::1 not expire, continuing until you have used it all. This applies to any top-up purchased from now on.
Started 04 Sep 2017 17:19:37

23 Mar 10:46:04
Details
Posted: 21 Feb 10:46:11
On the 28/2/18 between 2am and 6pm BTW have planned works to migrate customers connected via the Cardiff POSI (Point of service Interconnects) nodes to new hardware. The work is expected to take 25 - 45 minutes.
Started 28 Feb 02:00:00 by Carrier
Closed 23 Mar 10:46:04
Previously expected 28 Feb 06:00:00 (Last Estimated Resolution Time from Carrier)

4 Mar 20:58:09
Details
Posted: 26 Feb 16:24:16
We have received notice of this planned work which affects connectivity for data SIMs in the listed areas at the listed times. The planned work windows are 7 hours, but the outage is expected to be less than 2 hours per window. Service Impact Start Time: 26/02/2018 23:00 (GMT) Restoration Time: 27/02/2018 06:00 (GMT) Areas affected: DERBYSHIRE LINCOLNSHIRE NOTTINGHAMSHIRE SOUTH YORKSHIRE NORTHAMPTONSHIRE Service Impact Start Time: 27/02/2018 23:00 (GMT) Restoration Time: 28/02/2018 06:00 (GMT) Areas affected: SHROPSHIRE CHESHIRE PEMBROKESHIRE POWYS STAFFORDSHIRE DERBYSHIRE CEREDIGION HEREFORDSHIRE WEST MIDLANDS Service Impact Start Time: 28/02/2018 23:00 (GMT) Restoration Time: 01/03/2018 06:00 (GMT) Areas affected: WEST MIDLANDS STAFFORDSHIRE WEST LOTHIAN WARWICKSHIRE Service Impact Start Time: 01/03/2018 23:00 (GMT) Restoration Time: 02/03/2018 06:00 (GMT) Areas affected: WEST MIDLANDS
Started 26 Feb 23:00:00 by Carrier
Closed 4 Mar 20:58:09

20 Feb 13:23:41
Details
Posted: 16 Feb 12:16:25
We are carrying out work on the servers that serve our web site. This work will be carried out on Tuesday morning. This should not affect the availability of our web site, but we consider this time an "at risk" period hence creating this planned work notice.
Update
20 Feb 13:24:19
This work has now been completed.
Started 20 Feb 11:00:00 by AA Staff
Closed 20 Feb 13:23:41
Previously expected 20 Feb 13:00:00

25 Jan 11:00:03
Details
Posted: 25 Jan 11:32:23
Between 10:45:43 - 10:57:33 we saw several blips on half of our new BT host links. They do seem stable at the moment; however we are still waiting on BT to comment on the cause.
Resolution BT Have confirmed that this was a problem on their side. We are awaiting further details about the incident.
Started 25 Jan 10:45:43
Closed 25 Jan 11:00:03

15 Jan 13:02:42
Details
Posted: 8 Jan 13:49:47

As mentioned in https://aastatus.net/2482 we are installing new fibres into BT to increase our capacity. We are now ready for customers on BT backhaul ADSL and VDSL circuits to test this.

Customers prefix their username in their router withe "test-" which will then connect through the new fibres. (eg if your username is example@a.1, change it to test-example@a.1) There should be no other differences in the service. We'd appreciate any feedback and observations being sent to trial@aa.net.uk

All being well, we hope to move the rest of the customer circuits in stages next week at which point you can remove the "test-" from your username.

Update
8 Jan 13:54:42
If customers go over their usage tariff during the testing we'll apply free top-ups - email trial@aa.net.uk if this happens or if you think it is likely.
Update
11 Jan 13:43:35

From Monday 15th January we will be moving circuits over to the new BT pipes, this process will take a number of days and therefore the trial will stop on 24th Jan.

Customers who changed their username can change it back on or before 24th January. Thank you for everyone's feedback and cooperation.

Started 8 Jan 13:00:00

15 Jan 11:19:42
[DNS, Email and Web Hosting] IMAP and POP3 server name change - Info
Details
Posted: 15 Jan 11:19:42

Over the years we have had various incarnations of the POP3 and IMAP server name that customers have been using to receive email. Last year we worked on making this simpler and for a while now we have been asking customers to use mail.aa.net.uk with SSL/TLS enabled using port 993 for IMAP or 995 for POP3. This has been done to improve the efficiency and privacy of our email service. Your outgoing SMTP settings are unchanged, and are still: smtp.aa.net.uk

From this week we are starting to contact customers by email who are using an old server name and sending details on what to do.

Features and stored email are all unaffected by this name change. We have a support page which has the correct settings and examples for some email programs: https://support.aa.net.uk/Incoming_Email_Settings:_Quick_Examples

Please contact Support if you need any assistance

Started 15 Jan 11:00:00

26 Jan 09:00:00
Details
Posted: 11 Jan 13:33:26

We do upgrades on our L2TP Network Servers over night one at a time as a rolling upgrade from time to time. Starting from Monday 1AM we starting the process. This will be ongoing over the next two weeks, impacting a small proportion of customers each night with a PPP drop and reconnect using the preferred time as set on the control pages. Depending on your equipment this could be anything from a fraction of a second outage to a few minutes.

In tandem with this, circuits on BT back-haul will reconnect using our new fibres into BT.

Update
12 Jan 11:51:52
A few BT back-haul connections on the "B" LNS moved to the "A" LNS just now, this was not quite as planned, but has proved to be a good test of the new BT links. Sorry for any inconvenience. We expect the further rolling updates and switches to new links to be somewhat more seamless during next week now.
Update
19 Jan 08:28:29
We have now upgraded half of our LNSs and half of our BT circuits are on our new BT interconnects.
Resolution All circuits are now on the new Interconnects.
Started 15 Jan 01:00:00
Closed 26 Jan 09:00:00
Previously expected 24 Jan 10:00:00

9 Jan 12:49:12
[DNS, Email and Web Hosting] IMAP indexing problem - Closed
Details
Posted: 22 Aug 2017 13:02:21
We are working on solving a problem that we're currently seeing with IMAP indexing on our mail servers. The symptoms customers are likely to see are small oddities such as emails appearing not to move between folders, or appearing twice. This problem is only index related and so doesn't actually affect the emails themselves. This problem is not causing email to be lost.
Update
22 Aug 2017 16:47:26
We are still investigating a proper fix for this problem, but in the mean time we are making changes that should work around it. There is a small risk that if you are using Sieve filtering that it may stop working. If that is the case, please contact support for assistance
Resolution Longer term fixes for this was implemented as part of planned work: https://aastatus.net/2487
Closed 9 Jan 12:49:12

8 Jan 22:10:00
Details
Posted: 5 Jan 15:05:14
We will be carrying out some work on our POP3/IMAP servers on Monday from 9PM. This involves some configuration changes at our side and will affect access to the servers whilst this work is carried out.
Update
8 Jan 21:18:38
This work has started.
Update
8 Jan 21:56:12
The configuration is still in progress but at this point customers will be able to view and receive email again as long G as they are using the IMAP/POP3 server of mail.aa.net.uk
Update
8 Jan 22:13:14
This work has been completed. Do contact support if you have any problems.
Started 8 Jan 21:00:00
Closed 8 Jan 22:10:00

5 Mar 08:57:59
Details
Posted: 5 Jan 12:36:27

Due to the 'Meltdown' and 'Spectre' vulnerabilities (https://meltdownattack.com/) We will be performing various server upgrades and reboots this evening and over the weekend.

The most noticeable of these will be IMAP/POP3 email servers. These will be updated this evening from 10PM. We expect IMAP and POP3 services to be disrupted for up to 30 minutes whilst this work is carried out.

Update
5 Jan 22:34:17
This work has completed successfully.
Started 5 Jan 22:00:00
Closed 5 Mar 08:57:59

4 Jan 11:20:00
Details
Posted: 3 Jan 10:30:19
We're seeing incorrect SMS delivery reports from SMSs bein set to the EE/t-mobile network. SMSs are being delivered successfully but we receive a delivery report saying the message was rejected. This looks like it may be a problem between our carrier and EE. This is being investigated.
Update
4 Jan 09:42:16
Update from our carrier: "I have escalated this issue to our senior systems team for review"
Update
5 Jan 16:20:06
This has been fixed.
Started 3 Jan 10:00:00
Closed 4 Jan 11:20:00

4 Jan 11:00:27
Details
Posted: 4 Jan 11:00:27

We're updating SSL certificates for our legacy email servers today. The old serial number is 130CAB. The new serial number is 1360FD. However, if you are reading this status post because are you seeing the change or a warning message then you are probably using out of date settings. Please change your POP3 or IMAP server name setting to our new one, which is simply: mail.aa.net.uk

See https://support.aa.net.uk/Email_Settings for more information about email settings

Started 4 Jan 10:00:00

3 Jan 17:27:06
Details
Posted: 3 Jan 17:26:31
We have upgraded our order pages to allow ordering and regrading to the new terabyte tariffs on BT lines (as was only on TT lines before). Not available on 20CN, but otherwise you can now select 1TB (on Home::1) or 2TB (on SoHo::1) as just another tariff choice. As usual regrades take effect from next month, and the quota bonus system applies to this tariffs as well. Importantly they can be "balanced" with lines on the same site that are not on terabyte levels allowing multi-line sites to have different tariffs on each line simply shared in total by both lines.
Started 3 Jan 17:26:49

18 Dec 2017 18:00:00
[DNS, Email and Web Hosting] Web hosting problems - Closed
Details
Posted: 14 Dec 2017 09:04:39
We're investigating problems with our web hosting.
Update
14 Dec 2017 09:14:46
There is a problem with the file server that hosts the files for web hosting.
Update
14 Dec 2017 09:26:32
The RAID array for the web hosting has broken. This is a simple two disk RAID1 mirror. It looks like the mirror became degraded overnight but for an unknown reason the RAID controller also removed the remaining drive! We are attempting to rebuild it, but if that fails we'll need to restore from backup.
Update
14 Dec 2017 09:51:55
We've managed to bring the RAID array back online, we are running a filesystem check as it looks like some files, or file attributes are corrupt.
Update
14 Dec 2017 10:25:18
Unfortunately, due to the problem we are having with web hosting we are needing to reboot the file server that manages email and web. This is happening now, and will mean access to email will be unavailable for a few minutes. We do apologise for this.
Update
14 Dec 2017 11:21:33
The file system did not come back after the reboot. A repair has been completed and the file system is mounted again.
Update
14 Dec 2017 12:29:08
Sadly, we're needing to restore websites from backup, This will be starting soon but it will take some time for data to be fully restored. We'll update on the progress at around 1PM though.
Update
14 Dec 2017 13:33:31
Websites are being restored from our Wednesday morning backup. About 16% of the files have been restored so far. We expect this to take a few more hours to complete.
Update
14 Dec 2017 14:10:32
Over 25% now.
Update
14 Dec 2017 15:51:57
Over 50% now.
Update
14 Dec 2017 19:00:30
Backups have been restored, and FTP/rsync are now live again. Web access logs have been lost - sorry for any inconvenience.
Update
15 Dec 2017 12:39:15
Some customers are unable to ftp or rsync files to their web space. This is being worked on now though.
Update
15 Dec 2017 17:26:35
The job to repair the ftp/rsync issues at this moment in time is still running.
Update
16 Dec 2017 12:25:00
FTP/rsync is now repaired, and access to the web space should be back to normal. Unfortunately, during the restore process, weblogs were not recreated for any domains using them. This means that customers will need to log back in to the web space to re-enable logs. Sorry for any inconvenience that causes!
Started 14 Dec 2017 09:00:00
Closed 18 Dec 2017 18:00:00

14 Dec 2017 12:30:00
[DNS, Email and Web Hosting] Email problems - Closed
Details
Posted: 14 Dec 2017 10:59:30
This is related to https://aastatus.net/2479 We are problems with the file server for most of our email users. Whilst working on fixing problems with web hosting a problem occurred with the mail store for email. This is being investigated.
Update
14 Dec 2017 11:30:53
As per https://aastatus.net/2479 the file system did not come back after a reboot. The file system was repaired and is mounted again. Mail should be working again, however it may be a little slow. We do apologise for the inconvenience.
Update
14 Dec 2017 12:29:42
Email is back online, things will be a little slow whilst it all catches up.
Started 14 Dec 2017 10:45:00
Closed 14 Dec 2017 12:30:00

14 Dec 2017 11:01:29
Details
Posted: 05 Dec 2017 09:41:51
We're investigating a problem with our email (IMAP/POP3) servers. Customers may have problems reading email. Update to follow shortly.
Update
05 Dec 2017 09:44:43
The problem is relating to our storage server, this is being investigated.
Update
05 Dec 2017 10:08:31
Email and web services are starting to recover.
Update
05 Dec 2017 11:25:08
The mail servers are busy catching up and are slow at the moment. Things should catch up soon though.
Update
05 Dec 2017 11:52:05
We're still experiencing load problems on the mail servers causing timeouts and delays. We are working improving this at the moment, but some customers will be having problems receiving email.
Update
05 Dec 2017 15:51:32
Email is still slow for some people and this is still being worked on.
Update
06 Dec 2017 10:33:22
Email is a little slow today still. We are planning some hardware upgrades to increase capacity, some of which will happen tomorrow and some today.
Update
06 Dec 2017 11:37:40
We will be restarting one of our IMAP servers at 12:30 this afternoon, we expect this to take less than one minute. This will not affect customers using the official IMAP/POP3 server hostname of mail.aa.net.uk. Customers using the now deprecated mail.[yourdomain] type hostname may experience a brief disconnect.
Update
06 Dec 2017 12:35:17
The restart has been completed.
Update
13 Dec 2017 16:13:05
Further work is being done to help with the performance of our email services, please see: https://aastatus.net/2478
Started 05 Dec 2017 09:20:00
Closed 14 Dec 2017 11:01:29

14 Dec 2017 10:59:55
Details
Posted: 14 Dec 2017 10:25:00
Unfortunately, due to the problem we are having with web hosting we are needing to reboot the file server that manages email and web. This is happening now, and will mean access to email will be unavailable for a few minutes. We do apologise for this.
Update
14 Dec 2017 10:52:01
There are problems with one of the mount points for email. We are investigating.
Resolution This post is being closed for more info see: https://aastatus.net/2481 and https://aastatus.net/2479
Started 14 Dec 2017 10:20:00
Closed 14 Dec 2017 10:59:55

13 Dec 2017 23:12:35
Details
Posted: 13 Dec 2017 14:58:28
We will performing a hardware upgrade of the storage server that is used for email and web page hosting. This will impact access to email and hosted web pages whilst this work is carried out, we expect this to take under 30 minutes. We're sorry for the short notice.
Update
13 Dec 2017 22:10:08
This work has started.
Update
13 Dec 2017 22:46:39
This is taking a bit longer than expected, but services should be resumed in 30 minutes or so. Update to follow.
Update
13 Dec 2017 23:08:14
Services are being restored at the moment.
Update
13 Dec 2017 23:13:12
All services should be working again now. Staff are monitoring.
Resolution This work went well, except overnight there was a problem with web hosting, this is detailed in post: https://aastatus.net/2479
Started 13 Dec 2017 22:00:00
Closed 13 Dec 2017 23:12:35

13 Dec 2017 16:11:18
Details
Posted: 27 Nov 2017 20:17:21

As you know, we are one of the few ISPs that monitor every line every second for loss, latency and throughput, and provide that data to customers to see how well your line is performing, and how we are performing as your ISP.

Well, the latest FireBrick code now provides these graphs in SVG format which means awesome scaleable graphs with even more detail and much clearer and easier to understand.

We are rolling out these as part of LNS upgrades over the coming weeks, and will have all systems using them soon. In the mean time you may see a mix of old (PNG) and new (SVG) graphs.

I hope you appreciate them as much as I do, and the hard work we have put in to making them. We take the quality of our service very seriously which is why we make this available to customers, and the nicer and clearer we can make that - the better.

Started 27 Nov 2017

13 Dec 2017 16:00:00
Details
Posted: 21 Oct 2017 17:30:55
We have identified a small number of TalkTalk connected circuits that have 1-4% packetloss in the evenings between 7pm and 9pm. We have worked with TalkTalk NOC and have helped them to identify a link running at full capacity within their network. Work is still ongoing to fix this.
Update
21 Oct 2017 17:42:22
To aid with TalkTalks investigations we have shutdown our interconnect in Harbour Exchange to TalkTalk. We still have plenty of capacity in Telehouse so this work is not service affecting.
Update
21 Oct 2017 17:50:43
TalkTalk say: "NOC troubleshooting has identified that one of three 10 gig circuit between the LTS and the LDN at hex is reaching capacity between 19:30 and 21:30hrs and is the root cause of the packet loss."
Update
10 Nov 2017 11:23:51
See: https://aastatus.net/2454 for details regarding testing the new TalkTalk network which will resolve this.
Update
22 Nov 2017 21:33:35
packetloss on TalkTalk connected circuits has increased over the past day or so. We're happy to move customers to trial TalkTalk's new network, do get in touch with trial@aa.net.uk
Update
24 Nov 2017 15:02:12
TalkTalk have done some work on their side which should help reduce the low levels of packetloss that some lines are seeing.
Update
12 Dec 2017 09:11:02
Work that TalkTalk have done recently has helped with congestion and we are now seeing far fewer lines with congestion.
We don't yet have a date when we'll move all customers across to the new network. We do hope this will be before Christmas bust due to the holidays it may be early January.
Resolution The congestion problem has been resolved. The longer term option of moving our customer base to TalkTalk's new network will happen in January 2018. More details of this will be posted nearer the time.
Broadband Users Affected 1.50%
Started 18 Oct 2017 19:00:00
Closed 13 Dec 2017 16:00:00

09 Dec 2017 11:51:09
Details
Posted: 09 Dec 2017 10:00:34
Support staff may be starting late today due to a lack of electricity at our office. It appears to be an underground cable fault.
Update
09 Dec 2017 10:00:52
Update
09 Dec 2017 12:38:51
A genset has been spliced into the mains incomer, meaning power to the offices has been restored whilst the energy supplier continue to work on repairing the faulty section of underground cable.
Update
11 Dec 2017 18:35:04
The generator is planned to be removed "later tonight" according to engineers, who are currently working hard to repair the fault having dug a few more holes and so far managed to avoid the fibre to the office. This work involves disconnecting other parts of the industrial estate to make it safe to disconnect power, then disconnecting ourselves from the genset. Once disconnected, our mains incomer will be spliced back to the repaired cable.
This is not expected to impact on operations in the office tomorrow morning as this disconnection and reconnection is happening out of hours, but updates will be posted should the situation change.
Resolution Generator provided.
Started 08 Dec 2017 20:15:00
Closed 09 Dec 2017 11:51:09
Previously expected 09 Dec 2017 10:00:00

04 Dec 2017 18:00:00
Details
Posted: 04 Dec 2017 11:35:04
We have routing problems to some authoritative DNS services hosted by Edgecast, this is causing some DNS lookups to fail. Example domains that are intermittently failing are  flightaware.com, expertflyer.com, lenovo.com, postoffice.co.uk, norwegian.com, twimg.com and some others. We have spoken to Edgecast NOC (after not hearing back from our emails to them) and are sending them further details.
Update
04 Dec 2017 14:52:29
We have spoken to Edgecast again, and they are looking in to this.
Update
04 Dec 2017 16:56:31
We've turned off one of our peerings with Edgecast, and note that the DNS resolution issues are looking somewhat better with this off for the moment from our tests.
Investigations from both sides continue as to why it's being broken.
Update
04 Dec 2017 18:11:43
We have worked around the problem temporarily and DNS is looking to be working ok again. We will investigate further though.
Update
04 Dec 2017 18:12:01
Edgecast have replied saying: "Thank you for the additional information. We are noticing some errors on a couple of our backbone links in one of our London PoP which is where the peering session located. We're currently working with our teams to get these links removed from production. Once this is done, we'll provide an update. "
Update
04 Dec 2017 21:52:23
Edgecast said they saw an issue, but we also found if we take down our peering to them over one of our LINX ports that the problem goes away. We're needing to do a bit more investigation to find out why, but since 6pm-ish routing and DNS lookups to Edgecast is working.
Started 04 Dec 2017 08:00:00
Closed 04 Dec 2017 18:00:00