Order posts by limited to posts

Today 02:00:00
22 May 16:29:22
Due to upstream upgrade work on core infrastructure, there may be up to 2 hours 30 minutes disruption to data SIM services on the morning of Thursday the 25th of May between the hours of 2:00 and 4:30.
Started Today 02:00:00 by Carrier
Previously expected Today 04:30:00

18 May 16:30:59
7 Feb 14:32:32

We are seeing issues with IPv6 on a few VDSL cabinets serving our customers. There is no apparent geographical commonality amongst these, as far as we can tell.

Lines pass IPv4 fine, but only intermittently passing IPv6 TCP/UDP for brief amounts of time, usually 4 or so packets, before breaking. Customers have tried BT modem, Asus modem, and our supplied ZyXEL as a modem and router, no difference on any. We also lent them a FireBrick to do some traffic dumps.

Traffic captures at our end and the customer end show that the IPv6 TCP and UDP packets are leaving us but not reaching the customer. ICMP (eg pings) do work.

The first case was reported to us in August 2016, and it has taken a while to get to this point. Until very recently there was only a single reported case. Now that we have four cases we have a bit more information and are able to look at commonalities between them.

Of these circuits, two are serving customers via TalkTalk and two are serving customers via BT backhaul. So this isn't a "carrier network issue", as far as we can make out. The only thing that we can find that is common is that the cabinets are all ECI. (Actually - one of the BT connected customers has migrated to TalkTalk backhaul (still with us, using the same cabinet and phone line etc) and the IPv6 bug has also moved to the new circuit via TalkTalk as the backhaul provider)

We are working with senior TalkTalk engineers to try to perform a traffic capture at the exchange - at the point the traffic leaves TalkTalk equipment and is passed on to Openreach - this will show if the packets are making it that far and will help in pinning down the point at which packets are being lost. Understandably this requires TalkTalk engineers working out of hours to perform this traffic capture and we're currently waiting for when this will happen.

2 Mar 11:14:48
Packet captures on an affected circuit carried out by TalkTalk have confirmed that this issue most likely lies in the Openreach network. Circuits that we have been made aware of are being pursued with both BT and TalkTalk for Openreach to make further investigations into the issue.
If you believe you may be affected please do contact support.
17 Mar 09:44:00
Having had TalkTalk capture the traffic in the exchange, the next step is to capture traffic at the road-side cabinet. This is being progresses with Openreach and we hope this to happen 'soon'.
29 Mar 09:52:52
We've received an update from BT advising that they have been able to replicate the missing IPv6 packets, this is believed to be a bug which they are pursuing with the vendor.

In the mean time they have also identified a fix which they are working to deploy. We're currently awaiting further details regarding this, and will update this post once further details become known.
18 May 16:30:59
We've been informed that the fix for this issue is currently being tested with Openreach's supplier, but should be released to them on the 25th May. Once released to Openreach, they will then perform internal testing of this before deploying it to their network. We haven't been provided with any estimation of dates for the final deployment of this fix yet.
In the interim, we've had all known affected circuits on TalkTalk backhaul have a linecard swap at the cabinet performed as a workaround, which has restored IPv6 on all TT circuits known to be affected by this issue.
BT have come back to us suggesting that they too have a workaround, so we have requested that it is implemented on all known affected BT circuits to restore IPv6 to the customers known to have this issue on BT backhaul.
Broadband Users Affected 0.05%
Started 7 Feb 09:00:00 by AA Staff
Update was expected Today 18:00:00

8 May 09:26:21
26 Apr 11:01:07
We have noticed packetloss between 8pm and 10pm on Tuesday (25th April) evening on a small number of TalkTalk connected lines. This may be related to TalkTalk maintenance. We will review this again tomorrow.
26 Apr 16:41:43
We are seeing packet loss this afternoon on some of these lines too. We are contacting TalkTalk.
26 Apr 16:44:23
26 Apr 17:58:06
We have moved TalkTalk traffic over to our Harbour Exchange interconnect to see if this makes a difference or not to the packet loss that we are seeing...
26 Apr 20:50:41
Moving the traffic made no difference. We've had calls with TalkTalk and they have opened an incident and are investigating further.
26 Apr 20:55:14
The pattern that we are seeing relates to which LAC TT are using to send traffic over to us. TT use two LACs at their end, and lines via one have loss whilst lines via the other have no loss.
26 Apr 21:32:30
Another example, showing the loss this evening:

26 Apr 22:26:50
TalkTalk have updated us with: "An issue has been reported that some Partners are currently experiencing quality of service issues, such as slow speed and package (SIC) loss, with their Broadband service. From initial investigations the NOC engineers have identified congestion to the core network connecting to Telehouse North as being the possible cause. This is impacting Partners across the network and not specific to one region, and the impacted volume cannot be determine at present. Preliminary investigations are underway with our NOC and Network Support engineers to determine the root cause of this network incident. At this stage we are unable to issue an ERT until the engineers have completed further diagnostics."
28 Apr 09:43:43
Despite Talktalk thinking they had fixed this we are still seeing packetloss on these circuits between 8pm and 10pm. It's not as much packetloss as we saw on Wednesday evening, but loss nonetheless. This has been reported back to TalkTalk.
4 May 15:42:04
We are now blocking the two affected Talkalk LACs on new connections, eg a PPP re-connect. This means that it will take a bit longer for a line to re-connect (depending upon the broadband router perhaps a minute or two).

This does mean that lines will not be on the LACs which have evening packetloss. We hope not to have to keep this blocking in place for very long as we hope TalkTalk fix this soon.

5 May 16:48:13
We've decided to not block the LACs that are showing packetloss as it was causing connection problems for a few customers. We have had a telephone call with TalkTalk today, and this issue is being escalated with TalkTalk.
5 May 19:47:37
We've had this update from TalkTalk today:

"I have had confirmation from our NOC that following further investigations by our IP Operations team an potential issue has been identified on our LTS (processes running higher than normal). After working with our vendor it has been recommended a card switch-over should resolve this.

This has been scheduled for 16th May. We will post further details next week.

8 May 09:26:21
Planned work has been scheduled for 16th May for this, details and updates of this work is on https://aastatus.net/2386
Broadband Users Affected 20%
Started 26 Apr 10:00:00
Update was expected 16 May 12:00:00

3 May 13:20:48
26 Apr 10:16:02
We have identified packet loss across our lines at MOSS SIDE that occurs between 8pm to 10pm. We have raised this with BT who suggest that they hope to have this resolved on May 30th. We will update you on completion of this work.
Broadband Users Affected 0.15%
Started 26 Apr 10:13:41 by BT
Update expected 30 May 12:00:00
Expected close 30 May (Estimated Resolution Time from BT)

6 Apr 01:00:00
30 Mar 11:19:03
Due to maintenance being performed by a carrier, disruption of up to 15 minutes to 3G/4G data services is expected some time between 01:00 and 03:00 on the 6th of April.
Started 6 Apr 01:00:00
Previously expected 6 Apr 03:00:00

9 Mar 20:00:00
8 Mar 12:29:14

We continue to work with TalkTalk to get to the bottom of the slow throughput issue as described on https://aastatus.net/2358

We will be performing some routing changes and tests this afternoon and this evening, we are not expecting this to cause any drops for customers, but this evening there will be times when throughput for 'single thread' downloads will be slow. Sorry for the short notice, please bear with us, this is being a tricky fault to track down.

8 Mar 22:39:39
Sorry, due to TalkTalk needing extra time to prepare for their changes this work has been moved to Thursday 9th evening.
Started 9 Mar 20:00:00
Update was expected 9 Mar 23:00:00

7 Feb 14:49:55
[DNS, Email and Web Hosting] - SMTP Settings Change - Open
7 Feb 14:52:42
For historical reasons, our SMTP servers allow sending authenticated email without TLS. This is insecure, and doesn't belong on the modern internet as it is possible for the username and password to be intercepted by a third party. We will no longer allow this as of the 4th of July. We are emailing customers who seem to be using insecure settings to warn them about the change. We have a support site page about what settings to change here: https://support.aa.net.uk/Enable_TLS_on_smtp.aa.net.uk Please contact support if you have any questions.
Started 7 Feb 14:49:55
Expected close 4 Jul 09:00:00

20 Sep 2016 11:26:07
20 Sep 2016 11:31:39
Our upstream provider have advised us that they will be carrying out firmware upgrades on core 3G infrastructure on 23rd September 2016 between 00:10 and 04:30 BST.
During this period data SIMs may briefly disconnect as sessions are migrated to other nodes in the upstream provider's network to facilitate the upgrades.
VoIP and SIMs Users Affected 25%
Started 20 Sep 2016 11:26:07 by Carrier
Update was expected 23 Sep 2016 11:30:00
Previously expected 23 Sep 2016 04:30:00 (Last Estimated Resolution Time from Carrier)

01 Sep 2016 17:18:26
01 Sep 2016 17:26:51
Our SMS gateway has always supported HTTPS but it still allows using HTTP only. As using HTTP only is probably a mistake these days we are going to change our gateway to redirect all HTTP requests to HTTPS next week. We don't expect anything to break as curl, etc., should "just work" but we're posting a PEW just in case anyone has a legacy script they think might need adjusting!
09 Sep 2016 10:06:35
Unfortunately we've had to back out of making this change this week as it caused some unforeseen problems! We'll resolve those problems and make the change again soon.
Started 01 Sep 2016 17:18:26
Previously expected 07 Sep 2016 17:18:26

20 Jul 2016 09:58:43
[Maidenhead Colocation] - Web and email outage - Open
04 Apr 2013 15:47:25

This is ongoing. We're investigating.

04 Apr 2013 16:07:41

This should now be fixed. Please let support know if you see any problems, or have any questions.

05 Apr 2013 09:15:04

This was resolved yesterday afternoon.

20 Jul 2016 09:58:43
This may be related to a wider issue on the internet with a power outage at a major london data centre. Thos routing problems are still ongoing.
Started 04 Apr 2013 15:46:11

13 Apr 2015
09 Apr 2015 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr 2015

08 Jan 2015 12:39:06
08 Jan 2015 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
09 Jan 2015 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
12 Jan 2015 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 08 Jan 2015 12:39:06 by AA Staff
Previously expected 15 Jan 2015 17:00:00

03 Jun 2014 17:00:00
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

[DNS, Email and Web Hosting] - At Risk Period for Web Hosting - Open
21 Feb 14:43:14
We are carrying out maintenance on our customer facing web servers during this Thursday's maintenance window. We expect no more than a couple of minutes of downtime but web services should be considered "at risk" during the work.
Previously expected 23 Feb 22:00:00

06 Jul 2015 12:49:42
We have been advised by Three of works on their network (22:30 8th July to 04:50 9th July 2015) Which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some partners may experience dropped connections. Any device reconnecting will automatically be routed via a new path. We apologise for any inconvenience this may cause and the short notice of this advisory.

12 Feb 2015 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

17 May 20:30:22
17 May 11:22:27
We have reduced numbers of staff in the office today due some being taken on a tour of a BT exchange & metro node.
Resolution All affected staff members are now back in service, and proven to be operational.
Started 17 May 09:00:00
Cause BT

17 May 12:30:00
17 May 09:10:40
17 May 09:30:01
Affected Customers went offline at 02:12 this morning. Further information: These exchanges are off line due to 20 tubes of fibre being damaged by the install of a retail advertising board on the A118 Southern [South?] Kings Rd. Notifications from TalkTalk were expecting service to be restored by 9am, but due to the nature of the fibre break it may well take longer to fix.
17 May 10:31:29
Update from TalkTalk: Virgin media have advised that work to restore service is ongoing but due to the extent of the damage this is taking longer than expected. In parallel our [TalkTalk] NOC are investigating if the Total Loss traffic can be re-routed. We will provide a further update as soon as more information is available.
17 May 10:55:56
From TalkTalk: Virgin Media have advised restoration work has completed on the majority of the damaged fibre, our NOC team have also confirmed a number of exchanges are now up and back in service. The exchanges that are now back in service are Muswell Hill, Ingrebourne, Loughton and Bowes Park.
17 May 11:16:28
From TalkTalk: Our NOC team have advised a number of exchanges are now in service. These are Muswell Hill, Bowes Park, Loughton, Ingrebourne, Bowes Park, Chingford, Highams Park, Leytonstone, Stratford and Upton Park.
17 May 11:30:54
That said, we are still seeing lines on exchanges mentioned above as being offline....
17 May 12:11:20
No further updates as yet.
Resolution It looks like most, if not all, of our affected lines are now back online. Update from TalkTalk: Virgin Media have advised 5 of the 8 impacted fibre tubes have been successfully spliced and their engineers are still on site restoring service to the remaining cables
Started 17 May 01:54:00 by AA Staff
Closed 17 May 12:30:00

13 May
12 May 18:02:27
The upgrades this week worked well, but the system for IPv4/IPv6 fallback was not quite right when it came to some delayed and fallback calls which are called sooner than they should be when targets are not in the DNS cache. This was obviously not that frequent, but we believe we have addressed it and will be upgrading this evening.
Started 12 May
Closed 13 May

11 May
8 May 15:14:16
We don't usually post about these as there are normally done out of hours when no calls in progress and so cause no down time. However, recently, there was a small issue with some calls not getting through after an upgrade. We are planning an upgrade of one of the VoIP servers this evening, and we will be monitoring for any issues. If you have problems with calls please let us know ASAP. If all goes well we'll do the other server later in the week. The upgrade improves the IPv4/IPv6 fallback handling where customers have both IPv4 and IPv6 endpoints.
10 May 08:24:28
No issues reported so we plan to upgrade the second call server today.
Started 8 May
Closed 11 May
Previously expected 9 May

4 May 18:00:00
[Broadband] - TT Blips - Closed
4 May 16:53:39
Some TT lines blipped at 16:34 and 16:46. It appears that the lines have recovered. We have reported this to TT.
5 May 08:48:41
This was caused by an incident in the TalkTalk network. This is from TalkTalk: "...Network support have advised that the problem was caused by the card failure which has now been taken offline..."
Started 4 May 16:36:27 by AAISP automated checking
Closed 4 May 18:00:00
Cause TT

27 Mar 09:30:00
19 Feb 18:35:15
We have seen some cases with degraded performance on some TT lines, and we are investigating. Not a lot to go on yet, but be assured we are working on this and engaging the engineers within TT to address this.
21 Feb 10:13:20

We have completed further tests and we are seeing congestion manifesting itself as slow throughput at peak times (evenings and weekends) on VDSL (FTTC) lines that connect to us through a certain Talk Talk LAC.

This has been reported to senior TalkTalk staff.

To explain further; VDSL circuits are routed from TalkTalk to us via two LACs. We are seeing slow thoughput at peak times on one LAC and not the other.

27 Feb 11:08:58
Very often with congestion it is easy to find the network port or system that is overloaded but so far, sadly, we've not found the cause. A&A staff and customers and TalkTalk network engineers have done a lot of checks and tests on various bits of the backhaul network but we are finding it difficult to locate the cause of the slow throughput. We are all still working on this and will update again tomorrow.
27 Feb 13:31:39
We've been in discussions with other TalkTalk wholesalers who have also reported the same problem to TalkTalk. There does seem to be more of a general problem within the TalkTalk network.
27 Feb 13:32:12
We have had an update from TalkTalk saying that based on multiple reports from ISPs that they are investigating further.
27 Feb 23:21:21
Further tests this evening by A&A staff shows that the throughput is not relating to a specific LAC, but that it looks like something in TalkTalk is limiting single TCP sessions to 7-9M max during peak times. Running single iperf tests results in 7-9M, but running ten at the same time can fill a 70M circuit. We've passed these findings on to TalkTalk.
28 Feb 09:29:56
As expected the same iperf throughput tests are working fine this morning. TT are shaping at peak times. We are pursuing this with senior TalkTalk staff.
28 Feb 11:27:45
TalkTalk are investigating. They have stated that circuits should not be rate limited and that they are not intentionally rate limiting. They are still investigating the cause.
28 Feb 13:14:52
Update from TalkTalk: Investigations are currently underway with our NOC team who are liaising with Juniper to determine the root cause of this incident.
1 Mar 16:38:54
TalkTalk are able to reproduce the throughput problem and investigations are still on going.
2 Mar 16:51:12
Some customers did see better throughput on Wednesday evening, but not everyone. We've done some further testing with TalkTalk today and they continue to work on this.
2 Mar 22:42:27
We've been in touch with the TalkTalk Network team this evening and have been performing further tests (see https://aastatus.net/2363 ). Investigations are still ongoing, but the work this evening has given a slight clue.
3 Mar 14:24:48
During tests yesterday evening we saw slow throughput when using the Telehouse interconnect and fast (normal) throughput over Harbour Exchange interconnect. Therefore, this morning, we disabled our Telehouse North interconnect. We will carry on running tests over the weekend and we welcome customers to do the same. We are expecting throughput to but fast for everyone. We will then liaise with TalkTalk engineers regarding this on Monday.
6 Mar 15:39:33

Tests over the weekend suggest that speeds are good when we only use our Harbour Exchange interconnect.

TalkTalk are moving the interconnect we have at Telehouse to a different port at their side so as to rule out a possible hardware fault.

6 Mar 16:38:28
TalkTalk have moved our THN port and we will be re-testing this evening. This may cause some TalkTalk customers to experience slow (single thread) downloads this evening. See: https://aastatus.net/2364 for the planned work notice.
6 Mar 21:39:55
The testing has been completed, and sadly we still see slow speeds when using the THN interconnect. We are now back to using the Harbour Exchange interconnect where we are seeing fast speeds as usual.
8 Mar 12:30:25
Further testing happening today: Thursday evening https://aastatus.net/2366 This is to try and help narrow down where the problem is occurring.
9 Mar 23:23:13
We've been testing, tis evening, this time with some more customers, so thank you to those who have been assisting. (We'd welcome more customers to be involved - you just need to run an iperf server on IPv4 or IPv6 and let one of our IPs through your firewall - contact Andrew if you're interested). We'll be passing the results on to TalkTalk, and the investigation continues.
10 Mar 15:13:43
Last night we saw some line slow and some line fast, so having extra lines to test against should help in figuring out why this is the case. Quite a few customers have set up iperf server for us and we are now testing 20+ lines. (Still happy to add more). Speed tests are being run three times an hour and we'll collate the results after the weekend and will report back to TalkTalk the findings.
11 Mar 20:10:21
13 Mar 15:22:43

We now have samples of lines which are affected by the slow throughput and those that are not.

Since 9pm Sunday we are using the Harbour Exchange interconnect in to TalkTalk and so all customers should be seeing fast speeds.

This is still being investigated by us and TalkTalk staff. We may do some more testing in the evenings this week and we are continuing to run iperf tests against the customers who have contacted us.
14 Mar 15:59:18

TalkTalk are doing some work this evening and will be reporting back to us tomorrow. We are also going to be carrying out some tests ourselves this evening too.

Our tests will require us to move traffic over to the Telehouse interconnect, which may mean some customers will see slow (single thread) download speeds at times. This will be between 9pm and 11pm

14 Mar 16:45:49
This is from the weekend:

17 Mar 10:42:28
We've stopped the iperf testing for the time being. We will start it back up again once we or TalkTalk have made changes that require testing to see if things are better or not, but at the moment there is no need for the testing as all customers should be seeing fast speeds due to the Telehouse interconnect not being in use. Customers who would like quota top-ups, please do email in.
17 Mar 18:10:41
To help with the investigations, we're also asking for customers with BT connected FTTC/VDSL lines to run iperf so we can test against them too - details on https://support.aa.net.uk/TTiperf Thank you!
20 Mar 12:54:02
Thanks to those who have set up iperf for us to test against. We ran some tests over the weekend whilst swapping back to the Telehouse interconnect, and tested BT and TT circuits for comparison. Results are that around half the TT lines slowed down but the BT circuits were unaffected.

TalkTalk are arranging some further tests to be done with us which will happen Monday or Tuesday evening this week.

22 Mar 09:37:30
We have scheduled testing of our Telehouse interlink with TalkTalk staff for this Thursday evening. This will not affect customers in any way.
22 Mar 09:44:09
In addition to the interconnect testing on Thursday mentioned above, TalkTalk have also asked us to retest DSL circuits to see if they are still slow. We will perform these tests this tonnight, Wednesday evening.

TT have confirmed that they have made a configuration change on the switch at their end in Telehouse - this is the reason for the speed testing this evening.

22 Mar 12:06:50
We'll be running iperf3 tests against our TT and BT volunteers this evening, very 15 minutes from 4pm through to midnight.
22 Mar 17:40:20
We'll be changing over to the Telehouse interconnect between 8pm and 9pm this evening for testing.
23 Mar 10:36:06

Here are the results from last night:

And BT Circuits:

Some of the results are rather up and down, but these lines are in use by customers so we would expect some fluctuations, but it's clear that a number of lines are unaffected and a number are affected.

Here's the interesting part. Since this problem started we have rolled out some extra logging on to our LNSs, this has taken some time as we only update one a day. However, we are now logging the IP address used at our side of L2TP tunnels from TalkTalk. We have eight live LNSs and each one has 16 IP addresses that are used. With this logging we've identified that circuits connecting over tunnels on 'odd' IPs are fast, whilst those on tunnels on 'even' IPs are slow. This points to a LAG issue within TalkTalk, which is what we have suspected from the start but this data should hopefully help TalkTalk with their investigations.

23 Mar 16:27:28
As mentioned above, we have scheduled testing of our Telehouse interlink with TalkTalk staff for this evening. This will not affect customers in any way.
23 Mar 22:28:53

We have been testing the Telehouse interconnect this evening with TalkTalk engineers. This involved a ~80 minute conference call and setting up a very simple test of a server our side plugged in to the switch which is connected to our 10G interconnect, and running iperf3 tests against a laptop on the TalkTalk side.

The test has highlighted a problem at the TalkTalk end with the connection between two of their switches. When plugged in to the second switch we got about 300Mbit/s, but when their laptop was in the switch directly connected to our interconnect we got near full speed or around 900Mb/s.

This has hopefully given them a big clue and they will now involve the switch vendor for further investigations.

23 Mar 23:02:34
TalkTalk have just called us back and have asked us to retest speeds on broadband circuits. We're moving traffic over to the Telehouse interconnect and will test....
23 Mar 23:07:31
Initial reports show that speeds are back to normal! Hooray! We've asked TalkTalk for more details and if this is a temporary or permanent fix.
24 Mar 09:22:13

Results from last night when we changed over to test the Telehouse interlink:

This shows that unlike the previous times, when we changed over to use the Telehouse interconnect at 11PM speeds did not drop.

We will perform hourly iperf tests over the weekend to be sure that this has been fixed.

We're still awaiting details from TalkTalk as to what the fix was and if it is a temporary or permanent fix.

24 Mar 16:40:24
We are running on the Telehouse interconnect and are running hourly iperf3 tests against a number of our customers over the weekend. This will tell us if the speed issues are fixed.
27 Mar 09:37:12

Speed tests against customers over the weekend do not show the peak time slow downs, this confrims that what TalkTalk did on Thursday night has fixed the problem. We are still awaiting the report from TalkTalk regarding this incident.

The graph above shows iperf3 speed test results taken once an hour over the weekend against nearly 30 customers. Although some are a bit spiky we are no longer seeing the drastic reduction in speeds at peak time. The spikyness is due to the lines being used as normal by the customers and so is expected.

28 Mar 10:52:25
We're expecting the report from TalkTalk at the end of this week or early next week (w/b 2017-04-03).
10 Apr 16:43:03
We've not yet had the report from TalkTalk, but we do expect it soon...
4 May 09:16:33
We've had an update saying: "The trigger & root cause of this problem is still un-explained; however investigations are continuing between our IP Operation engineers and vendor".

This testing is planned for 16th May.

Resolution This has been fixed, we're awaiting the full report from TalkTalk.
Started 18 Feb
Closed 27 Mar 09:30:00
Cause TT

2 May 11:50:00
2 May 11:13:13
We're investigating a problem with incoming calls not working where they are configured to redirect (also-ring) to other number(s).
2 May 11:22:49
This looks to be affecting one of our call servers and not the other, so this explains why calls are working some of the time. Investigations continue.
2 May 11:46:20
We are moving calls off of the affected server, and we are starting to see calls working again.
2 May 11:51:11
Customers with problem with calls may need to get their VoIP phones to re-register - the easiest way is often vie the web interface of the VoIP phone, or by unplugging the power and plugging it back in again!
Started 2 May 10:45:00
Closed 2 May 11:50:00

1 May 14:30:14
[DNS, Email and Web Hosting] - Email server rebuild - Completed
26 Apr 15:15:38
We will be taking one of our customer facing mail servers down for a rebuild on Monday the 1st of May. We will be moving its IP addresses over to an alternative server to keep disruption to a minimum. Once we are happy with the rebuilt server we will put it back into service. This may be the next morning, depending on how long it takes to test the rebuilt server. Although disruption should be minimal, please consider IMAP and POP3 services "at risk" during this time.
1 May 14:31:00
This work has now been completed and the rebuilt mail server is back in service.
Started 1 May 09:00:00
Closed 1 May 14:30:14

26 Apr 14:33:04
25 Apr 12:52:27
Our support pages/wiki will be unavailable for approximately an hour starting at 2017-04-26T13:00:00 whilst the virtual machine is rebuilt.
26 Apr 13:01:53
These planned works are now underway.
Resolution The PEW on the support pages is now complete - the pages went back online shortly after 1400 however some images were not loading due to a permissions problem which has now been resolved .
Started 26 Apr 13:00:00
Closed 26 Apr 14:33:04
Previously expected 26 Apr 14:00:00 (Last Estimated Resolution Time from AAISP)

25 Apr 09:43:08
25 Apr 09:40:38
Unfortunately we are needing to to an emergency reboot of one of our POP3/IMAP servers. This is happening at the moment and will mean access to email will stop for some customers for a couple of minutes. Sorry for this inconvenience.
Resolution The mail server is back online.
Started 25 Apr 09:38:40
Closed 25 Apr 09:43:08

26 Apr 13:39:44
[DNS, Email and Web Hosting] - Email server rebuild - Completed
18 Apr 16:41:37
We will be taking one of our customer facing mail servers down for a rebuild on Tuesday the 25th of April. We will be moving its IP addresses over to an alternative server to keep disruption to a minimum. Once we are happy with the rebuilt server we will put it back into service. This may be the next morning, depending on how long it takes to test the rebuilt server. Although disruption should be minimal, please consider IMAP and POP3 services "at risk" during this time.
26 Apr 13:42:52
The rebuild mail server is in service and operating normally.
Started 25 Apr 08:00:00
Closed 26 Apr 13:39:44
Previously expected 26 Apr 10:00:00

17 May 09:10:43
22 Apr 11:12:14

We are making some major upgrades to the way we do notifications to customers for orders, faults, etc.

One of the first things to change will be the up/down notifications we send. These can be tweets, texts or emails. Part of the work will be changing the options you have for controlling these.

Over the coming weeks we expect to add new settings, and remove old settings over time. We are aiming to try and mirror most of the existing functionality, in particular the sending of tweets or texts for unexpected line drops for individual lines.

You will also start to see options to load a PGP public key for emailed notifications.

Over coming months we expect to move more and more of our systems over to the new process, including order tracking and engineer appointments, etc.

The end result should be more flexible, and consistent, notifications.

23 Apr 17:54:32

We are mostly ready to switch over to the new system - probably happening on Monday. The wording of the messages will change slightly with the new system. There are already more options on the control pages, and once we have switched over these will replace the old settings.

Once done, we will start using the new system for more and more aspects of the control pages, ordering, and faults, over the coming weeks.

24 Apr 14:23:06
We have switched the line up/down messages to the new KCI system. Please do let us know if you have any concerns or comments.
24 Apr 16:17:34
Customers can now paste a PGP public key in to the control pages login page.
25 Apr 16:02:39
After a few comments we have made sure the postcode and login is shown in the line/up down messages. The new system seems to be working well, and we are ready to start extending it to more messages soon.
Started 23 Apr
Closed 17 May 09:10:43
Expected close 1 Jun

22 Apr 11:13:59
2 Feb 21:19:15

22 Apr 11:13:43
4 Apr 18:01:17
The first stage of some updates we are doing involves changes to the accounts systems.

When you log in you will see a Update Contact link which allows you to change some basic preferences with immediate effect. These include whether to send PDF, or XML with documents, and if to use PGP/MIME, and if we send a confirmation link on documents. We may add new options in future.

You can also request a change to contact details (address, email, etc). And if you wish you can paste in a PGP/GPG public key. We email you confirmation (encrypted if you asked for that) and you follow a link to confirm the changes (within 2 days please). If you are encrypting you can click an option to stop doing so, and confirm in the email. If you change email address, we email the old address (using old encryption setting, if any) to advise of the change.

We believe we have all of the emails from accounts covered with this, but if you find any that are not, please do let us know. The only special case is Direct Debit notices. As these are the bank account holder which may not be the same as the A&A account holder, they can be a different email address. As such they only encrypt if you have the same email on the account and have asked for encryption.

If this is all nonsense to you - do not worry, but if you have questions, please let us know.

5 Apr 08:30:30
Having worked fine for my key it seems it is not accepting keys from customers, so we are investigating that now. Thank you all for trying.
5 Apr 08:48:17
This should be working now, sorry for any inconvenience
Started 4 Apr 17:00:00

22 Apr 11:13:22
11 Apr 15:01:24
Our office will be closed from 6PM Thursday 13th April and will be open again at 8AM on Tuesday 18th April.
Started 11 Apr 15:01:01

21 Apr 13:07:49
21 Apr 09:59:57
At 9:30 we saw a number of TalkTalk connected circuits drop and reconnect. We're investigating, but seems like a TalkTalk backhaul issue

21 Apr 11:24:54
Most lines logged back in very quickly. A few are still down. It seems TalkTalk had some sort if incident at the Telehouse Harbour Exchange datacentre, we're expecting further information shortly.
21 Apr 12:54:00
TalkTalk have some sort of outage in their Harbour Exchange datacentre. As we have an interconnect in both Telehouse and Harbour Exchange the traffic has moved over to Telehouse. There are a very small number of customers (<10) who are still offline, they may just need a reboot of their router or modem, but we are contacting them individually.
21 Apr 13:08:47
TalkTalk say: Root cause analysis conducted by the NOC and our Network Support team has identified that this incident was caused by a crashed routing card for the LTS. The card was reloaded by our Network Support team and full service has now been restored.
Resolution Any customer still offline, please reboot your router/modem and then contact Support if still off.
Started 21 Apr 09:30:50
Closed 21 Apr 13:07:49

5 Apr 10:04:33
7 Mar 10:27:14

Customers expecting Direct Debit collection on Wednesday 8th March will actually have the collection made on Thursday 9th March.

We do apologise for any confusion this may cause.

As this is within the permitted 3 working day window in which we can make a collection we will not be sending separate individual notices of this change.

Started 8 Mar

2 Apr 18:11:09
2 Apr 17:18:16
We have had reports of customers not having received emailed invoices. We are looking in to this and hope to ensure any unsent invoices are correctly emailed in the morning. As we have said this is a new email system but we do not yet know the exact reason why some emails seem not to have been sent,
Resolution We have identified the issue - it appears to relate only to emailing of invoices, so Direct Debit notices were sent correctly as far as we can see. Invoices will be resent tonight. Apologies if anyone gets the same invoice sent more than once, but we have decided to re-send all invoices that have been issued this month as the logs suggest it is most if not all invoices that were not sent.
Started 31 Mar
Closed 2 Apr 18:11:09
Previously expected 3 Apr 17:00:07

31 Mar 13:49:33
31 Mar 09:50:00

We have had reports from customers that emailed PDF attachments are not opening.

We have been updating the emailing systems recently (as we plan to launch some new features soon), but after a lot of testing all seemed to be working. We are looking in to the matter now and hope to have it sorted today. Most customers will get invoices tomorrow.

You can, at any time, log in to the accounts systems to access invoices, including downloading a PDF. We appreciate some customers would like the "pretty" version of the invoices to file or print. Please feel free to log in and get copies of your invoices. If necessary you can ask accounts to re-email you the invoice.

We'd like to remind all customers that the plain text content of the emailed invoice is the formal invoice with all of the details for the invoice to be paid in accordance with HMRC/VAT rules, and as such the broken PDF is no reason not to pay the invoice, sorry.

31 Mar 13:27:44
Our tests all working, but still some customers not able to open, so still investigating.
Resolution The problem appears to be additional white space in the base64 coding of the PDF. Whitespace is valid in base64 normally, which is why it worked fine on the various email clients with which we tested this change. However, we have made a change to resolve this and confirmed that the impacted customers can now open PDFs.
Started 30 Mar
Closed 31 Mar 13:49:33
Previously expected 31 Mar 17:12:15

28 Mar 16:00:00
28 Mar 17:43:33

Outgoing external SMS were not working for a period from yesterday afternoon (not exactly sure of time) until this afternoon. We have identified the cause of the problem and it has been rectified. This will have impacted our normal SMS line/up down notifications as well.

The status code returned by our API would have indicated no parts of message sent. The messages are not queued and so will not be re-sent. They were not charged for.

Sorry for any inconvenience. We are looking in to ways we can pick up issues like this sooner in future.

Obviously we appreciate the fault report from the customer that made us aware of this issue.

Started 27 Mar 15:00:00
Closed 28 Mar 16:00:00
Previously expected 28 Mar 16:00:00

19 Mar 11:00:00
16 Mar 11:02:52
We will be performing software upgrades to two of our core network switches on Sunday morning in Telehouse. This will cause a few minutes of disruption for each upgrade. This work will happen between 10AM and noon on Sunday morning. We will have staff at the datacentre overseeing this work.
19 Mar 09:54:14
This work will start shortly and we do expect it to cause a few re-connects for customers whilst the upgrades are happening.
19 Mar 10:44:01
The first switch has been updated, the second switch will be updated in a few minutes.
19 Mar 10:52:38
The second switch has now been upgraded. The upgrade work is complete, we're performing final checks before closing this incident.
Resolution This work has been completed. We were expecting this to cause a brief outage for customers, but in practice the outage did impact customers more that we would have liked. Customers on BT backhaul were more impacted than those on TalkTalk backhaul for example. This type of upgrade is rare but we will be looking in to how things can be improved. We do apologise to customers who were impacted by this.
Started 19 Mar 10:00:00
Closed 19 Mar 11:00:00

14 Mar 21:10:00
14 Mar 21:05:28
Looks like we just had some sort of blip affecting broadband customers. We're investigating.
Resolution This was a LNS crash, and so affected customers on the "i" LNS. The cause is being investigated, but preliminary investigations show that it's probably a problem that is fixed in software that is scheduled to be loaded on to this LNS in a couple of days time as part of the rolling software update that we're performing at the moment.
Broadband Users Affected 12%
Started 14 Mar 21:00:57
Closed 14 Mar 21:10:00

10 Mar 15:27:30
10 Mar 09:06:46
We've had a few reports of calls having one way audio - we're currently investigating.
10 Mar 09:23:00
This looks to be carrier specific so we've disable that carrier for the moment - calls are sounding better. We'll carry on investigating.
10 Mar 09:39:41
The issue seems to be that outbound calls via one of our carriers are failing to connect. We have disabled this carrier for the time being. Our carrier reports that they are currently experiencing DNS issues.
10 Mar 15:30:50
The issue with our supplier has now been resolved and they have been reintroduced into production.
Started 10 Mar 09:01:00
Closed 10 Mar 15:27:30

6 Mar 21:37:45
6 Mar 16:41:32
As part of the slow throughput problem described in https://aastatus.net/2358 we will be performing further tests this evening. This will involve moving TalkTalk traffic to the interconnect which we believe is slow. Customers may see poor speeds this evening during the times that we carry out tests. The tests are expected to last less than 30 minutes between 8 and 10 pm.
Resolution This work has been completed.
Started 6 Mar 20:00:00
Closed 6 Mar 21:37:45

7 Mar 12:02:32
2 Mar 16:31:46
We will be doing some spring cleaning on Monday 6th March at our Maidenhead datacentre. This involves removing old equipment. We don't expect any interruptions but it is an at-risk period.
Resolution This work has been completed.
Started 6 Mar 18:30:00
Closed 7 Mar 12:02:32

2 Mar 22:10:44
2 Mar 21:48:39
Relating to https://aastatus.net/2358 we are undergoing currently in an emergency at-risk period as we perform some tests along side TalkTalk staff. We don't expect any problems, but this work involves re-routing TalkTalk traffic within our network. This work is happening now. Sorry for the no notice.
2 Mar 21:53:05
We have successfully and cleanly moved all TalkTalk traffic off our THN interconnect and on to our HEX Interconnect. (Usually we use both all the time, but for this testing we are forcing traffic through the HEX side)
2 Mar 21:55:52
We're bringing back routing across both links now...
2 Mar 22:03:40
We are now moving traffic to our THN interconnect.
Resolution We're now back to using both the TalkTalk links. Tests completed.
Started 2 Mar 21:46:17
Closed 2 Mar 22:10:44

28 Feb 11:44:16
28 Feb 11:44:16

We cordially invite customers, friends, peers, associates, and small furry creatures from Alpha Centauri to come along, at any time between 2pm and 10pm, and to stay for as long as you like, drink, eat (there will be a BBQ) and be merry.

We wanted to just keep an eye on number of attendees, hence the EventBrite. The tickets are free though, of course.

Sun 2 April 2017 14:00 – 22:00 BST


Started 28 Feb 11:00:00

25 Feb 18:58:41
25 Feb 15:53:47
Our accounts systems, and hence ordering, are off line briefly. A very minor change has proven to take rather longer than expected, and at this stage we have no choice but to simply wait for the process to complete. Sorry for any inconvenience.
25 Feb 16:46:16
This is progressing, but could take until 6pm at the current rate. Apologies for any inconvenience.
25 Feb 18:00:08
We are making progress, but will be a while longer. Sorry for inconvenience.
Resolution Finally completed, sorry for hassle.
Started 25 Feb 15:30:00
Closed 25 Feb 18:58:41
Previously expected 25 Feb 17:00:00

13 Jun 2015 10:57:07
12 Mar 2015 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.
Closed 13 Jun 2015 10:57:07

13 Jun 2015 10:57:07
12 Jun 2015 11:00:23
Our office connectivity blipped. We have internet again, but our phones are down. We're investigating. This is not customer affecting.
12 Jun 2015 11:06:49
Phones are working again.
Closed 13 Jun 2015 10:57:07
Previously expected 12 Jun 2015 14:57:07