6 Apr 01:00:00
Details
Yesterday 11:19:03
Due to maintenance being performed by a carrier, disruption of up to 15 minutes to 3G/4G data services is expected some time between 01:00 and 03:00 on the 6th of April.
Planned start 6 Apr 01:00:00
Expected close 6 Apr 03:00:00

28 Mar 16:00:00
Details
28 Mar 17:43:33

Outgoing external SMS were not working for a period from yesterday afternoon (not exactly sure of time) until this afternoon. We have identified the cause of the problem and it has been rectified. This will have impacted our normal SMS line/up down notifications as well.

The status code returned by our API would have indicated no parts of message sent. The messages are not queued and so will not be re-sent. They were not charged for.

Sorry for any inconvenience. We are looking in to ways we can pick up issues like this sooner in future.

Obviously we appreciate the fault report from the customer that made us aware of this issue.

Started 27 Mar 15:00:00
Closed 28 Mar 16:00:00
Previously expected 28 Mar 16:00:00

27 Mar 09:30:00
Details
19 Feb 18:35:15
We have seen some cases with degraded performance on some TT lines, and we are investigating. Not a lot to go on yet, but be assured we are working on this and engaging the engineers within TT to address this.
Update
21 Feb 10:13:20

We have completed further tests and we are seeing congestion manifesting itself as slow throughput at peak times (evenings and weekends) on VDSL (FTTC) lines that connect to us through a certain Talk Talk LAC.

This has been reported to senior TalkTalk staff.

To explain further; VDSL circuits are routed from TalkTalk to us via two LACs. We are seeing slow thoughput at peak times on one LAC and not the other.

Update
27 Feb 11:08:58
Very often with congestion it is easy to find the network port or system that is overloaded but so far, sadly, we've not found the cause. A&A staff and customers and TalkTalk network engineers have done a lot of checks and tests on various bits of the backhaul network but we are finding it difficult to locate the cause of the slow throughput. We are all still working on this and will update again tomorrow.
Update
27 Feb 13:31:39
We've been in discussions with other TalkTalk wholesalers who have also reported the same problem to TalkTalk. There does seem to be more of a general problem within the TalkTalk network.
Update
27 Feb 13:32:12
We have had an update from TalkTalk saying that based on multiple reports from ISPs that they are investigating further.
Update
27 Feb 23:21:21
Further tests this evening by A&A staff shows that the throughput is not relating to a specific LAC, but that it looks like something in TalkTalk is limiting single TCP sessions to 7-9M max during peak times. Running single iperf tests results in 7-9M, but running ten at the same time can fill a 70M circuit. We've passed these findings on to TalkTalk.
Update
28 Feb 09:29:56
As expected the same iperf throughput tests are working fine this morning. TT are shaping at peak times. We are pursuing this with senior TalkTalk staff.
Update
28 Feb 11:27:45
TalkTalk are investigating. They have stated that circuits should not be rate limited and that they are not intentionally rate limiting. They are still investigating the cause.
Update
28 Feb 13:14:52
Update from TalkTalk: Investigations are currently underway with our NOC team who are liaising with Juniper to determine the root cause of this incident.
Update
1 Mar 16:38:54
TalkTalk are able to reproduce the throughput problem and investigations are still on going.
Update
2 Mar 16:51:12
Some customers did see better throughput on Wednesday evening, but not everyone. We've done some further testing with TalkTalk today and they continue to work on this.
Update
2 Mar 22:42:27
We've been in touch with the TalkTalk Network team this evening and have been performing further tests (see https://aastatus.net/2363 ). Investigations are still ongoing, but the work this evening has given a slight clue.
Update
3 Mar 14:24:48
During tests yesterday evening we saw slow throughput when using the Telehouse interconnect and fast (normal) throughput over Harbour Exchange interconnect. Therefore, this morning, we disabled our Telehouse North interconnect. We will carry on running tests over the weekend and we welcome customers to do the same. We are expecting throughput to but fast for everyone. We will then liaise with TalkTalk engineers regarding this on Monday.
Update
6 Mar 15:39:33

Tests over the weekend suggest that speeds are good when we only use our Harbour Exchange interconnect.

TalkTalk are moving the interconnect we have at Telehouse to a different port at their side so as to rule out a possible hardware fault.

Update
6 Mar 16:38:28
TalkTalk have moved our THN port and we will be re-testing this evening. This may cause some TalkTalk customers to experience slow (single thread) downloads this evening. See: https://aastatus.net/2364 for the planned work notice.
Update
6 Mar 21:39:55
The testing has been completed, and sadly we still see slow speeds when using the THN interconnect. We are now back to using the Harbour Exchange interconnect where we are seeing fast speeds as usual.
Update
8 Mar 12:30:25
Further testing happening today: Thursday evening https://aastatus.net/2366 This is to try and help narrow down where the problem is occurring.
Update
9 Mar 23:23:13
We've been testing, tis evening, this time with some more customers, so thank you to those who have been assisting. (We'd welcome more customers to be involved - you just need to run an iperf server on IPv4 or IPv6 and let one of our IPs through your firewall - contact Andrew if you're interested). We'll be passing the results on to TalkTalk, and the investigation continues.
Update
10 Mar 15:13:43
Last night we saw some line slow and some line fast, so having extra lines to test against should help in figuring out why this is the case. Quite a few customers have set up iperf server for us and we are now testing 20+ lines. (Still happy to add more). Speed tests are being run three times an hour and we'll collate the results after the weekend and will report back to TalkTalk the findings.
Update
11 Mar 20:10:21
Update
13 Mar 15:22:43

We now have samples of lines which are affected by the slow throughput and those that are not.

Since 9pm Sunday we are using the Harbour Exchange interconnect in to TalkTalk and so all customers should be seeing fast speeds.

This is still being investigated by us and TalkTalk staff. We may do some more testing in the evenings this week and we are continuing to run iperf tests against the customers who have contacted us.
Update
14 Mar 15:59:18

TalkTalk are doing some work this evening and will be reporting back to us tomorrow. We are also going to be carrying out some tests ourselves this evening too.

Our tests will require us to move traffic over to the Telehouse interconnect, which may mean some customers will see slow (single thread) download speeds at times. This will be between 9pm and 11pm

Update
14 Mar 16:45:49
This is from the weekend:

Update
17 Mar 10:42:28
We've stopped the iperf testing for the time being. We will start it back up again once we or TalkTalk have made changes that require testing to see if things are better or not, but at the moment there is no need for the testing as all customers should be seeing fast speeds due to the Telehouse interconnect not being in use. Customers who would like quota top-ups, please do email in.
Update
17 Mar 18:10:41
To help with the investigations, we're also asking for customers with BT connected FTTC/VDSL lines to run iperf so we can test against them too - details on https://support.aa.net.uk/TTiperf Thank you!
Update
20 Mar 12:54:02
Thanks to those who have set up iperf for us to test against. We ran some tests over the weekend whilst swapping back to the Telehouse interconnect, and tested BT and TT circuits for comparison. Results are that around half the TT lines slowed down but the BT circuits were unaffected.

TalkTalk are arranging some further tests to be done with us which will happen Monday or Tuesday evening this week.

Update
22 Mar 09:37:30
We have scheduled testing of our Telehouse interlink with TalkTalk staff for this Thursday evening. This will not affect customers in any way.
Update
22 Mar 09:44:09
In addition to the interconnect testing on Thursday mentioned above, TalkTalk have also asked us to retest DSL circuits to see if they are still slow. We will perform these tests this tonnight, Wednesday evening.

TT have confirmed that they have made a configuration change on the switch at their end in Telehouse - this is the reason for the speed testing this evening.

Update
22 Mar 12:06:50
We'll be running iperf3 tests against our TT and BT volunteers this evening, very 15 minutes from 4pm through to midnight.
Update
22 Mar 17:40:20
We'll be changing over to the Telehouse interconnect between 8pm and 9pm this evening for testing.
Update
23 Mar 10:36:06

Here are the results from last night:

And BT Circuits:

Some of the results are rather up and down, but these lines are in use by customers so we would expect some fluctuations, but it's clear that a number of lines are unaffected and a number are affected.

Here's the interesting part. Since this problem started we have rolled out some extra logging on to our LNSs, this has taken some time as we only update one a day. However, we are now logging the IP address used at our side of L2TP tunnels from TalkTalk. We have eight live LNSs and each one has 16 IP addresses that are used. With this logging we've identified that circuits connecting over tunnels on 'odd' IPs are fast, whilst those on tunnels on 'even' IPs are slow. This points to a LAG issue within TalkTalk, which is what we have suspected from the start but this data should hopefully help TalkTalk with their investigations.

Update
23 Mar 16:27:28
As mentioned above, we have scheduled testing of our Telehouse interlink with TalkTalk staff for this evening. This will not affect customers in any way.
Update
23 Mar 22:28:53

We have been testing the Telehouse interconnect this evening with TalkTalk engineers. This involved a ~80 minute conference call and setting up a very simple test of a server our side plugged in to the switch which is connected to our 10G interconnect, and running iperf3 tests against a laptop on the TalkTalk side.

The test has highlighted a problem at the TalkTalk end with the connection between two of their switches. When plugged in to the second switch we got about 300Mbit/s, but when their laptop was in the switch directly connected to our interconnect we got near full speed or around 900Mb/s.

This has hopefully given them a big clue and they will now involve the switch vendor for further investigations.

Update
23 Mar 23:02:34
TalkTalk have just called us back and have asked us to retest speeds on broadband circuits. We're moving traffic over to the Telehouse interconnect and will test....
Update
23 Mar 23:07:31
Initial reports show that speeds are back to normal! Hooray! We've asked TalkTalk for more details and if this is a temporary or permanent fix.
Update
24 Mar 09:22:13

Results from last night when we changed over to test the Telehouse interlink:

This shows that unlike the previous times, when we changed over to use the Telehouse interconnect at 11PM speeds did not drop.

We will perform hourly iperf tests over the weekend to be sure that this has been fixed.

We're still awaiting details from TalkTalk as to what the fix was and if it is a temporary or permanent fix.

Update
24 Mar 16:40:24
We are running on the Telehouse interconnect and are running hourly iperf3 tests against a number of our customers over the weekend. This will tell us if the speed issues are fixed.
Update
27 Mar 09:37:12

Speed tests against customers over the weekend do not show the peak time slow downs, this confrims that what TalkTalk did on Thursday night has fixed the problem. We are still awaiting the report from TalkTalk regarding this incident.

The graph above shows iperf3 speed test results taken once an hour over the weekend against nearly 30 customers. Although some are a bit spiky we are no longer seeing the drastic reduction in speeds at peak time. The spikyness is due to the lines being used as normal by the customers and so is expected.

Update
28 Mar 10:52:25
We're expecting the report from TalkTalk at the end of this week or early next week (w/b 2017-04-03).
Resolution This has been fixed, we're awaiting the full report from TalkTalk.
Started 18 Feb
Closed 27 Mar 09:30:00
Cause TT

19 Mar 11:00:00
Details
16 Mar 11:02:52
We will be performing software upgrades to two of our core network switches on Sunday morning in Telehouse. This will cause a few minutes of disruption for each upgrade. This work will happen between 10AM and noon on Sunday morning. We will have staff at the datacentre overseeing this work.
Update
19 Mar 09:54:14
This work will start shortly and we do expect it to cause a few re-connects for customers whilst the upgrades are happening.
Update
19 Mar 10:44:01
The first switch has been updated, the second switch will be updated in a few minutes.
Update
19 Mar 10:52:38
The second switch has now been upgraded. The upgrade work is complete, we're performing final checks before closing this incident.
Resolution This work has been completed. We were expecting this to cause a brief outage for customers, but in practice the outage did impact customers more that we would have liked. Customers on BT backhaul were more impacted than those on TalkTalk backhaul for example. This type of upgrade is rare but we will be looking in to how things can be improved. We do apologise to customers who were impacted by this.
Started 19 Mar 10:00:00
Closed 19 Mar 11:00:00

9 Mar 20:00:00
Details
8 Mar 12:29:14

We continue to work with TalkTalk to get to the bottom of the slow throughput issue as described on https://aastatus.net/2358

We will be performing some routing changes and tests this afternoon and this evening, we are not expecting this to cause any drops for customers, but this evening there will be times when throughput for 'single thread' downloads will be slow. Sorry for the short notice, please bear with us, this is being a tricky fault to track down.

Update
8 Mar 22:39:39
Sorry, due to TalkTalk needing extra time to prepare for their changes this work has been moved to Thursday 9th evening.
Started 9 Mar 20:00:00
Update was expected 9 Mar 23:00:00

14 Mar 21:10:00
Details
14 Mar 21:05:28
Looks like we just had some sort of blip affecting broadband customers. We're investigating.
Resolution This was a LNS crash, and so affected customers on the "i" LNS. The cause is being investigated, but preliminary investigations show that it's probably a problem that is fixed in software that is scheduled to be loaded on to this LNS in a couple of days time as part of the rolling software update that we're performing at the moment.
Broadband Users Affected 12%
Started 14 Mar 21:00:57
Closed 14 Mar 21:10:00

10 Mar 15:27:30
Details
10 Mar 09:06:46
We've had a few reports of calls having one way audio - we're currently investigating.
Update
10 Mar 09:23:00
This looks to be carrier specific so we've disable that carrier for the moment - calls are sounding better. We'll carry on investigating.
Update
10 Mar 09:39:41
The issue seems to be that outbound calls via one of our carriers are failing to connect. We have disabled this carrier for the time being. Our carrier reports that they are currently experiencing DNS issues.
Update
10 Mar 15:30:50
The issue with our supplier has now been resolved and they have been reintroduced into production.
Started 10 Mar 09:01:00
Closed 10 Mar 15:27:30

7 Mar 12:02:32
Details
2 Mar 16:31:46
We will be doing some spring cleaning on Monday 6th March at our Maidenhead datacentre. This involves removing old equipment. We don't expect any interruptions but it is an at-risk period.
Resolution This work has been completed.
Started 6 Mar 18:30:00
Closed 7 Mar 12:02:32

7 Mar 10:27:14
Details
7 Mar 10:27:14

Customers expecting Direct Debit collection on Wednesday 8th March will actually have the collection made on Thursday 9th March.

We do apologise for any confusion this may cause.

As this is within the permitted 3 working day window in which we can make a collection we will not be sending separate individual notices of this change.

Started 8 Mar

6 Mar 21:37:45
Details
6 Mar 16:41:32
As part of the slow throughput problem described in https://aastatus.net/2358 we will be performing further tests this evening. This will involve moving TalkTalk traffic to the interconnect which we believe is slow. Customers may see poor speeds this evening during the times that we carry out tests. The tests are expected to last less than 30 minutes between 8 and 10 pm.
Resolution This work has been completed.
Started 6 Mar 20:00:00
Closed 6 Mar 21:37:45

2 Mar 22:10:44
Details
2 Mar 21:48:39
Relating to https://aastatus.net/2358 we are undergoing currently in an emergency at-risk period as we perform some tests along side TalkTalk staff. We don't expect any problems, but this work involves re-routing TalkTalk traffic within our network. This work is happening now. Sorry for the no notice.
Update
2 Mar 21:53:05
We have successfully and cleanly moved all TalkTalk traffic off our THN interconnect and on to our HEX Interconnect. (Usually we use both all the time, but for this testing we are forcing traffic through the HEX side)
Update
2 Mar 21:55:52
We're bringing back routing across both links now...
Update
2 Mar 22:03:40
We are now moving traffic to our THN interconnect.
Resolution We're now back to using both the TalkTalk links. Tests completed.
Started 2 Mar 21:46:17
Closed 2 Mar 22:10:44

2 Mar 11:14:48
Details
7 Feb 14:32:32

We are seeing issues with IPv6 on a few VDSL cabinets serving our customers. There is no apparent geographical commonality amongst these, as far as we can tell.

Lines pass IPv4 fine, but only intermittently passing IPv6 TCP/UDP for brief amounts of time, usually 4 or so packets, before breaking. Customers have tried BT modem, Asus modem, and our supplied ZyXEL as a modem and router, no difference on any. We also lent them a FireBrick to do some traffic dumps.

Traffic captures at our end and the customer end show that the IPv6 TCP and UDP packets are leaving us but not reaching the customer. ICMP (eg pings) do work.

The first case was reported to us in August 2016, and it has taken a while to get to this point. Until very recently there was only a single reported case. Now that we have four cases we have a bit more information and are able to look at commonalities between them.

Of these circuits, two are serving customers via TalkTalk and two are serving customers via BT backhaul. So this isn't a "carrier network issue", as far as we can make out. The only thing that we can find that is common is that the cabinets are all ECI. (Actually - one of the BT connected customers has migrated to TalkTalk backhaul (still with us, using the same cabinet and phone line etc) and the IPv6 bug has also moved to the new circuit via TalkTalk as the backhaul provider)

We are working with senior TalkTalk engineers to try to perform a traffic capture at the exchange - at the point the traffic leaves TalkTalk equipment and is passed on to Openreach - this will show if the packets are making it that far and will help in pinning down the point at which packets are being lost. Understandably this requires TalkTalk engineers working out of hours to perform this traffic capture and we're currently waiting for when this will happen.

Update
2 Mar 11:14:48
Packet captures on an affected circuit carried out by TalkTalk have confirmed that this issue most likely lies in the Openreach network. Circuits that we have been made aware of are being pursued with both BT and TalkTalk for Openreach to make further investigations into the issue.
If you believe you may be affected please do contact support.
Update
17 Mar 09:44:00
Having had TalkTalk capture the traffic in the exchange, the next step is to capture traffic at the road-side cabinet. This is being progresses with Openreach and we hope this to happen 'soon'.
Update
Wednesday 09:52:52
We've received an update from BT advising that they have been able to replicate the missing IPv6 packets, this is believed to be a bug which they are pursuing with the vendor.

In the mean time they have also identified a fix which they are working to deploy. We're currently awaiting further details regarding this, and will update this post once further details become known.
Broadband Users Affected 0.05%
Started 7 Feb 09:00:00 by AA Staff

8 Feb
Details
8 Feb 15:30:07
We are expecting more router upgrades later in the month - this should address a couple of issues we have seen the last few weeks. These should have little or no disruption. Routers are normally done in evening or over night. LNSs are done as a rolling change moving lines to a new LNS over night for a series of nights. Do note that you can configure a preferred time of night on our control pages. Exact dates for upgrades not determined yet but should all be done by end of Feb.
Update
14 Feb 17:53:22
We will be doing some of the routers this evening, and starting the rolling LNS upgrades tonight.
Update
15 Feb 17:03:39
We will be doing the rest of the routers this evening, and continuing the LNS roll over which will run for several more days (we have quite a few LNSs now).
Update
27 Feb 10:02:24
We are expecting to do further upgrades at the start of March.
Update
1 Mar 16:35:50
We are starting a rolling LNS upgrade tonight, and doing some BGP router upgrades this evening.
Started 8 Feb
Expected close Tomorrow

28 Feb 11:44:16
Details
28 Feb 11:44:16

We cordially invite customers, friends, peers, associates, and small furry creatures from Alpha Centauri to come along, at any time between 2pm and 10pm, and to stay for as long as you like, drink, eat (there will be a BBQ) and be merry.

We wanted to just keep an eye on number of attendees, hence the EventBrite. The tickets are free though, of course.

Sun 2 April 2017 14:00 – 22:00 BST

https://www.eventbrite.co.uk/e/aaispissup-aka-nerdstock-2017-tickets-32350365815

Started 28 Feb 11:00:00