Order posts by limited to posts

Yesterday 20:23:24
Details
17 Jun 15:24:16
We've seen very slight packet loss on a number of TalkTalk connected lines this week in the evenings. This looks to be congestion, it's may show up on our CQM graphs as a few pixels of red at the top of the graph between 7pm and midnight. We have an incident open with TalkTalk. We moved traffic to our Telehouse interconnect on Friday afternoon and Friday evening looked to be better. This may mean that th econgestion is related to TalkTalk in Harbour Exchange, but it's a little too early to tell at the moment. We are monitoring this and will update again after the weekend.
Update
19 Jun 16:49:34

TalkTalk did some work on the Telehouse side of our interconnect on Friday as follows:

"The device AA connect into is a chassis with multiple cards and interfaces creating a virtual switch. The physical interface AA plugged into was changed to another physical interface. We suspect this interface to be faulty as when swapped to another it looks to have resolved the packet loss."

We will be testing both of our interconnects individually over the next couple of days.

Update
20 Jun 10:29:05
TalkTalk are doing some work on our Harbour Exchange side today. Much like the work they did on the Telehouse side, they are moving our port. This will not affect customers though.
Update
28 Jun 20:46:34

Sadly, we are still seeing very low levels of packetloss on some TalkTalk connected circuits in the evenings. We have raised this with TalkTalk today, they have investigated this afternoon and say: "Our Network team have been running packet captures at Telehouse North and replicated the packet loss. We have raised this into our vendor as a priority and are due an update tomorrow."

We'll keep this post updated.

Update
29 Jun 22:12:17

Update from TalkTalk regarding their investigations today:- Our engineering team have been working through this all day with the Vendor. I have nothing substantial for you just yet, I have been told I will receive a summary of today's events this evening but I expect the update to be largely "still under investigation". Either way I will review and fire an update over as soon as I receive it. Our Vendor are committing to a more meaningful update by midday tomorrow as they continue to work this overnight.

Update
1 Jul 09:39:48
Update from TT: Continued investigation with Juniper, additional PFE checks performed. Currently seeing the drops on both VC stacks at THN and Hex. JTAC have requested additional time to investigate the issue. They suspect they have an idea what the problem is, however they need to go through the data captures from today to confirm that it is a complete match. Actions Juniper - Review logs captured today, check with engineering. Some research time required, Juniper hope to have an update by CoB Monday. Discussions with engineering will be taking place during this time.
Update
2 Jul 21:19:57

Here is an example - the loss is quite small on individual lines, but as we are seeing this sort of loss on many circuits and the same time (evenings) it make this more severe. It's only due to to our constant monitoring that this gets picked up.

Update
3 Jul 21:47:31
Today's update from Talktalk: "JTAC [TT's vendor's support] have isolated the issue to one FPC [(Flexible PIC Concentrator] and now need Juniper Engineering to investigate further... unfortunately Engineering are US-based and have a public holiday which will potentially delay progress... Actions: Juniper - Review information by [TalkTalk] engineering – Review PRs - if this is a match to a known issue or it's new. Some research time required, Juniper hope to have an update by Thursday"
Update
7 Jul 08:41:26
Update from TalkTalk yesterday evening: "Investigations have identified a limitation when running a mix mode VC (EX4200’s and EX4550's), the VC cable runs at 16gbps rather than 32gbps (16gbps each way). This is why we are seeing slower than expected speeds between VC’s. Our engineering team are working with the vendor exploring a number of solutions."
Update
17 Jul 14:29:29

Saturday 15th and Sunday 16th evenings were a fair bit worse than previous evenings. On Saturday and Sunday evening we saw higher levels of packet loss (between 1% and 3% on many lines) and we also saw slow single TCP thread speeds much like we saw in April. We did contact TalkTalk over the weekend and this has been blamed on a faulty card that TalkTalk had on Thursday that was replaced but has caused traffic imbalance on this part of the network.

We expect things to improve but we will be closely monitoring this on Monday evening (17th) and will report back on Tuesday.

Update
Yesterday 20:23:24
TalkTalk are planning network hardware changes relating to this in the early hours of 1st August. Details here: https://aastatus.net/2414
Started 14 Jun 15:00:00
Update expected 1 Aug 11:00:00

13 Jul 11:21:37
[Broadband] - TT blip - Open
Details
13 Jul 11:21:37
We are investigating an issue with some TalkTalk lines that disconnected at 10:51 this morning, most have come back but there are about 20 that are still off line. We are chasing TalkTalk business.
Update
13 Jul 11:23:50
Latest update from TT..... We have just had further reports from other reseller are also experiencing mass amount of circuit drops at the similar time. This is currently being investigated by our NOC team and updates to follow after investigation.
Started 13 Jul 10:51:49 by AAISP Pro Active Monitoring Systems
Previously expected 13 Jul 15:19:49

29 Jun 22:10:54
Details
26 Apr 10:16:02
We have identified packet loss across our lines at MOSS SIDE that occurs between 8pm to 10pm. We have raised this with BT who suggest that they hope to have this resolved on May 30th. We will update you on completion of this work.
Broadband Users Affected 0.15%
Started 26 Apr 10:13:41 by BT
Update was expected 30 May 12:00:00
Previously expected 30 May (Last Estimated Resolution Time from BT)

29 Jun 20:00:00
Details
27 Jun 16:50:52
Our voice SIM carrier is carrying out emergency maintenance on their GGSNs between 20:00 and 00:00 on the 29th of June. This is expected to cause at least 15 minutes of downtime for voice SIMs.
Update
1 Jul 10:00:59
Our carrier has planned additional work which may affect users on the 5th, 6th, 11th, and 12th of July in the early hours of the morning.
Started 29 Jun 20:00:00 by Carrier
Previously expected 30 Jun 00:05:00

28 Jun 02:00:00
Details
15 Jun 15:38:56
We have been advised of essential scheduled partner maintenance to upgrade core infrastructure. The window is as follows; Service Impact Start Time: 28/6/2017 02:00 Service Impact End Time: 28/6/2017 04:30 Impact Time Expected: 30 minutes Throughout the duration of this window, customers may see disruption of up to 30 minutes to their 3G/4G data services in the following areas: Ilford, Barking, Dagenham, Woolwich Thamesmead, Bexleyheath, East Ham, West Ham, Poplar, Stepney, Bow, Greenwich Deptford, Lewisham and surrounding areas. If you have any questions regarding this maintenance, please don't hesitate to contact our support team on 03333 400 999.
Started 28 Jun 02:00:00 by Carrier
Previously expected 28 Jun 04:30:00 (Last Estimated Resolution Time from Carrier)

12 Jun 10:01:38
[DNS, Email and Web Hosting] - Issues with outbound email - Open
Details
1 Jun 10:27:24
One of the key IP reputation services appears to have blacklisted both of our outgoing servers which is causing issues for some email customers. We have noticed this occurring for any emails destined for icloud.com or mac.com email addresses, however some other servers may be affected. We have contacted the service in question requesting that we are unblocked. We will update this status page as any updates become available.
Update
1 Jun 11:01:17
We have adjusted our outgoing servers to use different IPs to circumvent the block. We would now suggest that mail customers attempt to resend any failed messages.
Update
12 Jun 09:41:39
We are seeing this again unfortunately. We are investigating the cause of this, however we may have to wait for any blocks to expire.
Update
12 Jun 10:01:38
We've applied a work around for the time being and sending email should be fine. We'll investigate the cause of why we were blocked in the first place and why our normal methods for blocking junk email in the first place did not work this time.
Started 1 Jun 06:55:00 by AA Staff

25 May 02:00:00
Details
22 May 16:29:22
Due to upstream upgrade work on core infrastructure, there may be up to 2 hours 30 minutes disruption to data SIM services on the morning of Thursday the 25th of May between the hours of 2:00 and 4:30.
Started 25 May 02:00:00 by Carrier
Previously expected 25 May 04:30:00

6 Apr 01:00:00
Details
30 Mar 11:19:03
Due to maintenance being performed by a carrier, disruption of up to 15 minutes to 3G/4G data services is expected some time between 01:00 and 03:00 on the 6th of April.
Started 6 Apr 01:00:00
Previously expected 6 Apr 03:00:00

9 Mar 20:00:00
Details
8 Mar 12:29:14

We continue to work with TalkTalk to get to the bottom of the slow throughput issue as described on https://aastatus.net/2358

We will be performing some routing changes and tests this afternoon and this evening, we are not expecting this to cause any drops for customers, but this evening there will be times when throughput for 'single thread' downloads will be slow. Sorry for the short notice, please bear with us, this is being a tricky fault to track down.

Update
8 Mar 22:39:39
Sorry, due to TalkTalk needing extra time to prepare for their changes this work has been moved to Thursday 9th evening.
Started 9 Mar 20:00:00
Update was expected 9 Mar 23:00:00

7 Feb 14:49:55
[DNS, Email and Web Hosting] - SMTP Settings Change - Open
Details
7 Feb 14:52:42
For historical reasons, our SMTP servers allow sending authenticated email without TLS. This is insecure, and doesn't belong on the modern internet as it is possible for the username and password to be intercepted by a third party. We will no longer allow this as of the 4th of July. We are emailing customers who seem to be using insecure settings to warn them about the change. We have a support site page about what settings to change here: https://support.aa.net.uk/Enable_TLS_on_smtp.aa.net.uk Please contact support if you have any questions.
Started 7 Feb 14:49:55
Previously expected 4 Jul 09:00:00

20 Sep 2016 11:26:07
Details
20 Sep 2016 11:31:39
Our upstream provider have advised us that they will be carrying out firmware upgrades on core 3G infrastructure on 23rd September 2016 between 00:10 and 04:30 BST.
During this period data SIMs may briefly disconnect as sessions are migrated to other nodes in the upstream provider's network to facilitate the upgrades.
VoIP and SIMs Users Affected 25%
Started 20 Sep 2016 11:26:07 by Carrier
Update was expected 23 Sep 2016 11:30:00
Previously expected 23 Sep 2016 04:30:00 (Last Estimated Resolution Time from Carrier)

01 Sep 2016 17:18:26
Details
01 Sep 2016 17:26:51
Our SMS gateway has always supported HTTPS but it still allows using HTTP only. As using HTTP only is probably a mistake these days we are going to change our gateway to redirect all HTTP requests to HTTPS next week. We don't expect anything to break as curl, etc., should "just work" but we're posting a PEW just in case anyone has a legacy script they think might need adjusting!
Update
09 Sep 2016 10:06:35
Unfortunately we've had to back out of making this change this week as it caused some unforeseen problems! We'll resolve those problems and make the change again soon.
Started 01 Sep 2016 17:18:26
Previously expected 07 Sep 2016 17:18:26

20 Jul 2016 09:58:43
[Maidenhead Colocation] - Web and email outage - Open
Details
04 Apr 2013 15:47:25

This is ongoing. We're investigating.

Update
04 Apr 2013 16:07:41

This should now be fixed. Please let support know if you see any problems, or have any questions.

Update
05 Apr 2013 09:15:04

This was resolved yesterday afternoon.

Update
20 Jul 2016 09:58:43
This may be related to a wider issue on the internet with a power outage at a major london data centre. Thos routing problems are still ongoing.
Started 04 Apr 2013 15:46:11

13 Apr 2015
Details
09 Apr 2015 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr 2015

08 Jan 2015 12:39:06
Details
08 Jan 2015 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
Update
09 Jan 2015 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
Update
12 Jan 2015 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 08 Jan 2015 12:39:06 by AA Staff
Previously expected 15 Jan 2015 17:00:00

03 Jun 2014 17:00:00
Details
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
Update
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
Details
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
Update
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

Update
22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
Update
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
Update
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
Update
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
Update
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
Update
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
Details
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
Details
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
Update
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
Details
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

Details
Yesterday 20:21:08

The following is from TalkTalk and relates to this incident post: https://aastatus.net/2401

"The wholesale LTS platform has been suffering from packet loss and congestion over the last few months. Juniper have now advised that this is due to a limitation with the hardware we have and we need to carry out some essential vendor recommended maintenance on our DSL Interconnects platform/switch at Telehouse North in order to fix customer slow throughput.

This RFC is to rebuild ldn-vc1.thn from a mixed EX4200/EX4550 estate to a entirely EX4550 virtual switch.

We have a limitation with our EX4550's - running in a mostly EX4200 VC (mixed mode), whereby the VC cables/modules will not support any more bandwidth past 16gbps (each way). This limitation is because we are running a mixed mode VC. We therefore need to upgrade this VC to run all EX4550s which will mean the VC cables/modules will support 32gbps and the hope is that low speed issues will be resolved."

During this work AAISP will move traffic over to our Harbour Exchange interconnect so as to minimise the impact on our customers, however there still may be drops between midnight and 4am.

Expected close 1 Aug 04:00:00

[DNS, Email and Web Hosting] - At Risk Period for Web Hosting - Open
Details
21 Feb 14:43:14
We are carrying out maintenance on our customer facing web servers during this Thursday's maintenance window. We expect no more than a couple of minutes of downtime but web services should be considered "at risk" during the work.
Previously expected 23 Feb 22:00:00

Details
06 Jul 2015 12:49:42
We have been advised by Three of works on their network (22:30 8th July to 04:50 9th July 2015) Which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some partners may experience dropped connections. Any device reconnecting will automatically be routed via a new path. We apologise for any inconvenience this may cause and the short notice of this advisory.

Details
12 Feb 2015 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

20 Jul 12:34:33
Details
20 Jul 12:20:48
We're investigating a routing problem that started a few minutes ago affecting broadband and general routing across our network.
Update
20 Jul 12:24:33
This is affecting our services between London and maidenhead, so is affecting some Ethernet circuits too.
Update
20 Jul 12:33:34
Things are getting back to normal now.
Resolution Unfortunately, this was caused by human error in a configuration on one of our core switches. The work was being done as part of our investigations to the problems we had last week, and this change was really not meant to cause any issue, except a mistake was made in the configuration. The change was rolled back and the process is being reviewed.
Started 20 Jul 12:15:00
Closed 20 Jul 12:34:33

19 Jul
Details
7 Feb 14:32:32

We are seeing issues with IPv6 on a few VDSL cabinets serving our customers. There is no apparent geographical commonality amongst these, as far as we can tell.

Lines pass IPv4 fine, but only intermittently passing IPv6 TCP/UDP for brief amounts of time, usually 4 or so packets, before breaking. Customers have tried BT modem, Asus modem, and our supplied ZyXEL as a modem and router, no difference on any. We also lent them a FireBrick to do some traffic dumps.

Traffic captures at our end and the customer end show that the IPv6 TCP and UDP packets are leaving us but not reaching the customer. ICMP (eg pings) do work.

The first case was reported to us in August 2016, and it has taken a while to get to this point. Until very recently there was only a single reported case. Now that we have four cases we have a bit more information and are able to look at commonalities between them.

Of these circuits, two are serving customers via TalkTalk and two are serving customers via BT backhaul. So this isn't a "carrier network issue", as far as we can make out. The only thing that we can find that is common is that the cabinets are all ECI. (Actually - one of the BT connected customers has migrated to TalkTalk backhaul (still with us, using the same cabinet and phone line etc) and the IPv6 bug has also moved to the new circuit via TalkTalk as the backhaul provider)

We are working with senior TalkTalk engineers to try to perform a traffic capture at the exchange - at the point the traffic leaves TalkTalk equipment and is passed on to Openreach - this will show if the packets are making it that far and will help in pinning down the point at which packets are being lost. Understandably this requires TalkTalk engineers working out of hours to perform this traffic capture and we're currently waiting for when this will happen.

Update
2 Mar 11:14:48
Packet captures on an affected circuit carried out by TalkTalk have confirmed that this issue most likely lies in the Openreach network. Circuits that we have been made aware of are being pursued with both BT and TalkTalk for Openreach to make further investigations into the issue.
If you believe you may be affected please do contact support.
Update
17 Mar 09:44:00
Having had TalkTalk capture the traffic in the exchange, the next step is to capture traffic at the road-side cabinet. This is being progresses with Openreach and we hope this to happen 'soon'.
Update
29 Mar 09:52:52
We've received an update from BT advising that they have been able to replicate the missing IPv6 packets, this is believed to be a bug which they are pursuing with the vendor.

In the mean time they have also identified a fix which they are working to deploy. We're currently awaiting further details regarding this, and will update this post once further details become known.
Update
18 May 16:30:59
We've been informed that the fix for this issue is currently being tested with Openreach's supplier, but should be released to them on the 25th May. Once released to Openreach, they will then perform internal testing of this before deploying it to their network. We haven't been provided with any estimation of dates for the final deployment of this fix yet.
In the interim, we've had all known affected circuits on TalkTalk backhaul have a linecard swap at the cabinet performed as a workaround, which has restored IPv6 on all TT circuits known to be affected by this issue.
BT have come back to us suggesting that they too have a workaround, so we have requested that it is implemented on all known affected BT circuits to restore IPv6 to the customers known to have this issue on BT backhaul.
Resolution A fix was rolled out on the last week of June, re-testing with impacted customers has showed that IPv6 is functioning correctly on their lines again after Openreach have applied this fix.
Broadband Users Affected 0.05%
Started 7 Feb 09:00:00 by AA Staff
Closed 19 Jul

14 Jul 17:00:47
Details
14 Jul 13:16:20
The issue affecting broadand services has also been affecting hosted servers and our own services in Maidenhead. This has also has some issues with Ethernet customers.
Update
14 Jul 14:43:15
Updates are being posted in https://aastatus.net/2411
Update
14 Jul 17:00:53
The network has been stable for a good few hours now. One of our 40G Telehouse-to-Harbour Exchange interconnects has been taken down and some devices have been moved off of the suspect switch. We have further work to do in investigating the root cause of this and what we plan to do try to stop this from happening again. We do apologise to our customers for the disruption these two outages have caused and we work on trying to prevent this from happening again.
Started 14 Jul 10:00:00
Closed 14 Jul 17:00:47

14 Jul 16:48:39
Details
14 Jul 10:33:01
We've just seen BT and TT lines drop. We are investigating.
Update
14 Jul 10:57:12
This is similar to yesterday's problem. We have lost BGP connectivity to both our carriers. Over the past fee minutes the BGP sessions have been going up and down, meaning customers are logging in and then out again. Updates to follow shortly.
Update
14 Jul 11:16:46
Sessions are looking a bit more stable now... customer lines are still reconnecting
Update
14 Jul 11:56:08
We have about half DSL lines logged in, but the remaining half are struggling due to what is looking like a Layer 2 issue on our network.
Update
14 Jul 12:23:19
More DSL lines are now back up. If customers are still off, a reboot may help.
Update
14 Jul 12:35:51
A number of TT and BT lines just dropped. They are starting to reconnect now though.
Update
14 Jul 12:52:31
This is still on going - most DSL lines are back, some have been dropping but the network is still unstable. We are continuing to investigate this.
Update
14 Jul 12:55:50
We managed to get a lot of lines up after around an hour, during which there was a lot of flapping. The majority of the rest (some of the talk talk back haul lines) came up, and flapped a bit, at 12:00 and came back properly around 12:40. However, we are still trying to understand the issue, and still have some problems, and we suspect there may be more outages. The problem appears to be something at layer 2 but impacting several sites at once.
Update
14 Jul 14:41:23
We believe these outages are due to a partial failure of one of our core switches. We've moved most services away from this switch and are running diagnostics on it at the moment. We are not expecting these diagnostics to affect other services.
Update
14 Jul 17:00:40
The network has been stable for a good few hours now. One of our 40G Telehouse-to-Harbour Exchange interconnects has been taken down and some devices have been moved off of the suspect switch. We have further work to do in investigating the root cause of this and what we plan to do try to stop this from happening again. We do apologise to our customers for the disruption these two outages have caused and we work on trying to prevent this from happening again.
Started 14 Jul 10:30:00
Closed 14 Jul 16:48:39

13 Jul 21:38:01
Details
13 Jul 18:56:13
Multiple circuits have disconnected and reconnected, staff are investigating
Update
13 Jul 19:00:34
Sessions seem to be repeatedly flapping rather than reconnecting - staff are investigating.
Update
13 Jul 20:05:47
We are still working on this, it's a rather nasty outage I'm afraid and is proving difficult to track down.
Update
13 Jul 20:17:42
Lines are re-connecting now...
Update
13 Jul 20:19:16
Apologies for the loss of graphs on a.gormless. We usually assume the worst and that out kit is the cause and tried a reboot of a FireBrick LNS. It did not help, but did clarify the real cause which was cisco switches. Sorry for the time it took to track this one down.
Update
13 Jul 20:19:27
Some line are being forced to reconnect so as to move them to the correct LNS, this will cause a logout/login for some customers...
Update
13 Jul 20:25:52
Lines are still connecting, not all are back, but the number of connected lines is increasing.
Update
13 Jul 20:29:55
We're doing some work which may cause some lines to go offline - we expect line to start reconnecting in 10 minutes time.
Update
13 Jul 20:34:35
We are rebooting stuff to try and find the issue. This is very unusual.
Update
13 Jul 20:42:56
Things are still not stable and lines are still dropping. We're needing to reboot some core network switches as part of our investigations and this is happening at the moment.
Update
13 Jul 20:52:29
Lines are reconnecting once more
Update
13 Jul 21:17:36
Looking stable.
Update
13 Jul 21:23:48
Most lines are back online now, if customers re still not online they a reboot of the router or modem may be required as the session may have got stuck inside the back haul network.
Update
13 Jul 21:41:38

We'll close this incident as lines have been stable for an hour. We'll update the post with further information as to the cause and any action we will be taking to help stop this type of incident from happening again.

We would like to thank our customers for their patience and support this evening. We had many customers in our IRC channel who were in good spirits and supportive to our staff whilst they worked on this incident.

Update
14 Jul 14:42:20
A similar problem occurred on Friday morning, this is covered on the following post: https://aastatus.net/2411
Closed 13 Jul 21:38:01

13 Jul 16:37:48
Details
13 Jul 16:37:48

Next week we will be making a change to our incoming email servers which is aimed at reducing the amount of spam email from reaching customer's mailboxes.

Historically, we've been purposefully cautious of rejecting email outright and prefer the method of marking messages as spam based on a 'spam score'. Customers have options as to what score to mark and to reject messages. However, due to the high number of spam messages, the cost of scanning each message and the extremely low risk of false positives we're going to introduce rejecting messages from IP addresses that are known spammers.

Specifically, the change is to reject messages from email servers that are listed in the "Spamhaus" block lists. These lists contain IP addresses that are known to be spam senders or are compromised machines in some way. Spamhaus have "a long-held reputation as having a false positive rate so low as to be unmeasurable and insignificant".

Many mail servers around the world use these same block lists, but if you are in anyway concerned about this then please to get in touch with us.
Update
19 Jul 16:15:43
We are making these changes at the moment. (Wednesday afternoon). As described above, we're not expecting this to impact customers in a negative way.
Started 13 Jul 16:30:00

10 Jul 22:57:37
Details
10 Jul 22:52:30

OVERVIEW:

Using encryption is strongly recommended wherever possible to as to ensure private communication between you and the server and to protect your username and passwords from being intercepted by other people.

Historically, accessing our email servers using SSL/TLS has been available but problematic in that the certificates we use is issued by 'CA CERT' which although is 'secure', email programs often popped up a warning about the security settings.

SOLUTION:

We have added a new set of servers which customers can use to access their email. These use a Let's Encrypt certificate which will be accepted by email clients without any warnings.

IN TESTING:

At the moment (July 2017) we are still testing this out, and this should be viewed as 'beta'. There are a few rough edges which we are still working on fixing before making this an official service. that said, customers may try this out and we will make any maintenance as seamless as possible - meaning that it should work fine but there may be short periods of time when it stops working.

USAGE:

A simple change needs to be made to your email account settings within your email program (eg account settings in Thunderbird, Outlook, Mail etc). Go to the account settings where the IMAP or POP3 server details are, and ensure the following is set:

For IMAP:

Server hostname: mail.aa.net.uk

Port: 993

Security options: SSL/TLS

For POP3:

Server hostname: mail.aa.net.uk

Port: 995

Security options: SSL/TLS

You must use SSL/TLS as we do not support insecure access on these servers.

CHANGING BACK:

Customers are free to change the settings back to what they had before. This will not affect the emails stored on the server in any way.
Started 10 Jul 22:00:00

10 Jul 22:54:46
[DNS, Email and Web Hosting] - SSL Certificates Updating - Info
Details
10 Jul 18:12:06

We're updating SSL certificates for our email servers today. The old serial number is 12AD7B. The new serial number is 130CAB. Users who don't have the CAcert root certificate installed may see errors. This does not affect webmail or outgoing SMTP. Details on http://aa.net.uk/cacert.html

We have a new email proxy that should fix these problems; those affected by this can try setting their incoming mail server to mail.aa.net.uk (TLS only, no STARTTLS). Please note that this is not yet "launched" and is therefore not yet officially supported. More info here: https://aastatus.net/2407

Started 10 Jul 15:00:00

10 Jul 02:21:59
[Broadband] - BT blip - Closed
Details
10 Jul 02:16:23
Looks like all lines on BT backhaul blipped at just before 2am. Lines reconnected right away though. Some lines on wrong LNS now so we may move them back - which with show a longer gap in the graphs.
Update
10 Jul 02:22:25
Sessions are all back, and on the right LNS again.
Started 10 Jul 01:59:03
Closed 10 Jul 02:21:59
Previously expected 10 Jul 02:30:00

7 Jul 10:39:42
Details
7 Jul 10:39:42

For the past few years we've been supplying the ZyXEL ZyXEL VMG1312-B10A router. This is being discontinued and we will start supplying its replacement, the ZyXEL VMG1312-B10D (note the subtle difference!)

The new router is smaller than the previous one and has a very similar feature set and web interface to the old one.

We are still working through our configuration process and are updating the Support site with documentation. We are hoping this model to resolve many of the niggles we have with the old one too.

Started 7 Jul 13:12:00

2 Jul 13:00:00
Details
2 Jul 11:32:17
It seems some emailed invoices and DD notices have failed to be delivered. We plan to re-send these, so some people may get duplicate emails. Sorry for any inconvenience caused.
Update
2 Jul 18:08:43
Invoices we re-emailed and Direct Debit notices re-issued. We think we managed with no duplicates.
Started 1 Jul 12:36:26
Closed 2 Jul 13:00:00
Previously expected 1 Jul 17:00:00

3 Jun 17:28:30
Details
3 Jun 17:06:27
Something definitely not looking right, seems to be intermittent and impacting Internet access.
Update
3 Jun 17:10:34
Looks like a denial of service attack of some sort.
Update
3 Jun 17:17:45
Looks like may be more widespread than just us.
Update
3 Jun 17:23:17
Definitely a denial of service attack, impacted some routers and one of the LNSs. Some graphs lost.
Resolution Target isolated for now.
Started 3 Jun 16:59:05
Closed 3 Jun 17:28:30

2 Jun 14:38:28
Details
30 May 15:11:00
We have been advised of essential scheduled carrier maintenance to upgrade core infrastructure. The windows are as follows; Window 1 Service Impact Start Time: 2/6/2017 02:00 Service Impact End Time: 2/6/2017 04:30 Impact Time Expected: 30 minutes Throughout the duration of this window, subscribers may see disruption of up to 30 minutes to their 3G/4G data services in the following areas: Kensington, Hampstead, Paddington, Hammersmith, Westminster, Battersea Window 2 Service Impact Start Time: 8/6/2017 01:00 Service Impact End Time: 8/6/2017 03:30 Impact Time Expected: 2 hours, 30 minutes Throughout the duration of this window, subscribers may see disruption of up to 30 minutes to their 3G/4G data services in the following areas: Camden, West Berkshire, City of London, Islington, Lambeth, Southwark, Westminster If you have any questions regarding this maintenance, please don't hesitate to contact our support team on 03333 400 999.
Started 2 Jun 02:00:00
Closed 2 Jun 14:38:28
Previously expected 8 Jun 03:30:00 (Last Estimated Resolution Time from Carrier)

8 Jun 10:49:27
Details
7 Jun 10:33:00
We are seeing some customers who are still down following a blip within TalkTalk. We currently have no root cause but are investigating.
Update
7 Jun 11:13:21
A small number of lines are still down, however most have now resumed service. We are still communicating with TalkTalk so we can restore service for all affected lines.
Update
7 Jun 11:23:02
Looks like we're seeing another blip affecting many more customers this time. We are still speaking to TalkTalk to determine the cause of this.
Update
7 Jun 11:59:53

TalkTalk have raised an incident with teh following information:

"We have received reports from a number of B2B customers (Wholesale ADSL) who are experiencing a loss of their Broadband services. The impact is believed to approximately 600 lines across 4 or 5 partners. All of the impacted customers would appear to route via Harbour Exchange. Our NOC have completed initial investigations and have passed this to our IP operations team to progress. "

As a result, we'll move TalkTalk traffic away from the Harbour Exchange datacentre to see if it helps. This move will be seamless and will not affect other customers.

Update
7 Jun 12:05:38
Our TalkTalk traffic has now been moved away from HEX89, if There are still a small number of customers offline, if they reboot their router/modem that may force a re-connection and a successful login.
Update
7 Jun 12:37:29
At 12:29 we saw around 80 lines drop, most of these are back online as of 12:37 though. The incident is still open with TalkTalk engineers.
Update
7 Jun 13:19:57
TalkTalk are really not having a good day. We're now seeing packetloss on lines as well as a few more drops. We're going to bring the HEX89 interconnect back up in case that is in any way related, we're also chasing TT on this.
Update
7 Jun 14:37:21
This is still an open incident with TalkTalk, it is affecting other ISPs using TalkTalk as their backhaul. We have chased TalkTalk for an update.
Update
7 Jun 15:37:21

Update from TalkTalk: "Network support have advised that service has been partially restored. Currently Network Support are continuing to re-balance traffic between both LTS’s (HEX & THN). This work is currently being completed manually by our Network support team who ideally need access to RadTools to enable them to balance traffic more efficiently. We are currently however experiencing an outage of RadTools which is being managed under incident 10007687. We will continue to provide updates on the progress as soon as available."

Probably as a result, we are still seeing low levels of packetloss on some TalkTalk lines.

Update
7 Jun 16:49:12
It's looking like the low levels of packetloss stopped at 16:10. Things are looking better.
Update
8 Jun 08:31:43
There are a handful of customers that are still offline, we have sent the list of these circuits to TalkTalk to investigate.
Update
8 Jun 10:26:02

Update from TalkTalk: "We have received reports from a number of B2B customers (Wholesale ADSL & FTTC) who are experiencing authentication issues with their Broadband services. The impact is believed to approximately 100 lines across 2 partners. All of the impacted customers would appear to route via Harbour Exchange. Our NOC have completed initial investigations and have passed this to our Network support team to progress."

We have actaully already taken down our Harbour Exchange interconnect but this has not helped.

Update
8 Jun 10:49:27
Over half of these remaining affected lines logged back in at 2017-06-08 10:38
Update
8 Jun 11:22:39
The remaining customers offline should try rebooting their router/modem and if still not online then please contact Support.
Resolution

From TalkTalk: The root cause of this issue is believed to have been caused by a service request which involved 3 network cards being installed in associated equipment at Harbour exchange. This caused BGP issues on card (10/1). To resolve this Network Support shut down card (10/1) but this did not resolve all issues. This was then raised this to Ericsson who recommended carrying out an XCRP switchover on the LTS. Once the switchover was carried out all subscribers connections dropped on the LTS and the majority switched over to the TeleHouse North LTS. Network support then attempted to rebalance the traffic across both LTS platform however were not able to due to an ongoing system incident impacting Radius Tools. Network support instead added 2 new 10G circuits to the LTS platform to relieve the congestion and resolve any impact. As no further issues have been identified this incident will now be closed and any further RCA investigation will be carried out by problem management.

Regarding the problem with a few circuits not able to establish PPP. the report from TalkTalk is as follows: Network support have advised that they have removed HEX (harbour exchange) from the radius to restore service until a permanent fix can be identified. Network support are liaising with Ericsson in regards to this and investigations are ongoing.

Broadband Users Affected 0.20%
Started 7 Jun 10:05:00
Closed 8 Jun 10:49:27

27 Mar 09:30:00
Details
19 Feb 18:35:15
We have seen some cases with degraded performance on some TT lines, and we are investigating. Not a lot to go on yet, but be assured we are working on this and engaging the engineers within TT to address this.
Update
21 Feb 10:13:20

We have completed further tests and we are seeing congestion manifesting itself as slow throughput at peak times (evenings and weekends) on VDSL (FTTC) lines that connect to us through a certain Talk Talk LAC.

This has been reported to senior TalkTalk staff.

To explain further; VDSL circuits are routed from TalkTalk to us via two LACs. We are seeing slow thoughput at peak times on one LAC and not the other.

Update
27 Feb 11:08:58
Very often with congestion it is easy to find the network port or system that is overloaded but so far, sadly, we've not found the cause. A&A staff and customers and TalkTalk network engineers have done a lot of checks and tests on various bits of the backhaul network but we are finding it difficult to locate the cause of the slow throughput. We are all still working on this and will update again tomorrow.
Update
27 Feb 13:31:39
We've been in discussions with other TalkTalk wholesalers who have also reported the same problem to TalkTalk. There does seem to be more of a general problem within the TalkTalk network.
Update
27 Feb 13:32:12
We have had an update from TalkTalk saying that based on multiple reports from ISPs that they are investigating further.
Update
27 Feb 23:21:21
Further tests this evening by A&A staff shows that the throughput is not relating to a specific LAC, but that it looks like something in TalkTalk is limiting single TCP sessions to 7-9M max during peak times. Running single iperf tests results in 7-9M, but running ten at the same time can fill a 70M circuit. We've passed these findings on to TalkTalk.
Update
28 Feb 09:29:56
As expected the same iperf throughput tests are working fine this morning. TT are shaping at peak times. We are pursuing this with senior TalkTalk staff.
Update
28 Feb 11:27:45
TalkTalk are investigating. They have stated that circuits should not be rate limited and that they are not intentionally rate limiting. They are still investigating the cause.
Update
28 Feb 13:14:52
Update from TalkTalk: Investigations are currently underway with our NOC team who are liaising with Juniper to determine the root cause of this incident.
Update
1 Mar 16:38:54
TalkTalk are able to reproduce the throughput problem and investigations are still on going.
Update
2 Mar 16:51:12
Some customers did see better throughput on Wednesday evening, but not everyone. We've done some further testing with TalkTalk today and they continue to work on this.
Update
2 Mar 22:42:27
We've been in touch with the TalkTalk Network team this evening and have been performing further tests (see https://aastatus.net/2363 ). Investigations are still ongoing, but the work this evening has given a slight clue.
Update
3 Mar 14:24:48
During tests yesterday evening we saw slow throughput when using the Telehouse interconnect and fast (normal) throughput over Harbour Exchange interconnect. Therefore, this morning, we disabled our Telehouse North interconnect. We will carry on running tests over the weekend and we welcome customers to do the same. We are expecting throughput to but fast for everyone. We will then liaise with TalkTalk engineers regarding this on Monday.
Update
6 Mar 15:39:33

Tests over the weekend suggest that speeds are good when we only use our Harbour Exchange interconnect.

TalkTalk are moving the interconnect we have at Telehouse to a different port at their side so as to rule out a possible hardware fault.

Update
6 Mar 16:38:28
TalkTalk have moved our THN port and we will be re-testing this evening. This may cause some TalkTalk customers to experience slow (single thread) downloads this evening. See: https://aastatus.net/2364 for the planned work notice.
Update
6 Mar 21:39:55
The testing has been completed, and sadly we still see slow speeds when using the THN interconnect. We are now back to using the Harbour Exchange interconnect where we are seeing fast speeds as usual.
Update
8 Mar 12:30:25
Further testing happening today: Thursday evening https://aastatus.net/2366 This is to try and help narrow down where the problem is occurring.
Update
9 Mar 23:23:13
We've been testing, tis evening, this time with some more customers, so thank you to those who have been assisting. (We'd welcome more customers to be involved - you just need to run an iperf server on IPv4 or IPv6 and let one of our IPs through your firewall - contact Andrew if you're interested). We'll be passing the results on to TalkTalk, and the investigation continues.
Update
10 Mar 15:13:43
Last night we saw some line slow and some line fast, so having extra lines to test against should help in figuring out why this is the case. Quite a few customers have set up iperf server for us and we are now testing 20+ lines. (Still happy to add more). Speed tests are being run three times an hour and we'll collate the results after the weekend and will report back to TalkTalk the findings.
Update
11 Mar 20:10:21
Update
13 Mar 15:22:43

We now have samples of lines which are affected by the slow throughput and those that are not.

Since 9pm Sunday we are using the Harbour Exchange interconnect in to TalkTalk and so all customers should be seeing fast speeds.

This is still being investigated by us and TalkTalk staff. We may do some more testing in the evenings this week and we are continuing to run iperf tests against the customers who have contacted us.
Update
14 Mar 15:59:18

TalkTalk are doing some work this evening and will be reporting back to us tomorrow. We are also going to be carrying out some tests ourselves this evening too.

Our tests will require us to move traffic over to the Telehouse interconnect, which may mean some customers will see slow (single thread) download speeds at times. This will be between 9pm and 11pm

Update
14 Mar 16:45:49
This is from the weekend:

Update
17 Mar 10:42:28
We've stopped the iperf testing for the time being. We will start it back up again once we or TalkTalk have made changes that require testing to see if things are better or not, but at the moment there is no need for the testing as all customers should be seeing fast speeds due to the Telehouse interconnect not being in use. Customers who would like quota top-ups, please do email in.
Update
17 Mar 18:10:41
To help with the investigations, we're also asking for customers with BT connected FTTC/VDSL lines to run iperf so we can test against them too - details on https://support.aa.net.uk/TTiperf Thank you!
Update
20 Mar 12:54:02
Thanks to those who have set up iperf for us to test against. We ran some tests over the weekend whilst swapping back to the Telehouse interconnect, and tested BT and TT circuits for comparison. Results are that around half the TT lines slowed down but the BT circuits were unaffected.

TalkTalk are arranging some further tests to be done with us which will happen Monday or Tuesday evening this week.

Update
22 Mar 09:37:30
We have scheduled testing of our Telehouse interlink with TalkTalk staff for this Thursday evening. This will not affect customers in any way.
Update
22 Mar 09:44:09
In addition to the interconnect testing on Thursday mentioned above, TalkTalk have also asked us to retest DSL circuits to see if they are still slow. We will perform these tests this tonnight, Wednesday evening.

TT have confirmed that they have made a configuration change on the switch at their end in Telehouse - this is the reason for the speed testing this evening.

Update
22 Mar 12:06:50
We'll be running iperf3 tests against our TT and BT volunteers this evening, very 15 minutes from 4pm through to midnight.
Update
22 Mar 17:40:20
We'll be changing over to the Telehouse interconnect between 8pm and 9pm this evening for testing.
Update
23 Mar 10:36:06

Here are the results from last night:

And BT Circuits:

Some of the results are rather up and down, but these lines are in use by customers so we would expect some fluctuations, but it's clear that a number of lines are unaffected and a number are affected.

Here's the interesting part. Since this problem started we have rolled out some extra logging on to our LNSs, this has taken some time as we only update one a day. However, we are now logging the IP address used at our side of L2TP tunnels from TalkTalk. We have eight live LNSs and each one has 16 IP addresses that are used. With this logging we've identified that circuits connecting over tunnels on 'odd' IPs are fast, whilst those on tunnels on 'even' IPs are slow. This points to a LAG issue within TalkTalk, which is what we have suspected from the start but this data should hopefully help TalkTalk with their investigations.

Update
23 Mar 16:27:28
As mentioned above, we have scheduled testing of our Telehouse interlink with TalkTalk staff for this evening. This will not affect customers in any way.
Update
23 Mar 22:28:53

We have been testing the Telehouse interconnect this evening with TalkTalk engineers. This involved a ~80 minute conference call and setting up a very simple test of a server our side plugged in to the switch which is connected to our 10G interconnect, and running iperf3 tests against a laptop on the TalkTalk side.

The test has highlighted a problem at the TalkTalk end with the connection between two of their switches. When plugged in to the second switch we got about 300Mbit/s, but when their laptop was in the switch directly connected to our interconnect we got near full speed or around 900Mb/s.

This has hopefully given them a big clue and they will now involve the switch vendor for further investigations.

Update
23 Mar 23:02:34
TalkTalk have just called us back and have asked us to retest speeds on broadband circuits. We're moving traffic over to the Telehouse interconnect and will test....
Update
23 Mar 23:07:31
Initial reports show that speeds are back to normal! Hooray! We've asked TalkTalk for more details and if this is a temporary or permanent fix.
Update
24 Mar 09:22:13

Results from last night when we changed over to test the Telehouse interlink:

This shows that unlike the previous times, when we changed over to use the Telehouse interconnect at 11PM speeds did not drop.

We will perform hourly iperf tests over the weekend to be sure that this has been fixed.

We're still awaiting details from TalkTalk as to what the fix was and if it is a temporary or permanent fix.

Update
24 Mar 16:40:24
We are running on the Telehouse interconnect and are running hourly iperf3 tests against a number of our customers over the weekend. This will tell us if the speed issues are fixed.
Update
27 Mar 09:37:12

Speed tests against customers over the weekend do not show the peak time slow downs, this confrims that what TalkTalk did on Thursday night has fixed the problem. We are still awaiting the report from TalkTalk regarding this incident.

The graph above shows iperf3 speed test results taken once an hour over the weekend against nearly 30 customers. Although some are a bit spiky we are no longer seeing the drastic reduction in speeds at peak time. The spikyness is due to the lines being used as normal by the customers and so is expected.

Update
28 Mar 10:52:25
We're expecting the report from TalkTalk at the end of this week or early next week (w/b 2017-04-03).
Update
10 Apr 16:43:03
We've not yet had the report from TalkTalk, but we do expect it soon...
Update
4 May 09:16:33
We've had an update saying: "The trigger & root cause of this problem is still un-explained; however investigations are continuing between our IP Operation engineers and vendor".

This testing is planned for 16th May.

Resolution From TT: Planned work took place on the 16th May which appears to have been a success. IP Ops engineers swapped the FPC 5 and a 10 gig module on the ldn-vc1.thn device They also performed a full reload to the entire virtual chassis (as planned). This appears to have resolved the slow speed issues seen by the iperf testing onsite. Prior to this IP ops were seeing consistent slow speeds with egress traffic sourced from FPC5 to any other FPC; therefore they are confident that this has now been fixed. IP Ops have moved A&A's port back to FPC 5 on LDN-vc1.thn.
Started 18 Feb
Closed 27 Mar 09:30:00
Cause TT

17 May 12:00:00
Details
26 Apr 11:01:07
We have noticed packetloss between 8pm and 10pm on Tuesday (25th April) evening on a small number of TalkTalk connected lines. This may be related to TalkTalk maintenance. We will review this again tomorrow.
Update
26 Apr 16:41:43
We are seeing packet loss this afternoon on some of these lines too. We are contacting TalkTalk.
Update
26 Apr 16:44:23
Update
26 Apr 17:58:06
We have moved TalkTalk traffic over to our Harbour Exchange interconnect to see if this makes a difference or not to the packet loss that we are seeing...
Update
26 Apr 20:50:41
Moving the traffic made no difference. We've had calls with TalkTalk and they have opened an incident and are investigating further.
Update
26 Apr 20:55:14
The pattern that we are seeing relates to which LAC TT are using to send traffic over to us. TT use two LACs at their end, and lines via one have loss whilst lines via the other have no loss.
Update
26 Apr 21:32:30
Another example, showing the loss this evening:

Update
26 Apr 22:26:50
TalkTalk have updated us with: "An issue has been reported that some Partners are currently experiencing quality of service issues, such as slow speed and package (SIC) loss, with their Broadband service. From initial investigations the NOC engineers have identified congestion to the core network connecting to Telehouse North as being the possible cause. This is impacting Partners across the network and not specific to one region, and the impacted volume cannot be determine at present. Preliminary investigations are underway with our NOC and Network Support engineers to determine the root cause of this network incident. At this stage we are unable to issue an ERT until the engineers have completed further diagnostics."
Update
28 Apr 09:43:43
Despite Talktalk thinking they had fixed this we are still seeing packetloss on these circuits between 8pm and 10pm. It's not as much packetloss as we saw on Wednesday evening, but loss nonetheless. This has been reported back to TalkTalk.
Update
4 May 15:42:04
We are now blocking the two affected Talkalk LACs on new connections, eg a PPP re-connect. This means that it will take a bit longer for a line to re-connect (depending upon the broadband router perhaps a minute or two).

This does mean that lines will not be on the LACs which have evening packetloss. We hope not to have to keep this blocking in place for very long as we hope TalkTalk fix this soon.

Update
5 May 16:48:13
We've decided to not block the LACs that are showing packetloss as it was causing connection problems for a few customers. We have had a telephone call with TalkTalk today, and this issue is being escalated with TalkTalk.
Update
5 May 19:47:37
We've had this update from TalkTalk today:

"I have had confirmation from our NOC that following further investigations by our IP Operations team an potential issue has been identified on our LTS (processes running higher than normal). After working with our vendor it has been recommended a card switch-over should resolve this.

This has been scheduled for 16th May. We will post further details next week.

Update
8 May 09:26:21
Planned work has been scheduled for 16th May for this, details and updates of this work is on https://aastatus.net/2386
Update
5 Jun 11:49:24
The planned work took place on the 16th May which appears to have been a success.
Broadband Users Affected 20%
Started 26 Apr 10:00:00
Closed 17 May 12:00:00

4 Jun 15:26:18
Details
31 May 17:42:21
We are upgrading our accounts system hardware this Sunday, the 4th of June. The accounts system and ordering will be unavailable for a period of time. We have a 4 hour window within which we expect to have completed the work, and do not expect the work to take the whole 4 hours. We have a set of prerequisite tests which we need to complete beforehand, and we will back out of the work and reschedule it if needed.
Update
4 Jun 10:00:46
Work is starting, the web pages for accounts system, and our ordering system will be off line for a while.
Update
4 Jun 13:30:54
The accounts pages are accessible again, but ordering is still off line.
Update
4 Jun 14:03:00
Apologies for the overrun - ordering still not up.
Update
4 Jun 14:38:16
This is still on-going, stuck on one silly bit of config not playing. Sorry for the delay.
Resolution The ordering systems are now working. We will continue to monitor things during the day.
Started 4 Jun 10:00:00
Closed 4 Jun 15:26:18
Previously expected 4 Jun 14:00:00

2 Jun 16:44:00
Details
2 Jun 12:48:45
We have had several customers notify us that they've having connectivity issues in the Cambridge area, all FTTC customers so far, where TCP packets larger than 1300 bytes appear to be dropped. ICMP appears unaffected.
We are currently in the process of reporting this to BT and will post further updates as they become available.
Update
2 Jun 13:08:24
Proactive have raised a fault, AF-AR-OP-3774655 This has also been discovered to affect FTTP customers, which makes sense as they use the same backbone infrastructure as the FTTC customers.
Update
2 Jun 13:35:22
Several customers are reporting that their lines are now performing as expected after restarting their PPP session, so it may be worth restarting your PPP session and letting us know what happens when you try that.
We're still awaiting an update from proactive.
Resolution We have been advised by BT that a link between Milton Keynes and Peterborough was erroring, and was taken out of service to resolve the issue earlier today.
Broadband Users Affected 0.20%
Started 2 Jun 11:47:00
Closed 2 Jun 16:44:00
Cause BT

2 Jun 12:15:10
[DNS, Email and Web Hosting] - DNS issues - Closed
Details
2 Jun 11:55:34
We are currently experiencing an issue with an authoritative nameserver which is resulting in customers being unable to resolve webpages. We are currently investigating and hope to have a fix very shortly.
Update
2 Jun 12:15:35
The authoritative DNS server has now been fixed.
Started 2 Jun 11:30:00
Closed 2 Jun 12:15:10
Cause AAISP

1 Jun 14:39:11
Details
1 Jun 14:44:53

This month customers may have seen unfamiliar attachments on their invoices.

One is an HTML attachment which is a form, and generally somewhat useless. You can ignore this.

The other is a signature.pgp, which is the separate PGP signature as per PGP/MIME email formatting. If you have PGP software on your email system this can be used to check the validity of the email.

The reason for this appears to be down to an issue with defaults after email changes recently. We have now changed all accounts not to send HTML or the separate PGP attachment.

If you would now like emails in PGP/MIME format, please do edit the settings on the accounts page to re-enable this. Sorry for any inconvenience.

Please note that in all cases the email included the plain text of the invoice, and this is the formal tax invoice. Being unable to open an attachment does not change this, and the invoice is still to be paid. Sorry if this sounds obvious or condescending, you would be surprised at the comments we sometimes get.

Started 1 Jun
Closed 1 Jun 14:39:11

31 May 17:40:36
Details
17 May 11:22:27
We have reduced numbers of staff in the office today due some being taken on a tour of a BT exchange & metro node.
Resolution All affected staff members are now back in service, and proven to be operational.
Started 17 May 09:00:00
Cause BT

17 May 12:30:00
Details
17 May 09:10:40
We have identified outages within the North London area affecting multiple exchanges. The affected exchanges are listed as: SHOEBURYNESS,THORPE BAY, CANVEY ISLAND, HADLEIGH – ESSEX, MARINE, NORTH BENFLEET, STANFORD LE HOPE, VANGE, WICKFORD, BLOOMSBURY AKA HOWLAND ST, HOLBORN, PRIMROSE HILL, KINGSLAND GREEN, TOTTENHAM, BOWES PARK, PALMERS GREEN, WALTHAM CROSS, WINCHMORE HILL, EDMONTON, NEW SOUTHGATE, EPPING, HAINAULT, ILFORD NORTH, ROMFORD, UPMINSTER, NORTH WEALD, STAMFORD HILL, DAGENHAM, ILFORD CENTRAL, GOODMAYES, STRATFORD, HIGHAMS PARK, LEYTONSTONE, WALTHAMSTOW, CHINGFORD, KENTISH TOWN and MUSWELL HILL. No root cause has been identified. We will update this status page as the updates become available from our supplier.
Update
17 May 09:30:01
Affected Customers went offline at 02:12 this morning. Further information: These exchanges are off line due to 20 tubes of fibre being damaged by the install of a retail advertising board on the A118 Southern [South?] Kings Rd. Notifications from TalkTalk were expecting service to be restored by 9am, but due to the nature of the fibre break it may well take longer to fix.
Update
17 May 10:31:29
Update from TalkTalk: Virgin media have advised that work to restore service is ongoing but due to the extent of the damage this is taking longer than expected. In parallel our [TalkTalk] NOC are investigating if the Total Loss traffic can be re-routed. We will provide a further update as soon as more information is available.
Update
17 May 10:55:56
From TalkTalk: Virgin Media have advised restoration work has completed on the majority of the damaged fibre, our NOC team have also confirmed a number of exchanges are now up and back in service. The exchanges that are now back in service are Muswell Hill, Ingrebourne, Loughton and Bowes Park.
Update
17 May 11:16:28
From TalkTalk: Our NOC team have advised a number of exchanges are now in service. These are Muswell Hill, Bowes Park, Loughton, Ingrebourne, Bowes Park, Chingford, Highams Park, Leytonstone, Stratford and Upton Park.
Update
17 May 11:30:54
That said, we are still seeing lines on exchanges mentioned above as being offline....
Update
17 May 12:11:20
No further updates as yet.
Resolution It looks like most, if not all, of our affected lines are now back online. Update from TalkTalk: Virgin Media have advised 5 of the 8 impacted fibre tubes have been successfully spliced and their engineers are still on site restoring service to the remaining cables
Started 17 May 01:54:00 by AA Staff
Closed 17 May 12:30:00

13 May
Details
12 May 18:02:27
The upgrades this week worked well, but the system for IPv4/IPv6 fallback was not quite right when it came to some delayed and fallback calls which are called sooner than they should be when targets are not in the DNS cache. This was obviously not that frequent, but we believe we have addressed it and will be upgrading this evening.
Started 12 May
Closed 13 May

11 May
Details
8 May 15:14:16
We don't usually post about these as there are normally done out of hours when no calls in progress and so cause no down time. However, recently, there was a small issue with some calls not getting through after an upgrade. We are planning an upgrade of one of the VoIP servers this evening, and we will be monitoring for any issues. If you have problems with calls please let us know ASAP. If all goes well we'll do the other server later in the week. The upgrade improves the IPv4/IPv6 fallback handling where customers have both IPv4 and IPv6 endpoints.
Update
10 May 08:24:28
No issues reported so we plan to upgrade the second call server today.
Started 8 May
Closed 11 May
Previously expected 9 May

4 May 18:00:00
[Broadband] - TT Blips - Closed
Details
4 May 16:53:39
Some TT lines blipped at 16:34 and 16:46. It appears that the lines have recovered. We have reported this to TT.
Update
5 May 08:48:41
This was caused by an incident in the TalkTalk network. This is from TalkTalk: "...Network support have advised that the problem was caused by the card failure which has now been taken offline..."
Started 4 May 16:36:27 by AAISP automated checking
Closed 4 May 18:00:00
Cause TT

2 May 11:50:00
Details
2 May 11:13:13
We're investigating a problem with incoming calls not working where they are configured to redirect (also-ring) to other number(s).
Update
2 May 11:22:49
This looks to be affecting one of our call servers and not the other, so this explains why calls are working some of the time. Investigations continue.
Update
2 May 11:46:20
We are moving calls off of the affected server, and we are starting to see calls working again.
Update
2 May 11:51:11
Customers with problem with calls may need to get their VoIP phones to re-register - the easiest way is often vie the web interface of the VoIP phone, or by unplugging the power and plugging it back in again!
Started 2 May 10:45:00
Closed 2 May 11:50:00