Today 08:56:09
Details
Yesterday 18:35:15
We have seen some cases with degraded performance on some TT lines, and we are investigating. Not a lot to go on yet, but be assured we are working on this and engaging the engineers within TT to address this.
Started Saturday
Update was expected Today 13:00:00

16 Feb 15:00:00
Details
16 Feb 16:00:49
We have spotted some odd latency that was affecting two of our LNSs (A and B gormless). These were also visible, as you would expect, on the graphs shown for people's lines.
Resolution We believe we have addressed the issue now, sorry for any inconvenience.
Started 15 Feb 02:00:00
Closed 16 Feb 15:00:00
Previously expected 16 Feb 15:00:00

24 Feb 14:57:44
Details
15 Feb 15:08:19
This applies to Aberdeenshire, Fife, Dundee, Inverness and surrounding areas Data SIMs in these areas are likely to see a couple of hours of disruption between 1:00 and 3:30 on the 23rd and 24th of February due to carrier hardware maintenance.
Closed 24 Feb 14:57:44

13 Feb 10:02:12
[Broadband] - LNS blip - Closed
Details
13 Feb 10:00:36
We just had an LNS blip - this would have caused some customers to drop PPP and reconnect.
Resolution There have been a few LNS blips recently. However, we do know the cause and have a software update to roll out which will fix the problem.
Started 13 Feb 09:56:00
Closed 13 Feb 10:02:12

9 Feb 13:00:00
Details
9 Feb 12:54:39
There appears to be a power outage in maidenhead. This is affecting all our VOIP and Email services. More information to come.
Update
9 Feb 12:59:03
Power is back. VOIP and Email services have been restored. Postmortem to come.
Resolution Power was restored within a few minutes - we do apologise for this unexpected power outage - the cause is suspected to be a faulty network switch which tripped the circuit breaker when it was lightly touched by one of our staff in the datacentre! The switch has been powered off and will be removed from the datacentre out of hours.
Started 9 Feb 12:52:45
Closed 9 Feb 13:00:00

8 Feb
Details
8 Feb 15:30:07
We are expecting more router upgrades later in the month - this should address a couple of issues we have seen the last few weeks. These should have little or no disruption. Routers are normally done in evening or over night. LNSs are done as a rolling change moving lines to a new LNS over night for a series of nights. Do note that you can configure a preferred time of night on our control pages. Exact dates for upgrades not determined yet but should all be done by end of Feb.
Update
14 Feb 17:53:22
We will be doing some of the routers this evening, and starting the rolling LNS upgrades tonight.
Update
15 Feb 17:03:39
We will be doing the rest of the routers this evening, and continuing the LNS roll over which will run for several more days (we have quite a few LNSs now).
Started 8 Feb
Expected close 1 Mar

7 Feb 14:49:55
[DNS, Email and Web Hosting] - SMTP Settings Change - Open
Details
7 Feb 14:52:42
For historical reasons, our SMTP servers allow sending authenticated email without TLS. This is insecure, and doesn't belong on the modern internet as it is possible for the username and password to be intercepted by a third party. We will no longer allow this as of the 4th of July. We are emailing customers who seem to be using insecure settings to warn them about the change. We have a support site page about what settings to change here: https://support.aa.net.uk/Enable_TLS_on_smtp.aa.net.uk Please contact support if you have any questions.
Started 7 Feb 14:49:55
Expected close 4 Jul 09:00:00

7 Feb 14:32:32
Details
7 Feb 14:32:32

We are seeing issues with IPv6 on a few VDSL cabinets serving our customers. There is no apparent geographical commonality amongst these, as far as we can tell.

Lines pass IPv4 fine, but only intermittently passing IPv6 TCP/UDP for brief amounts of time, usually 4 or so packets, before breaking. Customers have tried BT modem, Asus modem, and our supplied ZyXEL as a modem and router, no difference on any. We also lent them a FireBrick to do some traffic dumps.

Traffic captures at our end and the customer end show that the IPv6 TCP and UDP packets are leaving us but not reaching the customer. ICMP (eg pings) do work.

The first case was reported to us in August 2016, and it has taken a while to get to this point. Until very recently there was only a single reported case. Now that we have four cases we have a bit more information and are able to look at commonalities between them.

Of these circuits, two are serving customers via TalkTalk and two are serving customers via BT backhaul. So this isn't a "carrier network issue", as far as we can make out. The only thing that we can find that is common is that the cabinets are all ECI. (Actually - one of the BT connected customers has migrated to TalkTalk backhaul (still with us, using the same cabinet and phone line etc) and the IPv6 bug has also moved to the new circuit via TalkTalk as the backhaul provider)

We are working with senior TalkTalk engineers to try to perform a traffic capture at the exchange - at the point the traffic leaves TalkTalk equipment and is passed on to Openreach - this will show if the packets are making it that far and will help in pinning down the point at which packets are being lost. Understandably this requires TalkTalk engineers working out of hours to perform this traffic capture and we're currently waiting for when this will happen.

Broadband Users Affected 0.05%
Started 7 Feb 09:00:00 by AA Staff
Update was expected 9 Feb 17:00:00

4 Feb 09:32:03
[Broadband] - LNS blip - Closed
Details
4 Feb 09:14:11
We had an LNS reset and lines will have re-connected for some customers. We're investigating the cause.
Resolution We have found the cause, and expect a permanent fix to be deployed on next round of LNS upgrades.
Broadband Users Affected 12%
Started 4 Feb 09:12:00
Closed 4 Feb 09:32:03

2 Feb 21:19:15
Details
2 Feb 21:19:15
http://www.euronews.com/2017/01/27/adrian-kennard-challenging-surveillance

28 Jan 10:43:34
Details
3 Jan 09:36:03
We are doing some general router upgrades. As usually these should cause little or no disruption, and we will be doing LNS upgrades as a rolling upgrade one per night. We are also going to be bringing two more LNSs on-line to increase our capacity further.
Update
3 Jan 17:39:54
There will only be a few routers this evening, tomorrow we will look to bring in the new LNSs.
Update
4 Jan 02:34:01
Further updates this morning mean we have now completed around half off our core router upgrades.
Update
4 Jan 13:50:30
It looks like we will start LNS rolling updates Friday night instead. Testing today has gone well though.
Update
4 Jan 18:42:46
Core routers all upgraded, only LNSs now.
Update
6 Jan 17:01:27
Rolling LNS updates will start tonight, once this is complete we will bring the two new LNSs on line.
Update
12 Jan 18:03:46
LNS roll over is complete, we have some further updates and will be bringing new LNSs on-line over the next few days.
Update
14 Jan 15:10:22
Two additional LNSs are on-line now. We expect to do another LNS roll over soon to spread the load evenly.
Update
16 Jan 15:36:05
We will be running a rolling LNS switch over starting tonight.
Update
17 Jan 08:56:51
LNS switch for a few customers on "H" LNS to "I" LNS did not work properly last night, this has been fixed, and so there may have been more than one PPP restart over night, and one just before 9am. Looks good now. Sorry for any inconvenience.
Resolution Upgrades completed
Started 3 Jan 18:00:00
Closed 28 Jan 10:43:34
Previously expected 24 Jan

31 Jan 16:43:03
Details
19 Jan 12:03:02
Due to upgrades on a carrier's network, data SIM services are at risk from 22:00 on Monday the 23rd January to 01:59 on Tuesday the 24th of January. There may also be brief periods of disruption throughout this window.
Closed 31 Jan 16:43:03
Previously expected 24 Jan 02:00:00

31 Jan 16:29:00
Details
31 Jan 16:24:03
Customers on one of our LNSs just lost their connection and would have logged back in again shortly after. We're investigating the cause
Update
31 Jan 16:41:32
Customers are back online. The CQM graphs for the day would have been lost for these lines. We do apologise for the inconvenience this caused.
Broadband Users Affected 12%
Started 31 Jan 16:16:00
Closed 31 Jan 16:29:00

27 Jan 16:55:00
Details
26 Jan 13:56:39
Our Data only SIMs will be 4G LTE enabled over the coming days. They will still support 3G as they do now but where available 4G will also be supported.
Resolution This has been done.
Started 26 Jan 13:50:00
Closed 27 Jan 16:55:00

24 Jan 18:15:00
Details
24 Jan 16:11:45
Some TalkTalk connected customers have high packetloss on their lines from around 3pm today. These lines are in the Chippenham/Bristol area. If affected you'll be experiencing slow speeds.
Update
24 Jan 16:19:23

Affected lines are looking like this. This shows the fault started just after 9am, but from 3pm there is severe packet loss.

Update
24 Jan 18:32:37
TalkTalk say "NOC & Network engineering are currently investigating congestion and packet loss across the core network." More details to follow.
Update
24 Jan 18:45:58
Problem looks fixed as of 18:15
Update
25 Jan 08:48:01
(This also affected some other circuits in other parts of the country.)
Resolution From TalkTalk: Root cause has not currently been identified.. The (TalkTalk) NOC engaged Network Support, who investigated and added a new link in order to alleviate congestion. The B2B Enterprise team are currently retesting with the affected customers and initial feedback indicates that this has resolved the issue
Broadband Users Affected 1%
Started 24 Jan 15:00:00
Closed 24 Jan 18:15:00

23 Jan 21:50:24
Details
23 Jan 21:17:18
Since 20:23 we're seeing ~20% packet loss on TalkTalk connected VDSL circuits, these customers will be experiencing very slow speeds. These are in the SALTERTON/DORCHESTER/WESTBOURNE/CRADDOCK area. We have contacted TalkTalk regarding this.
Update
23 Jan 21:50:48
This looks to have been fixed.
Resolution This was due to a card failure at Yeovil
Started 23 Jan 20:23:00
Closed 23 Jan 21:50:24
Cause TT

23 Jan 17:03:31
[DNS, Email and Web Hosting] - SSL Certificates Updating - Info
Details
23 Jan 17:03:31
We're updating SSL certificates for our email servers today. The old serial number is 124247. The new serial number is 12AD7B. Users who don't have the CAcert root certificate installed may see errors. This does not affect webmail or outgoing SMTP. Details on http://aa.net.uk/cacert.html

24 Jan
Details
23 Jan 08:21:07
Sorry to say that the new LNSs (H and I) were not archiving graphs and so the CQM graphs for customers on these LNSs have not been recorded.
Resolution Fixed
Started 16 Jan
Closed 24 Jan
Previously expected 24 Jan