Order posts by limited to posts

18 May 16:30:59
Details
7 Feb 14:32:32

We are seeing issues with IPv6 on a few VDSL cabinets serving our customers. There is no apparent geographical commonality amongst these, as far as we can tell.

Lines pass IPv4 fine, but only intermittently passing IPv6 TCP/UDP for brief amounts of time, usually 4 or so packets, before breaking. Customers have tried BT modem, Asus modem, and our supplied ZyXEL as a modem and router, no difference on any. We also lent them a FireBrick to do some traffic dumps.

Traffic captures at our end and the customer end show that the IPv6 TCP and UDP packets are leaving us but not reaching the customer. ICMP (eg pings) do work.

The first case was reported to us in August 2016, and it has taken a while to get to this point. Until very recently there was only a single reported case. Now that we have four cases we have a bit more information and are able to look at commonalities between them.

Of these circuits, two are serving customers via TalkTalk and two are serving customers via BT backhaul. So this isn't a "carrier network issue", as far as we can make out. The only thing that we can find that is common is that the cabinets are all ECI. (Actually - one of the BT connected customers has migrated to TalkTalk backhaul (still with us, using the same cabinet and phone line etc) and the IPv6 bug has also moved to the new circuit via TalkTalk as the backhaul provider)

We are working with senior TalkTalk engineers to try to perform a traffic capture at the exchange - at the point the traffic leaves TalkTalk equipment and is passed on to Openreach - this will show if the packets are making it that far and will help in pinning down the point at which packets are being lost. Understandably this requires TalkTalk engineers working out of hours to perform this traffic capture and we're currently waiting for when this will happen.

Update
2 Mar 11:14:48
Packet captures on an affected circuit carried out by TalkTalk have confirmed that this issue most likely lies in the Openreach network. Circuits that we have been made aware of are being pursued with both BT and TalkTalk for Openreach to make further investigations into the issue.
If you believe you may be affected please do contact support.
Update
17 Mar 09:44:00
Having had TalkTalk capture the traffic in the exchange, the next step is to capture traffic at the road-side cabinet. This is being progresses with Openreach and we hope this to happen 'soon'.
Update
29 Mar 09:52:52
We've received an update from BT advising that they have been able to replicate the missing IPv6 packets, this is believed to be a bug which they are pursuing with the vendor.

In the mean time they have also identified a fix which they are working to deploy. We're currently awaiting further details regarding this, and will update this post once further details become known.
Update
18 May 16:30:59
We've been informed that the fix for this issue is currently being tested with Openreach's supplier, but should be released to them on the 25th May. Once released to Openreach, they will then perform internal testing of this before deploying it to their network. We haven't been provided with any estimation of dates for the final deployment of this fix yet.
In the interim, we've had all known affected circuits on TalkTalk backhaul have a linecard swap at the cabinet performed as a workaround, which has restored IPv6 on all TT circuits known to be affected by this issue.
BT have come back to us suggesting that they too have a workaround, so we have requested that it is implemented on all known affected BT circuits to restore IPv6 to the customers known to have this issue on BT backhaul.
Broadband Users Affected 0.05%
Started 7 Feb 09:00:00 by AA Staff
Update was expected Today 18:00:00

8 May 09:26:21
Details
26 Apr 11:01:07
We have noticed packetloss between 8pm and 10pm on Tuesday (25th April) evening on a small number of TalkTalk connected lines. This may be related to TalkTalk maintenance. We will review this again tomorrow.
Update
26 Apr 16:41:43
We are seeing packet loss this afternoon on some of these lines too. We are contacting TalkTalk.
Update
26 Apr 16:44:23
Update
26 Apr 17:58:06
We have moved TalkTalk traffic over to our Harbour Exchange interconnect to see if this makes a difference or not to the packet loss that we are seeing...
Update
26 Apr 20:50:41
Moving the traffic made no difference. We've had calls with TalkTalk and they have opened an incident and are investigating further.
Update
26 Apr 20:55:14
The pattern that we are seeing relates to which LAC TT are using to send traffic over to us. TT use two LACs at their end, and lines via one have loss whilst lines via the other have no loss.
Update
26 Apr 21:32:30
Another example, showing the loss this evening:

Update
26 Apr 22:26:50
TalkTalk have updated us with: "An issue has been reported that some Partners are currently experiencing quality of service issues, such as slow speed and package (SIC) loss, with their Broadband service. From initial investigations the NOC engineers have identified congestion to the core network connecting to Telehouse North as being the possible cause. This is impacting Partners across the network and not specific to one region, and the impacted volume cannot be determine at present. Preliminary investigations are underway with our NOC and Network Support engineers to determine the root cause of this network incident. At this stage we are unable to issue an ERT until the engineers have completed further diagnostics."
Update
28 Apr 09:43:43
Despite Talktalk thinking they had fixed this we are still seeing packetloss on these circuits between 8pm and 10pm. It's not as much packetloss as we saw on Wednesday evening, but loss nonetheless. This has been reported back to TalkTalk.
Update
4 May 15:42:04
We are now blocking the two affected Talkalk LACs on new connections, eg a PPP re-connect. This means that it will take a bit longer for a line to re-connect (depending upon the broadband router perhaps a minute or two).

This does mean that lines will not be on the LACs which have evening packetloss. We hope not to have to keep this blocking in place for very long as we hope TalkTalk fix this soon.

Update
5 May 16:48:13
We've decided to not block the LACs that are showing packetloss as it was causing connection problems for a few customers. We have had a telephone call with TalkTalk today, and this issue is being escalated with TalkTalk.
Update
5 May 19:47:37
We've had this update from TalkTalk today:

"I have had confirmation from our NOC that following further investigations by our IP Operations team an potential issue has been identified on our LTS (processes running higher than normal). After working with our vendor it has been recommended a card switch-over should resolve this.

This has been scheduled for 16th May. We will post further details next week.

Update
8 May 09:26:21
Planned work has been scheduled for 16th May for this, details and updates of this work is on https://aastatus.net/2386
Broadband Users Affected 20%
Started 26 Apr 10:00:00
Update was expected 16 May 12:00:00

3 May 13:20:48
Details
26 Apr 10:16:02
We have identified packet loss across our lines at MOSS SIDE that occurs between 8pm to 10pm. We have raised this with BT who suggest that they hope to have this resolved on May 30th. We will update you on completion of this work.
Broadband Users Affected 0.15%
Started 26 Apr 10:13:41 by BT
Update expected 30 May 12:00:00
Expected close 30 May (Estimated Resolution Time from BT)

17 May 12:30:00
Details
17 May 09:10:40
We have identified outages within the North London area affecting multiple exchanges. The affected exchanges are listed as: SHOEBURYNESS,THORPE BAY, CANVEY ISLAND, HADLEIGH – ESSEX, MARINE, NORTH BENFLEET, STANFORD LE HOPE, VANGE, WICKFORD, BLOOMSBURY AKA HOWLAND ST, HOLBORN, PRIMROSE HILL, KINGSLAND GREEN, TOTTENHAM, BOWES PARK, PALMERS GREEN, WALTHAM CROSS, WINCHMORE HILL, EDMONTON, NEW SOUTHGATE, EPPING, HAINAULT, ILFORD NORTH, ROMFORD, UPMINSTER, NORTH WEALD, STAMFORD HILL, DAGENHAM, ILFORD CENTRAL, GOODMAYES, STRATFORD, HIGHAMS PARK, LEYTONSTONE, WALTHAMSTOW, CHINGFORD, KENTISH TOWN and MUSWELL HILL. No root cause has been identified. We will update this status page as the updates become available from our supplier.
Update
17 May 09:30:01
Affected Customers went offline at 02:12 this morning. Further information: These exchanges are off line due to 20 tubes of fibre being damaged by the install of a retail advertising board on the A118 Southern [South?] Kings Rd. Notifications from TalkTalk were expecting service to be restored by 9am, but due to the nature of the fibre break it may well take longer to fix.
Update
17 May 10:31:29
Update from TalkTalk: Virgin media have advised that work to restore service is ongoing but due to the extent of the damage this is taking longer than expected. In parallel our [TalkTalk] NOC are investigating if the Total Loss traffic can be re-routed. We will provide a further update as soon as more information is available.
Update
17 May 10:55:56
From TalkTalk: Virgin Media have advised restoration work has completed on the majority of the damaged fibre, our NOC team have also confirmed a number of exchanges are now up and back in service. The exchanges that are now back in service are Muswell Hill, Ingrebourne, Loughton and Bowes Park.
Update
17 May 11:16:28
From TalkTalk: Our NOC team have advised a number of exchanges are now in service. These are Muswell Hill, Bowes Park, Loughton, Ingrebourne, Bowes Park, Chingford, Highams Park, Leytonstone, Stratford and Upton Park.
Update
17 May 11:30:54
That said, we are still seeing lines on exchanges mentioned above as being offline....
Update
17 May 12:11:20
No further updates as yet.
Resolution It looks like most, if not all, of our affected lines are now back online. Update from TalkTalk: Virgin Media have advised 5 of the 8 impacted fibre tubes have been successfully spliced and their engineers are still on site restoring service to the remaining cables
Started 17 May 01:54:00 by AA Staff
Closed 17 May 12:30:00

4 May 18:00:00
[Broadband] - TT Blips - Closed
Details
4 May 16:53:39
Some TT lines blipped at 16:34 and 16:46. It appears that the lines have recovered. We have reported this to TT.
Update
5 May 08:48:41
This was caused by an incident in the TalkTalk network. This is from TalkTalk: "...Network support have advised that the problem was caused by the card failure which has now been taken offline..."
Started 4 May 16:36:27 by AAISP automated checking
Closed 4 May 18:00:00
Cause TT

21 Apr 13:07:49
Details
21 Apr 09:59:57
At 9:30 we saw a number of TalkTalk connected circuits drop and reconnect. We're investigating, but seems like a TalkTalk backhaul issue

Update
21 Apr 11:24:54
Most lines logged back in very quickly. A few are still down. It seems TalkTalk had some sort if incident at the Telehouse Harbour Exchange datacentre, we're expecting further information shortly.
Update
21 Apr 12:54:00
TalkTalk have some sort of outage in their Harbour Exchange datacentre. As we have an interconnect in both Telehouse and Harbour Exchange the traffic has moved over to Telehouse. There are a very small number of customers (<10) who are still offline, they may just need a reboot of their router or modem, but we are contacting them individually.
Update
21 Apr 13:08:47
TalkTalk say: Root cause analysis conducted by the NOC and our Network Support team has identified that this incident was caused by a crashed routing card for the LTS. The card was reloaded by our Network Support team and full service has now been restored.
Resolution Any customer still offline, please reboot your router/modem and then contact Support if still off.
Started 21 Apr 09:30:50
Closed 21 Apr 13:07:49

2 Mar 22:10:44
Details
2 Mar 21:48:39
Relating to https://aastatus.net/2358 we are undergoing currently in an emergency at-risk period as we perform some tests along side TalkTalk staff. We don't expect any problems, but this work involves re-routing TalkTalk traffic within our network. This work is happening now. Sorry for the no notice.
Update
2 Mar 21:53:05
We have successfully and cleanly moved all TalkTalk traffic off our THN interconnect and on to our HEX Interconnect. (Usually we use both all the time, but for this testing we are forcing traffic through the HEX side)
Update
2 Mar 21:55:52
We're bringing back routing across both links now...
Update
2 Mar 22:03:40
We are now moving traffic to our THN interconnect.
Resolution We're now back to using both the TalkTalk links. Tests completed.
Started 2 Mar 21:46:17
Closed 2 Mar 22:10:44

16 Feb 15:00:00
Details
16 Feb 16:00:49
We have spotted some odd latency that was affecting two of our LNSs (A and B gormless). These were also visible, as you would expect, on the graphs shown for people's lines.
Resolution We believe we have addressed the issue now, sorry for any inconvenience.
Started 15 Feb 02:00:00
Closed 16 Feb 15:00:00
Previously expected 16 Feb 15:00:00

13 Feb 10:02:12
[Broadband] - LNS blip - Closed
Details
13 Feb 10:00:36
We just had an LNS blip - this would have caused some customers to drop PPP and reconnect.
Resolution There have been a few LNS blips recently. However, we do know the cause and have a software update to roll out which will fix the problem.
Started 13 Feb 09:56:00
Closed 13 Feb 10:02:12

4 Feb 09:32:03
[Broadband] - LNS blip - Closed
Details
4 Feb 09:14:11
We had an LNS reset and lines will have re-connected for some customers. We're investigating the cause.
Resolution We have found the cause, and expect a permanent fix to be deployed on next round of LNS upgrades.
Broadband Users Affected 12%
Started 4 Feb 09:12:00
Closed 4 Feb 09:32:03

31 Jan 16:29:00
Details
31 Jan 16:24:03
Customers on one of our LNSs just lost their connection and would have logged back in again shortly after. We're investigating the cause
Update
31 Jan 16:41:32
Customers are back online. The CQM graphs for the day would have been lost for these lines. We do apologise for the inconvenience this caused.
Broadband Users Affected 12%
Started 31 Jan 16:16:00
Closed 31 Jan 16:29:00

24 Jan 18:15:00
Details
24 Jan 16:11:45
Some TalkTalk connected customers have high packetloss on their lines from around 3pm today. These lines are in the Chippenham/Bristol area. If affected you'll be experiencing slow speeds.
Update
24 Jan 16:19:23

Affected lines are looking like this. This shows the fault started just after 9am, but from 3pm there is severe packet loss.

Update
24 Jan 18:32:37
TalkTalk say "NOC & Network engineering are currently investigating congestion and packet loss across the core network." More details to follow.
Update
24 Jan 18:45:58
Problem looks fixed as of 18:15
Update
25 Jan 08:48:01
(This also affected some other circuits in other parts of the country.)
Resolution From TalkTalk: Root cause has not currently been identified.. The (TalkTalk) NOC engaged Network Support, who investigated and added a new link in order to alleviate congestion. The B2B Enterprise team are currently retesting with the affected customers and initial feedback indicates that this has resolved the issue
Broadband Users Affected 1%
Started 24 Jan 15:00:00
Closed 24 Jan 18:15:00

23 Jan 21:50:24
Details
23 Jan 21:17:18
Since 20:23 we're seeing ~20% packet loss on TalkTalk connected VDSL circuits, these customers will be experiencing very slow speeds. These are in the SALTERTON/DORCHESTER/WESTBOURNE/CRADDOCK area. We have contacted TalkTalk regarding this.
Update
23 Jan 21:50:48
This looks to have been fixed.
Resolution This was due to a card failure at Yeovil
Started 23 Jan 20:23:00
Closed 23 Jan 21:50:24
Cause TT

24 Jan
Details
23 Jan 08:21:07
Sorry to say that the new LNSs (H and I) were not archiving graphs and so the CQM graphs for customers on these LNSs have not been recorded.
Resolution Fixed
Started 16 Jan
Closed 24 Jan
Previously expected 24 Jan

18 Jan 20:30:00
Details
18 Jan 20:36:56
We're looking in to why some broadband lines and mobile SIMs dropped and reconnected at around 20:30 this evening....
Resolution Lines are back online, most reconnected within a few minutes. This blip affected about 1/8th of our customers, and was caused by one of our LNS restarting unexpectedly. We do apologise for the inconvenience this caused. We'll be investigating the cause of this.
Started 18 Jan 20:35:58
Closed 18 Jan 20:30:00
Cause LNS restart/crash

17 Jan 09:48:47
Details
17 Jan 08:35:28
Once again we are seeing an issue where TT lines are failing to connect. This is not impacting lines that are currently connected unless they drop and reconnect for some reason. This looks like only half of TTs LACs that is impacted, and so lines are eventually reconnecting after several tries. It has been reported to TalkTalk and we will update this post as soon as we get an update.
Update
17 Jan 09:50:18
All affected lines appear to have reconnected.
Resolution We are still investigating the root cause
Broadband Users Affected 1%
Started 17 Jan 01:00:00
Closed 17 Jan 09:48:47
Previously expected 17 Jan 12:31:59