Order posts by limited to posts

Yesterday
[Broadband and Ethernet] - At risk: Router upgrades - Open
Details
28 Apr 09:29:41
We may do some router upgrades and changes over the weekend as part of ongoing network tidy-up. This should not be disruptive.
Update
Yesterday 08:52:48
The first changes have been done, with most likely impact (if any) being on our office connections. If all goes well today we'll do more changes tomorrow (Sunday) morning. As I say, these should have no noticeable impact, but are to improve some of our internal routing logic.
Update
Today 05:47:14
The next stage was done this morning. Any issues please let us know on irc. We have more changes later today (Sunday).
Update
Today 09:53:27
Some further work was done before 8am today (Sunday). We plan to do the next stage tomorrow (Monday) morning. So far all seems to be going to plan.
Started Yesterday
Expected close Tuesday

5 Feb 16:32:23
Details
5 Feb 16:32:23
Some call recordings from today have been lost. We've identified the problem and have worked around it until we can fix it. We should be able to fix it with no customer impact. If you are expecting a call recording that you have not received, support can check whether the call was affected by this problem. The call recordings lost are in the minority and most calls were recorded as expected. This problem is unrelated to yesterday's problems with lack of ring tone!
Started 5 Feb 16:25:24

12 Jun 2015 11:06:49
Details
12 Jun 2015 11:00:23
Our office connectivity blipped. We have internet again, but our phones are down. We're investigating. This is not customer affecting.
Update
12 Jun 2015 11:06:49
Phones are working again.
Previously expected 12 Jun 2015 14:57:07

13 Apr 2015
Details
09 Apr 2015 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr 2015

12 Mar 2015 09:48:01
Details
12 Mar 2015 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.

08 Jan 2015 12:39:06
Details
08 Jan 2015 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
Update
09 Jan 2015 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
Update
12 Jan 2015 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 08 Jan 2015 12:39:06 by AA Staff
Previously expected 15 Jan 2015 17:00:00

03 Jun 2014 17:00:00
Details
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
Update
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
Details
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
Update
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

Update
22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
Update
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
Update
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
Update
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
Update
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
Update
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
Details
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
Details
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
Update
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
Details
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

Details
06 Jul 2015 12:49:42
We have been advised by Three of works on their network (22:30 8th July to 04:50 9th July 2015) Which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some partners may experience dropped connections. Any device reconnecting will automatically be routed via a new path. We apologise for any inconvenience this may cause and the short notice of this advisory.

Details
12 Feb 2015 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

28 Apr 16:18:20
Details
28 Apr 15:30:05
We are seeing a number of broadband lines in the south of London suffering from severe packet loss and high latency. This looks like a fibre problem with TalkTalk. This has been reported and as soon as we have an update we will let you know.
Update
28 Apr 15:36:41
Here is an example graph
Update
28 Apr 16:18:44
TalkTalk have fixed this - that was quick.....
Started 28 Apr 15:27:37
Closed 28 Apr 16:18:20
Previously expected 28 Apr 19:27:37

28 Apr 04:55:55
Details
27 Apr 11:21:25
We are making a correction to routing. This will impact external connectivity for a few seconds but should not cause lines to drop PPP. We aim to do this over night or early tomorrow morning. Sorry for any inconvenience.
Resolution Changes complete
Started 28 Apr
Closed 28 Apr 04:55:55
Previously expected 28 Apr 08:00:00

27 Apr 06:09:11
[Broadband and Ethernet] - At risk: Routing changes - Completed
Details
26 Apr 16:58:57
We are tidying up some of our internal routing logic following from our major network upgrades recently. This is expected to have no impact, but there is a risk it could impact Ethernet and broadband customers and maybe other services. The plan is to get this all done well before 8am so that we can sort any issues that come up or reverse changes that are made.
Resolution The work is complete and all looks well - any issues, please let us know. P.S. an issue with BGP connections via DSL was resolved.
Started 27 Apr
Closed 27 Apr 06:09:11
Previously expected 27 Apr 09:00:00

26 Apr 17:00:00
Details
26 Apr 16:40:46

Some Ethernet customers have seen a routing blip today. This is mainly those that have a secondary block of IPs routed via your Ethernet link and related to that additional block.

We should have posted an "at risk" notice about some very minor routing changes tidying up after all of the recent work in our core network. I genuinely did not expect any impact - we have done simple routing changes many times in the past without experiencing any issues.

Unfortunately the changes uncovered an underlying problem with the redundancy logic, and hence something that should have had no impact did in fact have an impact. This was fixed quickly within minutes of the report.

There is other work we expect to do tomorrow morning before 9am, including sorting the underlying issue we have uncovered today. If there are any issues you encounter with routing in the morning, please do contact support.

Sorry for any inconvenience caused.

Started 26 Apr 13:00:00
Closed 26 Apr 17:00:00
Previously expected 26 Apr 17:00:00

19 Apr 12:42:22
[General] - DRIPA - Info
Details
16 Jul 2014 10:13:15
Just to be clear on our policy here - when DRIPA comes in to force, and if A&A become subject to a retention notice for all customers, we aim to work on all practical legal means to minimise the amount of data retained under that legislation - making full use of the bad wording in the Schedule in the 2009 regulations where possible. We also aim to clearly publish what is retained under such a notice and what steps we have taken to minimise such data. Such steps may mean separate companies running email or other services, or even hosting some servers outside the UK, if those are practical steps we can take. Why? Because blanket mass surveillance is illegal under EU law as it is against our basic human right to privacy as decided by a court, that's why! Feel free to comment on my blog.
Started 15 Jul 2014

17 Apr 19:33:54
Details
14 Apr 16:22:27

As customers may recall, some time ago we ported numbers in several exchange area codes to a company called Incoming Numbers Ltd, and then leased use of those numbers back to A&A. The effect was no technical change to operation, but it meant the numbers were technically ported to another provider. When you ordered numbers in those areas, the order form confirmed you were porting to INL. The bills showed the numbers were ported to INL. This is only a small number of area codes.

We're pleased to confirm that we have today reversed that and the ported numbers are now back with us, Andrews & Arnold Ltd. So the bill will no longer show the note about Incoming Numbers Ltd.

The reason for this was that OFCOM started charging for numbers, after having provided them for no charge, and waited for us to start using the blocks. Something that in any other "business" would be an unacceptable business practice, in my view. There was a discount for ported out numbers, hence the porting.

The plan worked, and we got the credit for ported out numbers off the bill. However, we found a much more effective solution. Most of our numbers are allocated to Andrews & Arnold (numbers) Ltd, a company which is not, itself, any more, a telco, and just leases the numbers to Andrews & Arnold Ltd to use. As it is not a telco, OFCOM do not charge it, and so problem solved.

Sorry for the puzzlement this may have caused some customers in the past. It caused me some paperwork as INL is a proper separate trading company with accounts, and everything, but not an A&A subsidiary. It will be wound up or made dormant for now.

I hope both of these methods, to avoid being charged for something we already had, have been helpful to OFCOM as they said the charging was an "experiment", and it was clearly our duty to provide the best data to them on how well it worked as an experiment.

Started 14 Apr 16:13:50
Previously expected 14 Apr 20:13:50

17 Apr 19:33:27
[Broadband and Ethernet] - Home::1 Terabyte bonus - Info
Details
4 Apr 20:26:22

Today we tweeted, somewhat as a joke, "Customers on Home::1 TB services downloading the Panama papers, let us know when you hit your 1TB quota and we'll topup 4 free #PanamaLeaks"

This was mainly because this was something, at 2.7TB, that was actually too big for our monthly allowance on the terabyte service. It shows that even 1TB is not enough for that amount of data, but it really is one of the few things that could cause you to download so much.

However, I also appreciate that we have had had some issues for some weeks now, and they are finally coming to an end with much needed capacity upgrades.

So, as a thank you for the customers that have embraced the new Home::1 Terabyte tariffs, I have actually added 3TB to this month's quota (as top-up, so carrying to end of next month). Thus honouring the tweet for all.

Thank you all - without customers we would be nothing.

Started 4 Apr 20:21:04
Previously expected 5 Apr 00:21:04

17 Apr 19:32:58
[Broadband and Ethernet] - LNS roll overs - Info
Details
14 Apr 08:29:19

I am pleased to confirm that as we now have many more LNSs in service we can return to the normal LNS roll-over strategy http://aa.net.uk/kb-broadband-duallns.html

We moved B.gormless to A.gormless last night, and would normally do one per night to move through all of the LNSs. However, this time we are doing some hardware reconfiguration of LNSs as well, so the next roll-over (C.gormless to B.gormless) will be some time next week.

The roll-over went smoothly as usual. Do remember that you can configure a preferred time in the control pages, as we know some people are up late working or gaming, and others are up early, and we want to minimise any disruption to your connection by routine maintenance like this.

Started 14 Apr 01:00:00
Previously expected 17 Apr 23:32:48

14 Apr 16:27:06
Details
01 Dec 2015 10:00:08
We are starting to use a new address for the Control Pages. Instead of https://clueless.aa.net.uk it'll be https://control.aa.net.uk. The reasons for this are two fold:
  1. Control.aa.net.uk is a better description of the service than clueless.aa.net.uk - some customers have been a little confused or wary of the name 'clueless'.
  2. The new name has given us the opportunity to update the SSL (security) settings.
The slight downside of this is that older clients may not have the modern ciphers needed. For example, the minimum browser versions would be:
  • Firefox 27
  • Chrome 22
  • IE 11
  • Opera 14
  • Safari 7
  • Android 4.4
  • Java 8

We would recommend that customers update their browsers if they are using the above.

As always, we're happy to receive feedback.

Resolution Launched
Started 01 Dec 2015 09:04:43

14 Apr 16:26:42
Details
21 Dec 2015 08:46:32
Please see our most recent newsletter: aa.net.uk/news-2016-01.html
Started 21 Dec 2015 08:45:00

14 Apr 16:25:00
Details
02 Aug 2014 12:02:18
We are planning an expansion of our VoIP services which will require the use of more IP addresses than we have told customers to allow through their firewalls, so have added additional IP ranges to the list. We are not using these new IP ranges yet, but are giving advanced notice of the change to give customers time to update their firewall rules. The list is at http://wiki.aa.org.uk/VoIP_Firewall
Started 02 Aug 2014 11:50:35
Previously expected 02 Aug 2014 11:50:35

14 Apr 16:24:41
Details
02 Jul 2015 10:44:33
Some customers have advised that they are unable to call some numbers which they were able to call last week. This is due to new rules which mandate that we charge a service charge set by the range holder plus our access fee (2p/min) for such numbers. OFCOM have failed to provide means for telcos to determine the service charge that applies. Their only suggestion was to use the Bt Carrier Price list. We have loaded this (and are doing so every day) but it seems to be lacking a number for 084 and 087 number ranges. Until those appear in the BT Carrier Price list, or OFCOM actually come up with a means for telcos to comply with the new regulations, the numbers cannot be called as we have no way to comply with OFCOM rules. Feel free to complain to OFCOM over the stupidity of these new rules.
Update
07 Jul 2015 11:14:47
We have managed a one-off load of current codes, but there still remains no official source of the codes and rates due to OFCOM's failure to define who is responsible for compiling and maintaining this information. As such there is still no sensible way to ensure this data remains correct and complete.
Started 01 Jul 2015
Previously expected 01 Aug 2015

14 Apr 16:24:06
[VoIP and SIMs] - 999 location data - Info
Details
12 Mar 2015 18:24:06
We allow users to provide address/location data to pass to emergency services in the event of a 999 calls. This goes via BT and BT are being a pain at present. As we have explained the data is simply a matter of what customers provide. VoIP has no inherent way to provide an accurate geographic location of callers. But BT are unhappy with some of the data and seem to have blocked us sending updates this week. They even suggested that they have to send OFCOM reports of incomplete and incorrect data as some sort of threat. We know a lot of people have multiple locations from which they make calls. We do this purely as a good will gesture and in the spirit of GC4, because we don't have location data in the first place. So if BT do not stop buggering about soon we'll be reporting BT to OFCOM instead. Anyway, the upshot is that a few customers that have provided location data over that last few days are not yet updated to BT and hence the emergency services. We are working to resolve this and hope to have it working again soon. The config pages for your VoIP set up show if there is an issue and you get an email once it is all confirmed. In the mean time, we do apologise for any concern this may cause. None of our services are provided to be used in any "safety of life" circumstances, as per our standard terms.
Update
16 Mar 2015 16:49:34
We are hoping to have this sorted, perhaps today. It is likely that anyone with existing data registered may get a new email at some point confirming the data as we are planning to re-send everything from scratch just to be sure. We would ask that customers review what they have entered and ensure it is as accurate as possible to assist emergency services in the event of a 999 call.
Update
18 Mar 2015 17:40:37
Data is updating again to BT. We would recommend customers ensure accurate data on the control pages for 999 calls. We will review this over the next few days, and customers may still get update emails in due course.
Started 08 Mar 2015
Previously expected 16 Mar 2015

11 Apr 12:49:31
Details
11 Apr 11:56:02
We are investigating an issue with one of our call servers (A.Voiceless). DNS has been updated to route via B, which should ensure service is restored for those that are affected but it is worth forcing phones to re-register if you are having issues (some VoIP devices do not cache DNS properly).
Update
11 Apr 12:09:53
Engineers on on their way to the datacentre. Calls and registrations are all being handled via our 'B' call server.
Update
11 Apr 12:34:29
Engineers are onsite.
Update
11 Apr 12:38:00
Hardware is being replaced.
Update
11 Apr 12:44:39
Hardware replaced, and server online.
Resolution The faulty server has been repaired and back on line.
VoIP and SIMs Users Affected 50%
Started 11 Apr 11:50:00
Closed 11 Apr 12:49:31
Previously expected 11 Apr 12:50:00

4 Apr 15:31:58
[Broadband and Ethernet] - Bogus uplink speeds from BT - Closed
Details
4 Apr 14:44:27
Since our software update on our LNSs on Sunday we are now able to pick up the uplink speeds for DSL lines from the carrier.

However, some customers are seeing slow upload speeds and also a red line on the bottom of our monitoring graphs indicating the link is full. The problem is that BT are sending bogus uplink speeds for some lines, and therefore we are limiting the throughput.

We are working on a fix for this now.

Update
4 Apr 15:32:16
This has now been resolved.
Started 4 Apr 14:42:19
Closed 4 Apr 15:31:58

17 Apr 19:32:20
[Broadband and Ethernet] - At risk: Routing changes - Completed
Details
3 Apr 17:44:24
We have a number of things to do over the coming few weeks with changes to the peering and transit links we use. This should not impact customers directly, and is some minor routing changes, but it is the final step in managing the capacity upgrades. So there is a small risk of some side effects. We'll aim to keep these outside office hours as much as possible.
Update
4 Apr 18:13:56
We have sorted one of the links to LONAP today, but later in the week we will be moving more transit and peering links, step by step, carefully. This should not impact service, but will allow more capacity and future proofing.
Update
13 Apr 10:37:38
The next stage of work is scheduled for Wednesday 20th, again we don't expect any customers to be affected by this as it is minor configuration changes at our side.
Started 4 Apr
Closed 17 Apr 19:32:20

3 Apr 16:36:34
[Broadband and Ethernet] - L2TP direct service - Info
Details
3 Apr 16:36:34

Now that we have the new network upgrade we are hoping to properly launch the new L2TP specific service. Previously this was a somewhat informal service with somewhat complicated billing.

This new service is an L2TP over IP service for people using other broadband lines to access us and use our IPs and services including fixed IPv4 and IPv6 blocks. The plan is £10/month for a service capped at 100Mb/s with 1TB monthly allowance, clamping to 3Mb/s from next month's allowance on reaching allowance. Very similar to Home::1. No date yet but we are hoping to make this our standard L2TP service some time in April.

Update
17 Apr 19:34:56
I'm sorry this is taking so long to sort - it is basically a tariff change, and order form, but we have been very keen to ensure the infrastructure is in place for this. I still hope to do this by end of month.
Started 1 Apr
Previously expected Today

3 Apr 14:54:34
Details
3 Apr 14:46:09
It looks like there was an issue with one of the ports on one of the LNSs for a while this afternoon. We are looking in to it, but it could be some sort of attack. Obviously this is never good, but this appears to be limited to half of one LNS.
Update
3 Apr 14:54:49
This does look like a targeted attack
Update
3 Apr 15:59:02
It looks like, in addition to the specific port on the LNS being impacted, there may have been some more general issues for routes on one of our peering points (via which the attack came in).
Resolution It seems to have subsided now. We are trying to investigate more on this, but it does not apparent to be related to the network restructure this morning, thankfully. This customer that was the target has been contacted.
Broadband and Ethernet Users Affected 8%
Started 3 Apr 14:30:00
Closed 3 Apr 14:54:34

3 Apr 08:00:00
Details
27 Mar 21:17:18
We are planning, at the very least, to upgrade the live LNSs all in one go. However, we are hopeful that we can also upgrade the routed links to both BT and Talk Talk at the same time. This should "just work" but if it does not we will roll back. The effect will be at least one, and possibly more PPP level restarts of all broadband lines and may be a few minutes of outage. If we are successful we will be able to deploy more active LNSs in the following week to spread load and allow more expansion. It will also allow us to more easily expand the number of links we have to carriers over the coming months. This is all part of a major ongoing network capacity upgrade.
Update
31 Mar 18:07:55
We have done a lot of work today - both in the data centre (deploying five more FB6000 series FireBricks) and from the office configuring systems ready for the weekend. Well done to all of the staff involved as I do not believe there was a single blip. I feel very confident that we are ready to the work on Sunday now.
Update
2 Apr 07:49:05
More preliminary testing yesterday and today but looking like we are on schedule for Sunday morning. We aim to upgrade all LNSs, change carrier interlink so we can use more LNSs and actually start using more LNSs.
Update
3 Apr 02:03:56
The work begins...
Update
3 Apr 02:10:26
LNSs are having their software updated, this will cause PPP blips for customers.
Update
3 Apr 02:22:29
Software has been updated on the LNSs. Stage two will start shortly, this is reconfiguring our links to BT and TT backhaul.
Update
3 Apr 03:15:58
We're about to start reconfiguring the TT interconnects. This will cause a PPP blip for customers on T backhaul.
Update
3 Apr 03:30:34
TT lines have just blipped as we reconfigured the backhaul links. Lines are logging back in now.
Update
3 Apr 03:49:37
TT is working but we are making sure the balancing works before we proceed to the BT side
Update
3 Apr 04:08:26
We're about to start reconfiguring the BT interconnects. This will cause a PPP blip for customers on BT backhaul.
Update
3 Apr 04:24:39
BT lines blipped their PPP as expected, and are logging back in now.
Update
3 Apr 04:32:52
BT lines looking good.
Update
3 Apr 04:41:02
All lines are about to have a short PPP blip as we load new software across the LNSs.
Update
3 Apr 04:52:13
We are not balancing the BT back haul links properly at the moment, this is being worked on. This is not affecting service at the moment but does mean we are only running on half our links to BT.
Update
3 Apr 05:13:15
We are close, but we do have to break things once more sadly, there will be some routing and PPP drop over the next 30 minutes.
Update
3 Apr 05:39:37
Lines are reconnecting and routes are being restored We'll then review where we are and will update again shortly.
Update
3 Apr 05:56:12
Customers are online, but we are still working on balancing the BT links.
Update
3 Apr 06:04:00
We are over running on the window for this planned work. However, customers are online, and we are working on some oddities in how the traffic over our BT links is being balanced.
Update
3 Apr 06:54:37
Things look stable and we have managed to balance the traffic properly we are still tracking down one issue this morning. There may be some re-balancing of traffic between LNSs later.
Resolution It took a long time, but the work has been completed, including the deployment of 3 new LNSs to our pool. Thanks to the staff involved, and to our customers for their patience.

Some further info about this work here: http://www.revk.uk/2016/04/growing-pains-pain-relief.html

Started 3 Apr 02:00:00
Closed 3 Apr 08:00:00
Previously expected 3 Apr 08:00:00

2 Apr 10:17:00
[Broadband and Ethernet] - Recursive DNS resolver issue - Closed
Details
2 Apr 10:21:16
There was an unexpected issue with DNS resolvers not responding due to a routing loop. This has now been rectified. Thanks to all those that contacted us.
Started 2 Apr 09:48:00
Closed 2 Apr 10:17:00

30 Mar 09:16:58
Details
6 Mar 13:25:01

Following the recent launch of our special Home::1 VDSL Terabyte package we are planning a new ADSL version.

Home::1 ADSL Terabyte will be just like normal Home::1 ADSL services but offering 1TB download allowance per month, then slowing to 3Mb/s if you hit the limit. No top-up or excess charges.

The service will be £40/month including VAT, but not including the phone line. You will need a phone line with someone, or we can provide a broadband only line for £10/month as usual.

The service is only available on TalkTalk back-haul, so not available everywhere. Anyone on BT back-haul will have 14 day lead time and a short outage on the line to regrade. Anyone on TalkTalk back-haul will just have this as a normal tariff change applied from the next bill date. Either way, no cost to change to this. The install price as a new service is the same as normal Home::1

At present we are waiting to resolve the capacity issues with Talk Talk back-haul across our core network before we launch the new package, so it should be available later this month.

Update
29 Mar 20:20:46
Expected to launch tomorrow - just confirming the new capacity upgrades work as expected tonight.
Resolution Now launched. http://aa.net.uk/broadband-home1-terabyte.html
Started 6 Mar 13:42:35
Closed 30 Mar 09:16:58
Previously expected 1 Apr

29 Mar 20:08:55
Details
4 Mar 13:41:26

Over the past week there have been times in the evenings where we have seen congestion on some of our TalkTalk ADSL and VDSL circuits. This has shown up in our CQM graphs as latency spikes and/or packet loss.

We would like to reassure customers that this is *not* congestion in TalkTalk's network; it is actually happening inside our own network. We are aware of the cause and are taking actions to stop it.

We mentioned in the January newsletter that we are planning various upgrades to our London network to increase overall capacity. This work is in progress but for a couple of reasons has been delayed and is taking slightly longer than initially planned.

Whilst the London upgrades have been happening we have been trying to mitigate the congestion problem by re-routing traffic in various ways. This has helped but not been a 100% solution, and has affected service for customers. For this we are sorry; we do aim never to be the "bottleneck" and in this case we've not managed to meet that standard.

Looking forward, we have works planned for the datacentre next week and the following week and we'll update this post with progress along the way.

We're happy to discuss this further, do pop in to our IRC channel or email or phone etc.

Update
6 Mar 13:16:44

This work was planned for last year but issues beyond our control (problems with new fibres) had delayed it. The plans now are for testing of fibres to hopefully be completed during this week (7th-11th March), and if all goes well we can start using the new fibre next weekend. Once this is in place we can move the impacted circuits on to the new fibres during the following week to remove the bottleneck in our core network. There is a lot to do, but the one issue right now is one of the TalkTalk back-haul links and that can be the first thing to be moved over.

Once again we apologise for the time this is taking.

We plan to launch a new Home::1 ADSL Terabyte package as soon as these issues are resolved and the capacity is in place.

Update
7 Mar 22:39:11
We have posted details of planned work for Sunday 13th: https://aastatus.net/2222
Update
14 Mar 20:55:09
Further work at the datacentre has been planned, please see: https://aastatus.net/2227
Resolution Network upgrades are now providing additional capacity for TT lines. Thank you all for your patience.
Started 4 Mar 13:00:00
Closed 29 Mar 20:08:55

27 Mar 09:07:44
Details
27 Mar 04:39:48
Since Thursday morning it seems that many of the Technicolor routers have been rather odd, generating thousands of DHCPv6 requests per second. We have remotely rebooted many of these, but some still remain. We will be taking steps later today (Sunday) to shut down IPv6 on the affected routers and advise customers that the router needs power cycling / rebooting. Sorry for any inconvenience, and we are at a loss as to why they are behaving like this.
Resolution We have managed to sort the problem routers for now (mostly by rebooting) - if you don't have IPv6 working properly on a Technicolor - please turn it off and back on again.
Started 24 Mar
Closed 27 Mar 09:07:44
Previously expected 29 Mar

29 Mar 20:00:29
[Broadband and Ethernet] - TT capacity upgrade - Completed
Details
27 Mar 09:11:44
Now we have the new network switches in place we are able to make changes to address Talk Talk capacity issues. We have a plan for this and it should be possible with no disruption. If we are able to, we will do this over the weekend, but if not, then the plan is to make the changes during the day on Tuesday. We'll post an update here when we start. Sorry for the rather wide window on this - it depends on completing testing of the new fibre links to our satisfaction first.
Update
29 Mar 17:40:10
Some of the preliminary switch config changes caused a port to flap unexpectedly which resulted in a blip of some lines, mostly Talk Talk lines at around 16:30. We are still planning to go ahead with the change to allow more TT capacity though and work is continuing in to the evening.
Update
29 Mar 18:42:56
We are starting the work to re-route one of the Talk Talk links now.
Update
29 Mar 19:43:44
Work has gone well so far and customers are seeing much more capacity to Talk Talk lines. We are monitoring the situation carefully at present.
Resolution We are watching things closely, but looking good. Next step is at the weekend.
Started 27 Mar
Closed 29 Mar 20:00:29
Previously expected 30 Mar

28 Mar
Details
24 Mar 08:03:43
We may do some routing adjustments and s/w upgrades over the weekend. These should be minor and not disrupt normal operations. However, there is always a small risk. Basically there is a bit of tidying up to do after the switch change, and we may be able to move some of the traffic around to make a start on capacity issues at some point during the weekend. At this stage we are not sure, so this is more of an "At risk" notice.
Update
25 Mar 14:14:11
We have updated routers and managed with minimal disruption - rerouting traffic around the upgrades as done. We have also made some config changes. We are still trying to work out if there is a way we can resolve some of the TT capacity issues over the weekend or not, so there may be some more work still.
Update
26 Mar 16:07:26
A number of changes have been done today as well. Again, with minimal disruption.
Update
26 Mar 16:08:14
We also found that all of the Technicolor routers were upset by something when we did the switch change on Thursday. We don't understand what exactly, but rebooting is fixing, so we are rebooting Technicolor routers one by one.
Update
27 Mar 04:35:49
The work over the weekend has gone quite well, with very little disruption. We do now have a clear plan for the changes to alleviate the capacity issues. I don't yet know if this will be possible over the weekend but should be possible on Tuesday.
Started 25 Mar
Closed 28 Mar
Previously expected 29 Mar

24 Mar 23:10:00
Details
24 Mar 23:20:23
We have just resolved a problem where by we were not using all our fibres to BT for traffic. this meant that for the past few hours we were running over 3 links instead of the usual 4 and customers were seeing congestion by way of lower than usual throughput as well as some latency and loss. We do apoloise for this. We are now back to normal.
Started 24 Mar 20:00:00
Closed 24 Mar 23:10:00

24 Mar 06:00:00
Details
18 Mar 16:52:32

This is a follow-up to https://aastatus.net/2222 and https://aastatus.net/2229

As we were not able to complete the planned work that was scheduled for Thursday 17th March we are planning a third visit on Thursday 24th March at 2am.

We are planning to replace our core network switches in Telehouse early Thursday morning. This is part of our network upgrades to increase our capacity. The work is being carried out in the early hours of Thursday to minimise impact on customers.

We do expect Internet connectivity to be affected temporarily as we move cables around. We aim to minimise disruption but there will be times when broadband customers will lose PPP and be offline for a few minutes. Other customers and wholesalers who we provide connectivity to will notice that BGP sessions will be taken down and brought back up one by one. Staff will be on IRC during this time and this status post will also be updated.

The work is expected to start from around 2am and will last approximately 4 hours.

We do apologise for the short notice of this work. The date it was dependent on some other factors outside of our control so it was difficult to give a longer notice period.

Update
23 Mar 20:49:16
Just a note to say that we expect the most disruptive period for customers will be between 2am and 2:30am. We will post further updates here though.
Update
24 Mar 02:00:02
Work is underway... Next update by 2:30am.
Update
24 Mar 02:00:10
Work is underway... Next update by 2:30am.
Update
24 Mar 02:45:02
All cables have been moved, we are reviewing the status of the network.
Update
24 Mar 02:52:41
The network is not yet stable, next update at 3am.
Update
24 Mar 03:32:37
We're still working on this, please to bare with us.
Update
24 Mar 04:02:48
Lines are starting to log back in and we are reviewing the network. Next update at around 4:30am
Update
24 Mar 04:30:37
DSL lines are logging back in and we are tracking down some routing problems at the moment. Next update at 5am
Update
24 Mar 05:05:31
Routing is looking better, we are still going through reviewing the network though.
Update
24 Mar 05:15:19
**Customers who are still off line, eg no PPP, should try rebooting their modem/router so as to force a reconnect.**
Update
24 Mar 05:34:02
There are a few remaining routes that we are working on fixing. Next update at 5:45am
Update
24 Mar 05:55:23
Most things are now back to normal.
Update
24 Mar 07:44:46
Our phone lines will be open from 8am as usual so do phone in if your line has still not come back up.
Resolution The disruptive work took a lot longer than expected, and we will be reviewing this, but I can confirm the switch change has been completed and we have our 80Gb/s link across London now. This does not yet fix the capacity issues, but allows them to be fixed. We need to review exactly where we are with the switches and the fibre link over the next few days before we can move to the next step. Thankfully this should not involve lots of staff in the datacentre in the middle of the night next time. Well done everyone. Thanks to all of our customers for your patience.
Started 24 Mar 02:00:00 by AA Staff
Closed 24 Mar 06:00:00

21 Mar 10:41:26
[Broadband and Ethernet] - Router upgrades - Completed
Details
20 Mar 15:33:14
There are a few router upgrades expected during Sunday - these should have no impact on service but routing may change slightly during the work.
Resolution Several routers were upgraded without any noticeable issues. Thank you for your patience.
Started 20 Mar
Closed 21 Mar 10:41:26
Previously expected 21 Mar

17 Mar 18:46:51
[Broadband and Ethernet] - Rebalancing lines - Completed
Details
17 Mar 18:34:32
We are clearing some lines on some LNSs - causing a PPP reconnect - to rebalance load after this mornings issues.
Resolution Rebalance completed
Started 17 Mar 18:33:54
Closed 17 Mar 18:46:51
Previously expected 17 Mar 19:00:00

17 Mar 08:27:18
Details
17 Mar 06:02:05
Planned work has finished but Talk Talk lines have not yet come back up. This is being worked on by engineers now. This seems to be coincidence and is a major issue impacting TT backhaul and stopping TT lines connect over all this wholesale customers, not just us.
Update
17 Mar 06:15:47
We are in contact with Talk Talk - it looks a lot like their RADIUS servers stopped talking to us during the work this morning and need some sort of a kick from their end.
Update
17 Mar 06:28:59
There will be a blip on half of our BT lines now - we are trying to eliminate anything in our network as the cause of the Talk Talk issue.
Update
17 Mar 06:39:17
As expected, that did not help the Talk Talk issue - we are waiting on them now.
Update
17 Mar 06:43:36
We have senior technical staff at Talk Talk working on this now.
Update
17 Mar 06:52:33
Talk Talk are expecting this to be sorted in under 20 minutes.
Update
17 Mar 07:06:30
During the planned work, both BT and TT lines went down at around 03:30 this morning. This was not quite as we had planned, as we expected lines to blip only for a few seconds. In fact the BT lines were down for over half an hour. However, the TT lines never came back. The planned work was actually fully reverted and that was completed by 6am but still TT lines did not come back. So this is looking very much like something in Talk Talk that is not right. We are hopeful that this will be resolved very soon but at this stage we are waiting for TT to fix their end.
Update
17 Mar 07:10:33
...
Update
17 Mar 07:26:41
Talk Talk are working on this still. Just to be clear, this is only the TT back-haul lines. All BT back-haul lines should be fine. As ever, we have seen routers which can "sulk" so always worth "turning it off and back on again" before calling/irc us if you have a line on BT back-haul which is down.
Update
17 Mar 07:30:00
This is only *our* TT lines, not other ISPs, just to be clear. We think it is a matter of TT's RADIUS servers deciding that our RADIUS servers have stopped responding (due to the planned work) and sulking. We think that they need some manual intervention to start talking to us again. We will be trying to confirm this is the case with Talk Talk once this is resolved and see if this operating can be changed in future as this is clearly unacceptable.
Update
17 Mar 07:33:23
Thank you all for your patience on this - we are very much at the mercy of Talk Talk here, and have been in constant contact with their engineers to try and get this sorted. It should be simple, but seems to be taking rather a long time.
Update
17 Mar 07:53:18
Talk talk agree it is definitely something wrong there end and are still working on it.
Update
17 Mar 07:56:36
Talk talk have found the issue and working in a fix now. Should be very soon now.
Update
17 Mar 08:05:14
Latest update from TT is: "Got it. Working on fix now. Our issue"
Update
17 Mar 08:11:35
Latest update. This looks like a coincidence and is actually impacting all TT wholesale lines and stopping them connect. For most ISPs customers stay on line but as we had planned work it has meant all of our TT backhaul lines are down and simply stayed down. TT are working on it.
Update
17 Mar 08:12:49
TT have confirmed that this was not in any way caused by our planned work, just really bad timing.
Update
17 Mar 08:13:55
Unfortunately as we have backed out all of the planned work we will have to schedule it again and send engineers in to London in the middle of the night. We'll announce more on that once this is sorted.
Update
17 Mar 08:23:24
Lines are coming back on now
Resolution TT have fixed the issue. We will be asking for more details of the cause. Thank you all for your patience.
Started 17 Mar 06:00:00
Closed 17 Mar 08:27:18

17 Mar 11:00:00
Details
14 Mar 14:30:46

This is a followup to https://aastatus.net/2222

As we were not able to complete the planned work that was scheduled for Sunday 13th March we are planning a second visit on Thursday 17th March at 2am.

We are planning to replace our core network switches in Telehouse early Thursday morning. This is part of our network upgrades to increase our capacity. The work is being carried out in the early hours of Thursday to minimise impact on customers.

We do expect Internet connectivity to be affected temporarily as we move cables around. We aim to minimise disruption but there will be times when broadband customers will lose PPP and be offline for a few minutes. Other customers and wholesalers who we provide connectivity to will notice that BGP sessions will be taken down and brought back up one by one. Staff will be on IRC during this time and this status post will also be updated.

The work is expected to start from around 2am and will last approximately 4 hours.

We do apologise for the short notice of this work. The date it was dependent on some other factors outside of our control so it was difficult to give a longer notice period.

Update
17 Mar 02:14:38
This work is about to commence...
Update
17 Mar 05:43:16
Talktalk lines are offline at the moment, we're working in bringing them back at the moment.
Update
17 Mar 06:03:34
Info about TT lines being down here: https://aastatus.net/2229
Update
17 Mar 06:48:57
We are still working on getting TalkTalk lines back, BT lines should be online - of not please do try a reboot of your router.
Resolution New planned work for this on: https://aastatus.net/2231
Started 17 Mar 02:00:00 by AA Staff
Closed 17 Mar 11:00:00

15 Mar 14:18:05
Details
15 Mar 14:18:05
Good news! Starting from 2nd April our office will be open on Saturdays.

We will have Technical Support staff available between 10am and 3pm each Saturday. Customers will be able to contact us in all the usual ways: Phone, email, IRC etc.

Whilst the staff are in the technical support department they will be able to answer general sales and accounts queries too.

In summary, from April, our opening times will be:
Mon-Fri 8am to 6pm
Sat 10am to 3pm
Started 15 Mar 14:00:00 by AA Staff

8 Mar 18:37:28
[Maidenhead Colocation] - Connectivity issues to MH - Closed
Details
8 Mar 18:34:43
We are seeing some odd routing issues and trying to resolve these now. They impact traffic to Maidenhead data centre so VoIP and some hosted services.
Update
8 Mar 19:08:13
We are actually trying to find the cause, so may be some slight disruption again this evening.
Update
8 Mar 19:08:19
We are actually trying to find the cause, so may be some slight disruption again this evening.
Resolution We can see the issue and should have it resolved any moment. Looks like an issue with some new routing code, which will be investigated.
Started 8 Mar 18:20:00
Closed 8 Mar 18:37:28

13 Mar 06:00:00
Details
7 Mar 22:38:04
We are planning to replace our core network switches in Telehouse early Sunday morning. This is part of our network upgrades to increase our capacity. The work is being carried out in the early hours of Sunday to minimise impact on customers.

We do expect Internet connectivity to be affected temporarily as we move cables around. We aim to minimise disruption but there will be times when broadband customers will lose PPP and be offline for a few minutes. Other customers and wholesalers who we provide connectivity to will notice that BGP sessions will be taken down and brought back up one by one. Staff will be on IRC during this time and this status post will also be updated.

The work is expected to start from around 2am and will last approximately 4 hours.

We do apologise for the short notice of this work. The date it was dependent on some other factors outside of our control so it was difficult to give a longer notice period.

Update
13 Mar 02:09:21
This work is underway
Update
13 Mar 03:26:19
We're still working on preparation work and have not yet started to cause any disruption. We'll update again shortly.
Update
13 Mar 04:19:49
Things will start to get disruptive as we move cables around. Expect PPP to drop and reconnect a few times and to have intermittent routing problems over the next 2 hours.
Update
13 Mar 09:41:33
Overnight work did cause DSL lines to drop and reconnect. We have watched as lines have reconnected and it is looking like customers pretty much all logged back in OK. If you are still without connectivity then the first thing to try is to reboot the modem or router which will force a reconnect.
Resolution This planned work period has ended. Sadly, not all the work has been completed and we will need to plan for another date in the near future. See: https://aastatus.net/2227
Started 13 Mar 02:00:00
Closed 13 Mar 06:00:00

10 Mar 20:33:00
Details
10 Mar 20:41:32
A few lines dropped PPP and other customers had routing problems for a couple of minutes. This was due to two of our routers restarting unexpectedly. Staff are looking in to why this happened.
Started 10 Mar 20:29:00
Closed 10 Mar 20:33:00

10 Mar 14:45:00
[London Data Centres] - Preparation work - Completed
Details
8 Mar 14:05:36
In preparation for the work we have planned for Sunday morning, staff will be visiting Telehouse to do some minor work. We do not anticipate any disruption to customers as a result of this.
Resolution This work has been completed.
Started 10 Mar 11:00:00 by AA Staff
Closed 10 Mar 14:45:00

4 Mar 17:12:05
[Broadband and Ethernet] - Billing for Terabyte services - Info
Details
4 Mar 17:10:32

Customers with Home::1 VDSL Terabyte services will notice a slight change to billing.

Instead of one line item at £60 for Home::1 VDSL Terabyte, you will now see "Home::1 Terabyte" at £40, and the "VDSL and copper pair" at £20. The total is the same.

SoHo::1 VDSL Terabyte customers will see a similarly split pricing.

This is, in part, to allow for launch of Home::1 ADSL Terabyte services soon.

Started 4 Mar 17:08:42