Order posts by limited to posts

13 May 20:54:24
13 May 21:45:49
Our upstream carriers have advised us of scheduled maintenance taking place between 02:00 - 03:00 on Friday 20th May 2016 affecting Data SIMs only.
Some customers may notice a momentary drop in connections during this time as any Data SIMs using that route will disconnect when the link is shut down. Any affected Data SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, Data SIM connectivity should be considered at risk throughout.
Voice SIMs are not affected by this maintenance.
Started 13 May 20:54:24
Update was expected 20 May 11:00:00

5 Feb 16:32:23
5 Feb 16:32:23
Some call recordings from today have been lost. We've identified the problem and have worked around it until we can fix it. We should be able to fix it with no customer impact. If you are expecting a call recording that you have not received, support can check whether the call was affected by this problem. The call recordings lost are in the minority and most calls were recorded as expected. This problem is unrelated to yesterday's problems with lack of ring tone!
Started 5 Feb 16:25:24

12 Jun 2015 11:06:49
12 Jun 2015 11:00:23
Our office connectivity blipped. We have internet again, but our phones are down. We're investigating. This is not customer affecting.
12 Jun 2015 11:06:49
Phones are working again.
Previously expected 12 Jun 2015 14:57:07

13 Apr 2015
09 Apr 2015 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr 2015

12 Mar 2015 09:48:01
12 Mar 2015 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.

08 Jan 2015 12:39:06
08 Jan 2015 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
09 Jan 2015 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
12 Jan 2015 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 08 Jan 2015 12:39:06 by AA Staff
Previously expected 15 Jan 2015 17:00:00

03 Jun 2014 17:00:00
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

9 May 16:57:03
Affected systems: Outbound SMS Only Affected customers: All customers sending SMS Expected Resolution Time: Scheduled Maintenance 07:00 to 09:00 BST on Tuesday 17th May 2016

As part of our carrier's ongoing work to upgrade their network and services, they will be performing an upgrade to internal SMS API. There will be a short delay of up to 30 seconds whilst they switch over to the new version. We don't expect any further disruption through these works however as with any works, there is always a risk element and so we want to keep you informed.

Update was expected 17 May 10:00:00
Previously expected 17 May 09:00:00 (Last Estimated Resolution Time from Carrier)

06 Jul 2015 12:49:42
We have been advised by Three of works on their network (22:30 8th July to 04:50 9th July 2015) Which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some partners may experience dropped connections. Any device reconnecting will automatically be routed via a new path. We apologise for any inconvenience this may cause and the short notice of this advisory.

12 Feb 2015 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

27 May 20:15:12
27 May 18:59:26

A number of customers were emailed that their 1st of month DD was cancelled.

This process was stopped as soon as we realised what was happening so not many people will be affected. You will receive normal DD collection notices in due course for your monthly bills or have collection within 3 working days as per exiting notice (so may be 3rd instead of 1st) as per DD rules.

Our DD management system is coded to be as cautious and customer focused as possible and a minor error meant it was not happy with the DDs for regular collections being in place before the invoice was issued. This has been corrected now.

Sorry for any confusion caused.

Started 27 May 18:00:00
Previously expected 27 May 22:56:44

27 May 04:30:00
27 May 02:24:53
At around 1:51 today TT dropped offline. We are investigating.
27 May 02:56:38
We've seen reports of other ISPs lines via TT also having the same problem.
27 May 03:16:06
This looks very much like a major outage within TalkTalk. Our BGP session to them that carry our traffic is down at one datacentre and up and our second datacentre but with no prefixes being advertised to us from TalkTalks side.
27 May 03:58:05
We are seeing some lines starting to reconnect.
27 May 03:58:26
We are seeing some lines starting to reconnect.
27 May 04:01:19
No word from TT as to the cause of this, but we'll update this post in due course.
27 May 04:14:03
Majority of lines are back on now - waiting for the last ones to reconnect.
Resolution It looks like this was caused by some TT planned work of which we were unaware, and clearly did not go to plan. We are trying to get a full explanation from TT. In the mean time the problem is resolved. If you are off line, try restarting your router to force it to reconnect.
Started 27 May 01:51:00
Closed 27 May 04:30:00

19 May 21:55:00
20 May 09:03:45
On the evening of 19th May a small number of lines all in the Guildford area (GU postcodes) were offline between around 21:35 and 21:55. This only affected TalkTalk connected circuits. We'll investigate the cause of this with TalkTalk and will update this post shortly.
Resolution TalkTalk have confirmed that they did have a major network outage. This was resolved by TalkTalk rebooting one of their main BRAS aggregation devices. They do apologise and have raised the problem with their vendor.
Started 19 May 21:35:00
Closed 19 May 21:55:00
Cause TT

14 May 17:36:17
14 May 17:36:17

I have added some more features to card printing.


It now included linear barcodes including code 39, code 128, EAN13, UPC, EAN8, EAN5 and ITF

Started 14 May 17:34:55

12 May 17:00:28
12 May 16:58:38

We use IRC all the time here at AAISP, both internally between staff teams and in helping our customers. Many customers have their own IRC client setup on their computer whilst others pop in to the IRC channel by using the web-based version we provide via our website.

To help improve the experience for customers using our web-based irc client we have a new one to try:


We'll be updating our links on the website to the new one next week, but feel free to try out the new client.
Started 12 May 16:50:00

5 May 16:52:53
5 May 14:42:29
We've seen several incidents, at 12:25, 12:33 and 14:26 where we lost connectivity over all of the VLANs we currently utilise on our Datahop circuits in Telehouse North. This has affected a few of our wholesalers who connect in to us over Datahop VLANs. This looks like it's relating to a fault within datahop. The link is up at the moment.
5 May 16:25:36
We've had an update from datahop saying that they have replaced a switch in their network. Service has remained stable since 14:30 this afternoon.
5 May 16:54:00
Datahop have completed their switch replacement work. This incident is now closed.
Resolution Faulty switch within Datahop has been replaced.
Started 5 May 12:30:00
Closed 5 May 16:52:53

28 Apr 16:18:20
28 Apr 15:30:05
We are seeing a number of broadband lines in the south of London suffering from severe packet loss and high latency. This looks like a fibre problem with TalkTalk. This has been reported and as soon as we have an update we will let you know.
28 Apr 15:36:41
Here is an example graph
28 Apr 16:18:44
TalkTalk have fixed this - that was quick.....
Started 28 Apr 15:27:37
Closed 28 Apr 16:18:20
Previously expected 28 Apr 19:27:37

2 May 17:36:35
[Broadband and Ethernet] - At risk: Router upgrades - Completed
28 Apr 09:29:41
We may do some router upgrades and changes over the weekend as part of ongoing network tidy-up. This should not be disruptive.
30 Apr 08:52:48
The first changes have been done, with most likely impact (if any) being on our office connections. If all goes well today we'll do more changes tomorrow (Sunday) morning. As I say, these should have no noticeable impact, but are to improve some of our internal routing logic.
1 May 05:47:14
The next stage was done this morning. Any issues please let us know on irc. We have more changes later today (Sunday).
1 May 09:53:27
Some further work was done before 8am today (Sunday). We plan to do the next stage tomorrow (Monday) morning. So far all seems to be going to plan.
2 May 11:01:20
One small causality was our NAT64 gateway, which is being restored now.
2 May 16:51:30
We have identified a possible peering MTU issue and are investigating. There may be more work this evening (Monday). Again, we expect no disruption, but a risk.
2 May 17:18:19
Yes, we have identified an issue with MTU on some peers, and this is being rectified now.
2 May 17:37:08
MTU fixes now applied
Resolution The main planned work was all completed before 8am this morning, all looks well. Any issues contact support or mention on irc or text the MSO number. We'll be keeping an eye on things today anyway.
Started 30 Apr
Closed 2 May 17:36:35
Previously expected 3 May

28 Apr 04:55:55
27 Apr 11:21:25
We are making a correction to routing. This will impact external connectivity for a few seconds but should not cause lines to drop PPP. We aim to do this over night or early tomorrow morning. Sorry for any inconvenience.
Resolution Changes complete
Started 28 Apr
Closed 28 Apr 04:55:55
Previously expected 28 Apr 08:00:00

27 Apr 06:09:11
[Broadband and Ethernet] - At risk: Routing changes - Completed
26 Apr 16:58:57
We are tidying up some of our internal routing logic following from our major network upgrades recently. This is expected to have no impact, but there is a risk it could impact Ethernet and broadband customers and maybe other services. The plan is to get this all done well before 8am so that we can sort any issues that come up or reverse changes that are made.
Resolution The work is complete and all looks well - any issues, please let us know. P.S. an issue with BGP connections via DSL was resolved.
Started 27 Apr
Closed 27 Apr 06:09:11
Previously expected 27 Apr 09:00:00

26 Apr 17:00:00
26 Apr 16:40:46

Some Ethernet customers have seen a routing blip today. This is mainly those that have a secondary block of IPs routed via your Ethernet link and related to that additional block.

We should have posted an "at risk" notice about some very minor routing changes tidying up after all of the recent work in our core network. I genuinely did not expect any impact - we have done simple routing changes many times in the past without experiencing any issues.

Unfortunately the changes uncovered an underlying problem with the redundancy logic, and hence something that should have had no impact did in fact have an impact. This was fixed quickly within minutes of the report.

There is other work we expect to do tomorrow morning before 9am, including sorting the underlying issue we have uncovered today. If there are any issues you encounter with routing in the morning, please do contact support.

Sorry for any inconvenience caused.

Started 26 Apr 13:00:00
Closed 26 Apr 17:00:00
Previously expected 26 Apr 17:00:00

19 Apr 12:42:22
[General] - DRIPA - Info
16 Jul 2014 10:13:15
Just to be clear on our policy here - when DRIPA comes in to force, and if A&A become subject to a retention notice for all customers, we aim to work on all practical legal means to minimise the amount of data retained under that legislation - making full use of the bad wording in the Schedule in the 2009 regulations where possible. We also aim to clearly publish what is retained under such a notice and what steps we have taken to minimise such data. Such steps may mean separate companies running email or other services, or even hosting some servers outside the UK, if those are practical steps we can take. Why? Because blanket mass surveillance is illegal under EU law as it is against our basic human right to privacy as decided by a court, that's why! Feel free to comment on my blog.
Started 15 Jul 2014

17 Apr 19:33:54
14 Apr 16:22:27

As customers may recall, some time ago we ported numbers in several exchange area codes to a company called Incoming Numbers Ltd, and then leased use of those numbers back to A&A. The effect was no technical change to operation, but it meant the numbers were technically ported to another provider. When you ordered numbers in those areas, the order form confirmed you were porting to INL. The bills showed the numbers were ported to INL. This is only a small number of area codes.

We're pleased to confirm that we have today reversed that and the ported numbers are now back with us, Andrews & Arnold Ltd. So the bill will no longer show the note about Incoming Numbers Ltd.

The reason for this was that OFCOM started charging for numbers, after having provided them for no charge, and waited for us to start using the blocks. Something that in any other "business" would be an unacceptable business practice, in my view. There was a discount for ported out numbers, hence the porting.

The plan worked, and we got the credit for ported out numbers off the bill. However, we found a much more effective solution. Most of our numbers are allocated to Andrews & Arnold (numbers) Ltd, a company which is not, itself, any more, a telco, and just leases the numbers to Andrews & Arnold Ltd to use. As it is not a telco, OFCOM do not charge it, and so problem solved.

Sorry for the puzzlement this may have caused some customers in the past. It caused me some paperwork as INL is a proper separate trading company with accounts, and everything, but not an A&A subsidiary. It will be wound up or made dormant for now.

I hope both of these methods, to avoid being charged for something we already had, have been helpful to OFCOM as they said the charging was an "experiment", and it was clearly our duty to provide the best data to them on how well it worked as an experiment.

Started 14 Apr 16:13:50
Previously expected 14 Apr 20:13:50

17 Apr 19:33:27
[Broadband and Ethernet] - Home::1 Terabyte bonus - Info
4 Apr 20:26:22

Today we tweeted, somewhat as a joke, "Customers on Home::1 TB services downloading the Panama papers, let us know when you hit your 1TB quota and we'll topup 4 free #PanamaLeaks"

This was mainly because this was something, at 2.7TB, that was actually too big for our monthly allowance on the terabyte service. It shows that even 1TB is not enough for that amount of data, but it really is one of the few things that could cause you to download so much.

However, I also appreciate that we have had had some issues for some weeks now, and they are finally coming to an end with much needed capacity upgrades.

So, as a thank you for the customers that have embraced the new Home::1 Terabyte tariffs, I have actually added 3TB to this month's quota (as top-up, so carrying to end of next month). Thus honouring the tweet for all.

Thank you all - without customers we would be nothing.

Started 4 Apr 20:21:04
Previously expected 5 Apr 00:21:04

17 Apr 19:32:58
[Broadband and Ethernet] - LNS roll overs - Info
14 Apr 08:29:19

I am pleased to confirm that as we now have many more LNSs in service we can return to the normal LNS roll-over strategy http://aa.net.uk/kb-broadband-duallns.html

We moved B.gormless to A.gormless last night, and would normally do one per night to move through all of the LNSs. However, this time we are doing some hardware reconfiguration of LNSs as well, so the next roll-over (C.gormless to B.gormless) will be some time next week.

The roll-over went smoothly as usual. Do remember that you can configure a preferred time in the control pages, as we know some people are up late working or gaming, and others are up early, and we want to minimise any disruption to your connection by routine maintenance like this.

Started 14 Apr 01:00:00
Previously expected 17 Apr 23:32:48

14 Apr 16:27:06
01 Dec 2015 10:00:08
We are starting to use a new address for the Control Pages. Instead of https://clueless.aa.net.uk it'll be https://control.aa.net.uk. The reasons for this are two fold:
  1. Control.aa.net.uk is a better description of the service than clueless.aa.net.uk - some customers have been a little confused or wary of the name 'clueless'.
  2. The new name has given us the opportunity to update the SSL (security) settings.
The slight downside of this is that older clients may not have the modern ciphers needed. For example, the minimum browser versions would be:
  • Firefox 27
  • Chrome 22
  • IE 11
  • Opera 14
  • Safari 7
  • Android 4.4
  • Java 8

We would recommend that customers update their browsers if they are using the above.

As always, we're happy to receive feedback.

Resolution Launched
Started 01 Dec 2015 09:04:43

14 Apr 16:26:42
21 Dec 2015 08:46:32
Please see our most recent newsletter: aa.net.uk/news-2016-01.html
Started 21 Dec 2015 08:45:00

14 Apr 16:25:00
02 Aug 2014 12:02:18
We are planning an expansion of our VoIP services which will require the use of more IP addresses than we have told customers to allow through their firewalls, so have added additional IP ranges to the list. We are not using these new IP ranges yet, but are giving advanced notice of the change to give customers time to update their firewall rules. The list is at http://wiki.aa.org.uk/VoIP_Firewall
Started 02 Aug 2014 11:50:35
Previously expected 02 Aug 2014 11:50:35

14 Apr 16:24:41
02 Jul 2015 10:44:33
Some customers have advised that they are unable to call some numbers which they were able to call last week. This is due to new rules which mandate that we charge a service charge set by the range holder plus our access fee (2p/min) for such numbers. OFCOM have failed to provide means for telcos to determine the service charge that applies. Their only suggestion was to use the Bt Carrier Price list. We have loaded this (and are doing so every day) but it seems to be lacking a number for 084 and 087 number ranges. Until those appear in the BT Carrier Price list, or OFCOM actually come up with a means for telcos to comply with the new regulations, the numbers cannot be called as we have no way to comply with OFCOM rules. Feel free to complain to OFCOM over the stupidity of these new rules.
07 Jul 2015 11:14:47
We have managed a one-off load of current codes, but there still remains no official source of the codes and rates due to OFCOM's failure to define who is responsible for compiling and maintaining this information. As such there is still no sensible way to ensure this data remains correct and complete.
Started 01 Jul 2015
Previously expected 01 Aug 2015

14 Apr 16:24:06
[VoIP and SIMs] - 999 location data - Info
12 Mar 2015 18:24:06
We allow users to provide address/location data to pass to emergency services in the event of a 999 calls. This goes via BT and BT are being a pain at present. As we have explained the data is simply a matter of what customers provide. VoIP has no inherent way to provide an accurate geographic location of callers. But BT are unhappy with some of the data and seem to have blocked us sending updates this week. They even suggested that they have to send OFCOM reports of incomplete and incorrect data as some sort of threat. We know a lot of people have multiple locations from which they make calls. We do this purely as a good will gesture and in the spirit of GC4, because we don't have location data in the first place. So if BT do not stop buggering about soon we'll be reporting BT to OFCOM instead. Anyway, the upshot is that a few customers that have provided location data over that last few days are not yet updated to BT and hence the emergency services. We are working to resolve this and hope to have it working again soon. The config pages for your VoIP set up show if there is an issue and you get an email once it is all confirmed. In the mean time, we do apologise for any concern this may cause. None of our services are provided to be used in any "safety of life" circumstances, as per our standard terms.
16 Mar 2015 16:49:34
We are hoping to have this sorted, perhaps today. It is likely that anyone with existing data registered may get a new email at some point confirming the data as we are planning to re-send everything from scratch just to be sure. We would ask that customers review what they have entered and ensure it is as accurate as possible to assist emergency services in the event of a 999 call.
18 Mar 2015 17:40:37
Data is updating again to BT. We would recommend customers ensure accurate data on the control pages for 999 calls. We will review this over the next few days, and customers may still get update emails in due course.
Started 08 Mar 2015
Previously expected 16 Mar 2015

11 Apr 12:49:31
11 Apr 11:56:02
We are investigating an issue with one of our call servers (A.Voiceless). DNS has been updated to route via B, which should ensure service is restored for those that are affected but it is worth forcing phones to re-register if you are having issues (some VoIP devices do not cache DNS properly).
11 Apr 12:09:53
Engineers on on their way to the datacentre. Calls and registrations are all being handled via our 'B' call server.
11 Apr 12:34:29
Engineers are onsite.
11 Apr 12:38:00
Hardware is being replaced.
11 Apr 12:44:39
Hardware replaced, and server online.
Resolution The faulty server has been repaired and back on line.
VoIP and SIMs Users Affected 50%
Started 11 Apr 11:50:00
Closed 11 Apr 12:49:31
Previously expected 11 Apr 12:50:00

4 Apr 15:31:58
[Broadband and Ethernet] - Bogus uplink speeds from BT - Closed
4 Apr 14:44:27
Since our software update on our LNSs on Sunday we are now able to pick up the uplink speeds for DSL lines from the carrier.

However, some customers are seeing slow upload speeds and also a red line on the bottom of our monitoring graphs indicating the link is full. The problem is that BT are sending bogus uplink speeds for some lines, and therefore we are limiting the throughput.

We are working on a fix for this now.

4 Apr 15:32:16
This has now been resolved.
Started 4 Apr 14:42:19
Closed 4 Apr 15:31:58

17 Apr 19:32:20
[Broadband and Ethernet] - At risk: Routing changes - Completed
3 Apr 17:44:24
We have a number of things to do over the coming few weeks with changes to the peering and transit links we use. This should not impact customers directly, and is some minor routing changes, but it is the final step in managing the capacity upgrades. So there is a small risk of some side effects. We'll aim to keep these outside office hours as much as possible.
4 Apr 18:13:56
We have sorted one of the links to LONAP today, but later in the week we will be moving more transit and peering links, step by step, carefully. This should not impact service, but will allow more capacity and future proofing.
13 Apr 10:37:38
The next stage of work is scheduled for Wednesday 20th, again we don't expect any customers to be affected by this as it is minor configuration changes at our side.
Started 4 Apr
Closed 17 Apr 19:32:20

3 Apr 16:36:34
[Broadband and Ethernet] - L2TP direct service - Info
3 Apr 16:36:34

Now that we have the new network upgrade we are hoping to properly launch the new L2TP specific service. Previously this was a somewhat informal service with somewhat complicated billing.

This new service is an L2TP over IP service for people using other broadband lines to access us and use our IPs and services including fixed IPv4 and IPv6 blocks. The plan is £10/month for a service capped at 100Mb/s with 1TB monthly allowance, clamping to 3Mb/s from next month's allowance on reaching allowance. Very similar to Home::1. No date yet but we are hoping to make this our standard L2TP service some time in April.

17 Apr 19:34:56
I'm sorry this is taking so long to sort - it is basically a tariff change, and order form, but we have been very keen to ensure the infrastructure is in place for this. I still hope to do this by end of month.
12 May 11:09:15
Information: http://aa.net.uk/broadband-l2tp.html
Order form: https://order.aa.net.uk/chaos2l2tp.html
Started 1 Apr
Previously expected 1 May

3 Apr 14:54:34
3 Apr 14:46:09
It looks like there was an issue with one of the ports on one of the LNSs for a while this afternoon. We are looking in to it, but it could be some sort of attack. Obviously this is never good, but this appears to be limited to half of one LNS.
3 Apr 14:54:49
This does look like a targeted attack
3 Apr 15:59:02
It looks like, in addition to the specific port on the LNS being impacted, there may have been some more general issues for routes on one of our peering points (via which the attack came in).
Resolution It seems to have subsided now. We are trying to investigate more on this, but it does not apparent to be related to the network restructure this morning, thankfully. This customer that was the target has been contacted.
Broadband and Ethernet Users Affected 8%
Started 3 Apr 14:30:00
Closed 3 Apr 14:54:34

3 Apr 08:00:00
27 Mar 21:17:18
We are planning, at the very least, to upgrade the live LNSs all in one go. However, we are hopeful that we can also upgrade the routed links to both BT and Talk Talk at the same time. This should "just work" but if it does not we will roll back. The effect will be at least one, and possibly more PPP level restarts of all broadband lines and may be a few minutes of outage. If we are successful we will be able to deploy more active LNSs in the following week to spread load and allow more expansion. It will also allow us to more easily expand the number of links we have to carriers over the coming months. This is all part of a major ongoing network capacity upgrade.
31 Mar 18:07:55
We have done a lot of work today - both in the data centre (deploying five more FB6000 series FireBricks) and from the office configuring systems ready for the weekend. Well done to all of the staff involved as I do not believe there was a single blip. I feel very confident that we are ready to the work on Sunday now.
2 Apr 07:49:05
More preliminary testing yesterday and today but looking like we are on schedule for Sunday morning. We aim to upgrade all LNSs, change carrier interlink so we can use more LNSs and actually start using more LNSs.
3 Apr 02:03:56
The work begins...
3 Apr 02:10:26
LNSs are having their software updated, this will cause PPP blips for customers.
3 Apr 02:22:29
Software has been updated on the LNSs. Stage two will start shortly, this is reconfiguring our links to BT and TT backhaul.
3 Apr 03:15:58
We're about to start reconfiguring the TT interconnects. This will cause a PPP blip for customers on T backhaul.
3 Apr 03:30:34
TT lines have just blipped as we reconfigured the backhaul links. Lines are logging back in now.
3 Apr 03:49:37
TT is working but we are making sure the balancing works before we proceed to the BT side
3 Apr 04:08:26
We're about to start reconfiguring the BT interconnects. This will cause a PPP blip for customers on BT backhaul.
3 Apr 04:24:39
BT lines blipped their PPP as expected, and are logging back in now.
3 Apr 04:32:52
BT lines looking good.
3 Apr 04:41:02
All lines are about to have a short PPP blip as we load new software across the LNSs.
3 Apr 04:52:13
We are not balancing the BT back haul links properly at the moment, this is being worked on. This is not affecting service at the moment but does mean we are only running on half our links to BT.
3 Apr 05:13:15
We are close, but we do have to break things once more sadly, there will be some routing and PPP drop over the next 30 minutes.
3 Apr 05:39:37
Lines are reconnecting and routes are being restored We'll then review where we are and will update again shortly.
3 Apr 05:56:12
Customers are online, but we are still working on balancing the BT links.
3 Apr 06:04:00
We are over running on the window for this planned work. However, customers are online, and we are working on some oddities in how the traffic over our BT links is being balanced.
3 Apr 06:54:37
Things look stable and we have managed to balance the traffic properly we are still tracking down one issue this morning. There may be some re-balancing of traffic between LNSs later.
Resolution It took a long time, but the work has been completed, including the deployment of 3 new LNSs to our pool. Thanks to the staff involved, and to our customers for their patience.

Some further info about this work here: http://www.revk.uk/2016/04/growing-pains-pain-relief.html

Started 3 Apr 02:00:00
Closed 3 Apr 08:00:00
Previously expected 3 Apr 08:00:00

2 Apr 10:17:00
[Broadband and Ethernet] - Recursive DNS resolver issue - Closed
2 Apr 10:21:16
There was an unexpected issue with DNS resolvers not responding due to a routing loop. This has now been rectified. Thanks to all those that contacted us.
Started 2 Apr 09:48:00
Closed 2 Apr 10:17:00

30 Mar 09:16:58
6 Mar 13:25:01

Following the recent launch of our special Home::1 VDSL Terabyte package we are planning a new ADSL version.

Home::1 ADSL Terabyte will be just like normal Home::1 ADSL services but offering 1TB download allowance per month, then slowing to 3Mb/s if you hit the limit. No top-up or excess charges.

The service will be £40/month including VAT, but not including the phone line. You will need a phone line with someone, or we can provide a broadband only line for £10/month as usual.

The service is only available on TalkTalk back-haul, so not available everywhere. Anyone on BT back-haul will have 14 day lead time and a short outage on the line to regrade. Anyone on TalkTalk back-haul will just have this as a normal tariff change applied from the next bill date. Either way, no cost to change to this. The install price as a new service is the same as normal Home::1

At present we are waiting to resolve the capacity issues with Talk Talk back-haul across our core network before we launch the new package, so it should be available later this month.

29 Mar 20:20:46
Expected to launch tomorrow - just confirming the new capacity upgrades work as expected tonight.
Resolution Now launched. http://aa.net.uk/broadband-home1-terabyte.html
Started 6 Mar 13:42:35
Closed 30 Mar 09:16:58
Previously expected 1 Apr

29 Mar 20:08:55
4 Mar 13:41:26

Over the past week there have been times in the evenings where we have seen congestion on some of our TalkTalk ADSL and VDSL circuits. This has shown up in our CQM graphs as latency spikes and/or packet loss.

We would like to reassure customers that this is *not* congestion in TalkTalk's network; it is actually happening inside our own network. We are aware of the cause and are taking actions to stop it.

We mentioned in the January newsletter that we are planning various upgrades to our London network to increase overall capacity. This work is in progress but for a couple of reasons has been delayed and is taking slightly longer than initially planned.

Whilst the London upgrades have been happening we have been trying to mitigate the congestion problem by re-routing traffic in various ways. This has helped but not been a 100% solution, and has affected service for customers. For this we are sorry; we do aim never to be the "bottleneck" and in this case we've not managed to meet that standard.

Looking forward, we have works planned for the datacentre next week and the following week and we'll update this post with progress along the way.

We're happy to discuss this further, do pop in to our IRC channel or email or phone etc.

6 Mar 13:16:44

This work was planned for last year but issues beyond our control (problems with new fibres) had delayed it. The plans now are for testing of fibres to hopefully be completed during this week (7th-11th March), and if all goes well we can start using the new fibre next weekend. Once this is in place we can move the impacted circuits on to the new fibres during the following week to remove the bottleneck in our core network. There is a lot to do, but the one issue right now is one of the TalkTalk back-haul links and that can be the first thing to be moved over.

Once again we apologise for the time this is taking.

We plan to launch a new Home::1 ADSL Terabyte package as soon as these issues are resolved and the capacity is in place.

7 Mar 22:39:11
We have posted details of planned work for Sunday 13th: https://aastatus.net/2222
14 Mar 20:55:09
Further work at the datacentre has been planned, please see: https://aastatus.net/2227
Resolution Network upgrades are now providing additional capacity for TT lines. Thank you all for your patience.
Started 4 Mar 13:00:00
Closed 29 Mar 20:08:55

27 Mar 09:07:44
27 Mar 04:39:48
Since Thursday morning it seems that many of the Technicolor routers have been rather odd, generating thousands of DHCPv6 requests per second. We have remotely rebooted many of these, but some still remain. We will be taking steps later today (Sunday) to shut down IPv6 on the affected routers and advise customers that the router needs power cycling / rebooting. Sorry for any inconvenience, and we are at a loss as to why they are behaving like this.
Resolution We have managed to sort the problem routers for now (mostly by rebooting) - if you don't have IPv6 working properly on a Technicolor - please turn it off and back on again.
Started 24 Mar
Closed 27 Mar 09:07:44
Previously expected 29 Mar

29 Mar 20:00:29
[Broadband and Ethernet] - TT capacity upgrade - Completed
27 Mar 09:11:44
Now we have the new network switches in place we are able to make changes to address Talk Talk capacity issues. We have a plan for this and it should be possible with no disruption. If we are able to, we will do this over the weekend, but if not, then the plan is to make the changes during the day on Tuesday. We'll post an update here when we start. Sorry for the rather wide window on this - it depends on completing testing of the new fibre links to our satisfaction first.
29 Mar 17:40:10
Some of the preliminary switch config changes caused a port to flap unexpectedly which resulted in a blip of some lines, mostly Talk Talk lines at around 16:30. We are still planning to go ahead with the change to allow more TT capacity though and work is continuing in to the evening.
29 Mar 18:42:56
We are starting the work to re-route one of the Talk Talk links now.
29 Mar 19:43:44
Work has gone well so far and customers are seeing much more capacity to Talk Talk lines. We are monitoring the situation carefully at present.
Resolution We are watching things closely, but looking good. Next step is at the weekend.
Started 27 Mar
Closed 29 Mar 20:00:29
Previously expected 30 Mar

28 Mar
24 Mar 08:03:43
We may do some routing adjustments and s/w upgrades over the weekend. These should be minor and not disrupt normal operations. However, there is always a small risk. Basically there is a bit of tidying up to do after the switch change, and we may be able to move some of the traffic around to make a start on capacity issues at some point during the weekend. At this stage we are not sure, so this is more of an "At risk" notice.
25 Mar 14:14:11
We have updated routers and managed with minimal disruption - rerouting traffic around the upgrades as done. We have also made some config changes. We are still trying to work out if there is a way we can resolve some of the TT capacity issues over the weekend or not, so there may be some more work still.
26 Mar 16:07:26
A number of changes have been done today as well. Again, with minimal disruption.
26 Mar 16:08:14
We also found that all of the Technicolor routers were upset by something when we did the switch change on Thursday. We don't understand what exactly, but rebooting is fixing, so we are rebooting Technicolor routers one by one.
27 Mar 04:35:49
The work over the weekend has gone quite well, with very little disruption. We do now have a clear plan for the changes to alleviate the capacity issues. I don't yet know if this will be possible over the weekend but should be possible on Tuesday.
Started 25 Mar
Closed 28 Mar
Previously expected 29 Mar

24 Mar 23:10:00
24 Mar 23:20:23
We have just resolved a problem where by we were not using all our fibres to BT for traffic. this meant that for the past few hours we were running over 3 links instead of the usual 4 and customers were seeing congestion by way of lower than usual throughput as well as some latency and loss. We do apoloise for this. We are now back to normal.
Started 24 Mar 20:00:00
Closed 24 Mar 23:10:00

24 Mar 06:00:00
18 Mar 16:52:32

This is a follow-up to https://aastatus.net/2222 and https://aastatus.net/2229

As we were not able to complete the planned work that was scheduled for Thursday 17th March we are planning a third visit on Thursday 24th March at 2am.

We are planning to replace our core network switches in Telehouse early Thursday morning. This is part of our network upgrades to increase our capacity. The work is being carried out in the early hours of Thursday to minimise impact on customers.

We do expect Internet connectivity to be affected temporarily as we move cables around. We aim to minimise disruption but there will be times when broadband customers will lose PPP and be offline for a few minutes. Other customers and wholesalers who we provide connectivity to will notice that BGP sessions will be taken down and brought back up one by one. Staff will be on IRC during this time and this status post will also be updated.

The work is expected to start from around 2am and will last approximately 4 hours.

We do apologise for the short notice of this work. The date it was dependent on some other factors outside of our control so it was difficult to give a longer notice period.

23 Mar 20:49:16
Just a note to say that we expect the most disruptive period for customers will be between 2am and 2:30am. We will post further updates here though.
24 Mar 02:00:02
Work is underway... Next update by 2:30am.
24 Mar 02:00:10
Work is underway... Next update by 2:30am.
24 Mar 02:45:02
All cables have been moved, we are reviewing the status of the network.
24 Mar 02:52:41
The network is not yet stable, next update at 3am.
24 Mar 03:32:37
We're still working on this, please to bare with us.
24 Mar 04:02:48
Lines are starting to log back in and we are reviewing the network. Next update at around 4:30am
24 Mar 04:30:37
DSL lines are logging back in and we are tracking down some routing problems at the moment. Next update at 5am
24 Mar 05:05:31
Routing is looking better, we are still going through reviewing the network though.
24 Mar 05:15:19
**Customers who are still off line, eg no PPP, should try rebooting their modem/router so as to force a reconnect.**
24 Mar 05:34:02
There are a few remaining routes that we are working on fixing. Next update at 5:45am
24 Mar 05:55:23
Most things are now back to normal.
24 Mar 07:44:46
Our phone lines will be open from 8am as usual so do phone in if your line has still not come back up.
Resolution The disruptive work took a lot longer than expected, and we will be reviewing this, but I can confirm the switch change has been completed and we have our 80Gb/s link across London now. This does not yet fix the capacity issues, but allows them to be fixed. We need to review exactly where we are with the switches and the fibre link over the next few days before we can move to the next step. Thankfully this should not involve lots of staff in the datacentre in the middle of the night next time. Well done everyone. Thanks to all of our customers for your patience.
Started 24 Mar 02:00:00 by AA Staff
Closed 24 Mar 06:00:00

21 Mar 10:41:26
[Broadband and Ethernet] - Router upgrades - Completed
20 Mar 15:33:14
There are a few router upgrades expected during Sunday - these should have no impact on service but routing may change slightly during the work.
Resolution Several routers were upgraded without any noticeable issues. Thank you for your patience.
Started 20 Mar
Closed 21 Mar 10:41:26
Previously expected 21 Mar

17 Mar 18:46:51
[Broadband and Ethernet] - Rebalancing lines - Completed
17 Mar 18:34:32
We are clearing some lines on some LNSs - causing a PPP reconnect - to rebalance load after this mornings issues.
Resolution Rebalance completed
Started 17 Mar 18:33:54
Closed 17 Mar 18:46:51
Previously expected 17 Mar 19:00:00