Order posts by limited to posts

Sunday 19:42:55
[Broadband and Ethernet] - World of Warcraft IPv6 issues - Open
Sunday 19:42:01

It seems that once again Blizzard have some IPv6 issues that are impacting World of Warcraft.

This should be simple for them to fix. They have connectivity to their network, which seems to be very good (via LINX) with low pings, but the individual servers their end do not seem to be working on the IPv6 addresses the client is trying.

This could simply be a change of IPv6 SLAAC address following hardware changes, or could be something else, we don't know. They are not interested in investigating.

Normally a failure in one IP protocol is not a huge issue as clients fall back to the other version, e.g. trying IPv6 first then legacy IPv4. However the WoW client seems not to be doing this, simply hanging and eventually giving an error.

Obviously this has been reported to Blizzard, but I am sorry to say that they are taking the backwards step of taking no notice of IPv6 issues, effectively withdrawing support for the current version of Internet Protocol, IPv6

As such, customers may have to change their config on their game client not to try IPv6. This is obviously not something we would normally recommend as it removes the usual redundancy of having both protocols. It also gives the mistaken impression that IPv6 is somehow unreliable, rather than the more accurate impression that Blizzard are unreliable.

Hopefully they will fix it eventually. We have yet to see any official statement as to withdrawal of support for IPv6, a move that is very unusual for any company to take.

Yesterday 06:57:26
Blizzard have now said they have engineers working on this. No ETA.
Started 22 Jul

20 Jul 09:58:43
[Maidenhead Colocation] - Web and email outage - Open
04 Apr 2013 15:47:25

This is ongoing. We're investigating.

04 Apr 2013 16:07:41

This should now be fixed. Please let support know if you see any problems, or have any questions.

05 Apr 2013 09:15:04

This was resolved yesterday afternoon.

20 Jul 09:58:43
This may be related to a wider issue on the internet with a power outage at a major london data centre. Thos routing problems are still ongoing.
Started 04 Apr 2013 15:46:11

9 Jun 19:53:32
[DNS, Email and Web Hosting] - Email problems to gmail addresses - Open
9 Jun 17:01:30
We're noticing a large number of email to gmail accounts are being bounced back by gmail servers this afternoon. We are investigating the cause of this.
9 Jun 19:53:32
We have put in a temporary fix (sending via IPv4 instead of IPv6) and messages are new being delivered to gmail addresses without being bounced.
Started 9 Jun 16:00:00

5 Feb 16:32:23
5 Feb 16:32:23
Some call recordings from today have been lost. We've identified the problem and have worked around it until we can fix it. We should be able to fix it with no customer impact. If you are expecting a call recording that you have not received, support can check whether the call was affected by this problem. The call recordings lost are in the minority and most calls were recorded as expected. This problem is unrelated to yesterday's problems with lack of ring tone!
Started 5 Feb 16:25:24

12 Jun 2015 11:06:49
12 Jun 2015 11:00:23
Our office connectivity blipped. We have internet again, but our phones are down. We're investigating. This is not customer affecting.
12 Jun 2015 11:06:49
Phones are working again.
Previously expected 12 Jun 2015 14:57:07

13 Apr 2015
09 Apr 2015 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr 2015

12 Mar 2015 09:48:01
12 Mar 2015 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.

08 Jan 2015 12:39:06
08 Jan 2015 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
09 Jan 2015 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
12 Jan 2015 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 08 Jan 2015 12:39:06 by AA Staff
Previously expected 15 Jan 2015 17:00:00

03 Jun 2014 17:00:00
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

06 Jul 2015 12:49:42
We have been advised by Three of works on their network (22:30 8th July to 04:50 9th July 2015) Which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some partners may experience dropped connections. Any device reconnecting will automatically be routed via a new path. We apologise for any inconvenience this may cause and the short notice of this advisory.

12 Feb 2015 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

3 Aug 11:31:20
Yesterday 11:36:34
SIP2SIM may be affected for up to an hour between the hours of 20:00 on the 2nd of August and 06:00 on the 3rd of August. This is due to upstream GGSN upgrade work and the resulting testing of failover.
Planned start 2 Aug 20:00:00 by Carrier
Closed 3 Aug 11:31:20
Expected close 3 Aug 06:00:00

Thursday 16:47:17
20 Jul 16:56:12
On Tuesday the 26th and Wednesday the 27th of July 2016, between 22:00 and 23:00, upstream router maintenance is expected to cause minor disruption to data SIM connectivity. The disruption should not be for more than a few minutes a time.
Planned start Today 22:00:00
Closed Thursday 16:47:17
Expected close Tomorrow 23:00:00

20 Jul 12:41:10
20 Jul 08:18:13
We can see a major routing issue, and we are investigating. Not entirely clear the extent of the problem at this stage.
20 Jul 08:21:55
At the moment we are seeing what appears to be an issue with the links between London and Maidenhead. This can impact some Ethernet customers, VoIP, hosting, etc. However, a number of systems have a fallback for this, so many are still working, e.g our VoIP services are OK.
20 Jul 08:22:23
The link is partially back, and we are investigating the cause.
20 Jul 10:19:15
This is looking like a major fault within BT's core network. This is affecting many ISPs in the UK. We'll post more details when we can.
20 Jul 10:19:48
also see http://www.bbc.co.uk/news/technology-36844712 and http://www.ukinternetreport.co.uk/?p=d&id=215 Which are reporting the problem within BT's network.
20 Jul 10:30:51
This fault is affecting our communications with BT's systems. This means that our sales and support staff have very limited access to BT's ordering, diagnostic and fault reporting problems. so please to bear with us as staff will be struggling to do their work at the moment. (TalkTalk lines are ok though)
20 Jul 10:35:40
This is looking related to a power issue at a major london datacentre
20 Jul 10:48:05
We've had an update from BT regarding this issue, they say: There is an intermittent incident affecting the Openreach B2B and XML Gateways, as a result of this issue you may be experiencing a delay in the receipt of KCI's for both Fulfilment and Assurance requests as well as failures in Dialogue Services.
20 Jul 12:40:00
It looks like everything affected by this outage has returned to normal now.
20 Jul 12:40:06
It looks like everything affected by this outage has returned to normal now.
20 Jul 12:40:23
It looks like everything affected by this outage has returned to normal now.
Started 20 Jul 08:04:00
Closed 20 Jul 12:41:10

12 Jul 21:21:00
12 Jul 21:24:48
We've had some lines dropping and reconnecting this evening, between 20:50 and 21:20. Things seem back to normal now, but we're investigating the cause.
12 Jul 21:26:13
During the times above customers would have experienced routing problems, perhaps packetloss and VoIP problems. (although my Netflix stream was unaffected smiley )
12 Jul 21:34:11
Sorry about that everyone, early indicators is that a few DLS lines were being DOSd, and although our automated systems did drop the PPP and blackhole their IPs across the most of the internet, it did cause problems for some customers. We'll continue to monitor the network this evening and will do some more investigations in the morning as to the cause and how to help prevent this sort of thing in the future.
Started 12 Jul 20:50:00
Closed 12 Jul 21:21:00

8 Jul 04:00:00
[Broadband and Ethernet] - TalkTalk Outage - Closed
8 Jul 04:17:51
We lost TalkTalk lines shortly after 2am, and they were down for around 100 minutes. Our links to TalkTalk were either down or not passing traffic for this time. Things are returning to normal now, and we're monitoring the links and chasing TalkTalk for an explanation.
8 Jul 08:00:48
Service was restored at around 4am. We are investigating the cause. It looks like there was planned work within TalkTalk at their Telehouse location, but there was some problems with lines re-connecting via our Telecity connection. More information to follow as we get it.
Started 8 Jul 02:10:35 by Customer report
Closed 8 Jul 04:00:00

4 Jul 12:48:25
30 Jun 15:10:48
We are aware of problems affecting our L2TP and data SIM service, this is being investigated by our engineers now.
30 Jun 16:13:47
Apologise this has not yet been resolved, we are seeing further problems and it's still being investigated.
Started 30 Jun 15:09:09
Closed 4 Jul 12:48:25
Previously expected 30 Jun 19:09:09

19 Jun 13:39:55
[Broadband and Ethernet] - L2TP direct service - Info
3 Apr 16:36:34

Now that we have the new network upgrade we are hoping to properly launch the new L2TP specific service. Previously this was a somewhat informal service with somewhat complicated billing.

This new service is an L2TP over IP service for people using other broadband lines to access us and use our IPs and services including fixed IPv4 and IPv6 blocks. The plan is £10/month for a service capped at 100Mb/s with 1TB monthly allowance, clamping to 3Mb/s from next month's allowance on reaching allowance. Very similar to Home::1. No date yet but we are hoping to make this our standard L2TP service some time in April.

17 Apr 19:34:56
I'm sorry this is taking so long to sort - it is basically a tariff change, and order form, but we have been very keen to ensure the infrastructure is in place for this. I still hope to do this by end of month.
12 May 11:09:15
Information: http://aa.net.uk/broadband-l2tp.html
Order form: https://order.aa.net.uk/chaos2l2tp.html
Started 1 Apr
Previously expected 1 May

17 Jun 18:38:00
17 Jun 16:38:45
Talktalk have informed us that they are experiencing issues with one of their network routers in London. This has impacted all Talktalk lines in the surrounding area, covering ADSL, VDSL(FTTC) and EoFTTC lines since approximately 15:21. Affected areas are Canonbury, Lower Holloway, Kentish Town, Upper Holloway, Kingland Green and Clerkenwell.
17 Jun 19:56:24
All affected customers appear to be back online now, and TalkTalk have advised us that a fibre break has been repaired.
Resolution Openreach have repaired a break in TalkTalk's fibres.
Started 17 Jun 15:21:00 by TalkTalk
Closed 17 Jun 18:38:00
Cause TT

17 Jun 04:00:00
16 Jun 21:23:10
Talktalk have informed us that they are experiencing issues with their network core in Swansea. This has impacted all Talktalk lines in the surrounding area, covering ADSL, VDSL(FTTC) and EoFTTC lines since approximately 19:53. Talktalk have engineers on their way to the equipment to ensure it is correctly restarted, and currently believe the outage is due to a power failure in the BT exchange which hosts the equipment. BT Openreach have been informed of the power failure, and are also investigating. The following area codes and exchanges are impacted by this outage: 01792, 01554, 01269, 01639, Llanboidy, Amman Valley, Pontardawe, Port Talbot, Llangennech and Crosshands
16 Jun 21:48:54
We have received reports that the outage is also effecting the 01656 area code, and lines around Bridgend. We will post further updates when we have information from Talktalk.
Resolution The cause of this incident has not been determined. BT confirmed they had a fibre related issue in Merthyr Tydfil and restored our service by moving the Talk Talk circuits to a spare fibre cable. As no further issues have been reported, we can confirm closure to this incident with further investigations taking place under the Problem Management process.
Started 16 Jun 19:53:00 by TalkTalk
Closed 17 Jun 04:00:00
Previously expected 16 Jun 22:30:00

17 Jun 10:39:29
6 Jun 14:00:34
Our carrier for Data only SIMS are performing a software update on their LNS, this will mean a short outage for data SIMS for a minute or two between 10PM and 11PM on Thursday 9th June.
10 Jun 10:10:01
Aql will be performing another upgrade on the remaining infrastructure on Thursday 16th June 2016 at 22:00. This part of the migration to the new LTE infrastructure. They have advised that the patch has been tested in lab conditions, and no issues are expected, however in the event of a fault they revert the patch and investigate. As with last night some of our sessions may disconnect and reroute while the patch is applied and the LNS restarted, however no more than five minutes disruption is anticipated.
17 Jun 10:40:45
Looks like planned works completed successfully last night. We will post further information on our carriers LTE upgrades as we receive them.
Started 16 Jun 22:00:00 by Carrier
Closed 17 Jun 10:39:29

4 Jun 12:33:49
3 Jun 09:33:49

We now have new options for VDSL (Fibre to the cabinet capped at 80/20) with terabyte allowance and free install. This no longer forces you to include the phone line copper pair.

We also now have the option if you are taking over the phone line and it has a BT number you want to keep where we can port the number to VoIP. This is a cost for phone line install or take over.

This gives a lot more choice on the Terabyte services. It also allows ordering where the line is on a satellite exchange and could not get TT MPF (phone line) before. Do try the order form to see if it is available to you now.


4 Jun 12:33:49
We have worked out that around half of the customers currently with ADSL services could upgrade to VDSL. We are working on showing the estimated speed for FTTC upgrade on the control pages and bills.
Started 3 Jun

27 May 20:15:12
27 May 18:59:26

A number of customers were emailed that their 1st of month DD was cancelled.

This process was stopped as soon as we realised what was happening so not many people will be affected. You will receive normal DD collection notices in due course for your monthly bills or have collection within 3 working days as per exiting notice (so may be 3rd instead of 1st) as per DD rules.

Our DD management system is coded to be as cautious and customer focused as possible and a minor error meant it was not happy with the DDs for regular collections being in place before the invoice was issued. This has been corrected now.

Sorry for any confusion caused.

Started 27 May 18:00:00
Previously expected 27 May 22:56:44

27 May 04:30:00
27 May 02:24:53
At around 1:51 today TT dropped offline. We are investigating.
27 May 02:56:38
We've seen reports of other ISPs lines via TT also having the same problem.
27 May 03:16:06
This looks very much like a major outage within TalkTalk. Our BGP session to them that carry our traffic is down at one datacentre and up and our second datacentre but with no prefixes being advertised to us from TalkTalks side.
27 May 03:58:05
We are seeing some lines starting to reconnect.
27 May 03:58:26
We are seeing some lines starting to reconnect.
27 May 04:01:19
No word from TT as to the cause of this, but we'll update this post in due course.
27 May 04:14:03
Majority of lines are back on now - waiting for the last ones to reconnect.
Resolution It looks like this was caused by some TT planned work of which we were unaware, and clearly did not go to plan. We are trying to get a full explanation from TT. In the mean time the problem is resolved. If you are off line, try restarting your router to force it to reconnect.
Started 27 May 01:51:00
Closed 27 May 04:30:00

19 May 21:55:00
20 May 09:03:45
On the evening of 19th May a small number of lines all in the Guildford area (GU postcodes) were offline between around 21:35 and 21:55. This only affected TalkTalk connected circuits. We'll investigate the cause of this with TalkTalk and will update this post shortly.
Resolution TalkTalk have confirmed that they did have a major network outage. This was resolved by TalkTalk rebooting one of their main BRAS aggregation devices. They do apologise and have raised the problem with their vendor.
Started 19 May 21:35:00
Closed 19 May 21:55:00
Cause TT

14 May 17:36:17
14 May 17:36:17

I have added some more features to card printing.


It now included linear barcodes including code 39, code 128, EAN13, UPC, EAN8, EAN5 and ITF

Started 14 May 17:34:55

20 May 14:30:35
13 May 21:45:49
Our upstream carriers have advised us of scheduled maintenance taking place between 02:00 - 03:00 on Friday 20th May 2016 affecting Data SIMs only.
Some customers may notice a momentary drop in connections during this time as any Data SIMs using that route will disconnect when the link is shut down. Any affected Data SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, Data SIM connectivity should be considered at risk throughout.
Voice SIMs are not affected by this maintenance.
Started 13 May 20:54:24
Closed 20 May 14:30:35

12 May 17:00:28
12 May 16:58:38

We use IRC all the time here at AAISP, both internally between staff teams and in helping our customers. Many customers have their own IRC client setup on their computer whilst others pop in to the IRC channel by using the web-based version we provide via our website.

To help improve the experience for customers using our web-based irc client we have a new one to try:


We'll be updating our links on the website to the new one next week, but feel free to try out the new client.
Started 12 May 16:50:00

5 May 16:52:53
5 May 14:42:29
We've seen several incidents, at 12:25, 12:33 and 14:26 where we lost connectivity over all of the VLANs we currently utilise on our Datahop circuits in Telehouse North. This has affected a few of our wholesalers who connect in to us over Datahop VLANs. This looks like it's relating to a fault within datahop. The link is up at the moment.
5 May 16:25:36
We've had an update from datahop saying that they have replaced a switch in their network. Service has remained stable since 14:30 this afternoon.
5 May 16:54:00
Datahop have completed their switch replacement work. This incident is now closed.
Resolution Faulty switch within Datahop has been replaced.
Started 5 May 12:30:00
Closed 5 May 16:52:53

28 Apr 16:18:20
28 Apr 15:30:05
We are seeing a number of broadband lines in the south of London suffering from severe packet loss and high latency. This looks like a fibre problem with TalkTalk. This has been reported and as soon as we have an update we will let you know.
28 Apr 15:36:41
Here is an example graph
28 Apr 16:18:44
TalkTalk have fixed this - that was quick.....
Started 28 Apr 15:27:37
Closed 28 Apr 16:18:20
Previously expected 28 Apr 19:27:37

2 May 17:36:35
[Broadband and Ethernet] - At risk: Router upgrades - Completed
28 Apr 09:29:41
We may do some router upgrades and changes over the weekend as part of ongoing network tidy-up. This should not be disruptive.
30 Apr 08:52:48
The first changes have been done, with most likely impact (if any) being on our office connections. If all goes well today we'll do more changes tomorrow (Sunday) morning. As I say, these should have no noticeable impact, but are to improve some of our internal routing logic.
1 May 05:47:14
The next stage was done this morning. Any issues please let us know on irc. We have more changes later today (Sunday).
1 May 09:53:27
Some further work was done before 8am today (Sunday). We plan to do the next stage tomorrow (Monday) morning. So far all seems to be going to plan.
2 May 11:01:20
One small causality was our NAT64 gateway, which is being restored now.
2 May 16:51:30
We have identified a possible peering MTU issue and are investigating. There may be more work this evening (Monday). Again, we expect no disruption, but a risk.
2 May 17:18:19
Yes, we have identified an issue with MTU on some peers, and this is being rectified now.
2 May 17:37:08
MTU fixes now applied
Resolution The main planned work was all completed before 8am this morning, all looks well. Any issues contact support or mention on irc or text the MSO number. We'll be keeping an eye on things today anyway.
Started 30 Apr
Closed 2 May 17:36:35
Previously expected 3 May

28 Apr 04:55:55
27 Apr 11:21:25
We are making a correction to routing. This will impact external connectivity for a few seconds but should not cause lines to drop PPP. We aim to do this over night or early tomorrow morning. Sorry for any inconvenience.
Resolution Changes complete
Started 28 Apr
Closed 28 Apr 04:55:55
Previously expected 28 Apr 08:00:00

27 Apr 06:09:11
[Broadband and Ethernet] - At risk: Routing changes - Completed
26 Apr 16:58:57
We are tidying up some of our internal routing logic following from our major network upgrades recently. This is expected to have no impact, but there is a risk it could impact Ethernet and broadband customers and maybe other services. The plan is to get this all done well before 8am so that we can sort any issues that come up or reverse changes that are made.
Resolution The work is complete and all looks well - any issues, please let us know. P.S. an issue with BGP connections via DSL was resolved.
Started 27 Apr
Closed 27 Apr 06:09:11
Previously expected 27 Apr 09:00:00

26 Apr 17:00:00
26 Apr 16:40:46

Some Ethernet customers have seen a routing blip today. This is mainly those that have a secondary block of IPs routed via your Ethernet link and related to that additional block.

We should have posted an "at risk" notice about some very minor routing changes tidying up after all of the recent work in our core network. I genuinely did not expect any impact - we have done simple routing changes many times in the past without experiencing any issues.

Unfortunately the changes uncovered an underlying problem with the redundancy logic, and hence something that should have had no impact did in fact have an impact. This was fixed quickly within minutes of the report.

There is other work we expect to do tomorrow morning before 9am, including sorting the underlying issue we have uncovered today. If there are any issues you encounter with routing in the morning, please do contact support.

Sorry for any inconvenience caused.

Started 26 Apr 13:00:00
Closed 26 Apr 17:00:00
Previously expected 26 Apr 17:00:00

19 Apr 12:42:22
[General] - DRIPA - Info
16 Jul 2014 10:13:15
Just to be clear on our policy here - when DRIPA comes in to force, and if A&A become subject to a retention notice for all customers, we aim to work on all practical legal means to minimise the amount of data retained under that legislation - making full use of the bad wording in the Schedule in the 2009 regulations where possible. We also aim to clearly publish what is retained under such a notice and what steps we have taken to minimise such data. Such steps may mean separate companies running email or other services, or even hosting some servers outside the UK, if those are practical steps we can take. Why? Because blanket mass surveillance is illegal under EU law as it is against our basic human right to privacy as decided by a court, that's why! Feel free to comment on my blog.
Started 15 Jul 2014

17 Apr 19:33:54
14 Apr 16:22:27

As customers may recall, some time ago we ported numbers in several exchange area codes to a company called Incoming Numbers Ltd, and then leased use of those numbers back to A&A. The effect was no technical change to operation, but it meant the numbers were technically ported to another provider. When you ordered numbers in those areas, the order form confirmed you were porting to INL. The bills showed the numbers were ported to INL. This is only a small number of area codes.

We're pleased to confirm that we have today reversed that and the ported numbers are now back with us, Andrews & Arnold Ltd. So the bill will no longer show the note about Incoming Numbers Ltd.

The reason for this was that OFCOM started charging for numbers, after having provided them for no charge, and waited for us to start using the blocks. Something that in any other "business" would be an unacceptable business practice, in my view. There was a discount for ported out numbers, hence the porting.

The plan worked, and we got the credit for ported out numbers off the bill. However, we found a much more effective solution. Most of our numbers are allocated to Andrews & Arnold (numbers) Ltd, a company which is not, itself, any more, a telco, and just leases the numbers to Andrews & Arnold Ltd to use. As it is not a telco, OFCOM do not charge it, and so problem solved.

Sorry for the puzzlement this may have caused some customers in the past. It caused me some paperwork as INL is a proper separate trading company with accounts, and everything, but not an A&A subsidiary. It will be wound up or made dormant for now.

I hope both of these methods, to avoid being charged for something we already had, have been helpful to OFCOM as they said the charging was an "experiment", and it was clearly our duty to provide the best data to them on how well it worked as an experiment.

Started 14 Apr 16:13:50
Previously expected 14 Apr 20:13:50

17 Apr 19:33:27
[Broadband and Ethernet] - Home::1 Terabyte bonus - Info
4 Apr 20:26:22

Today we tweeted, somewhat as a joke, "Customers on Home::1 TB services downloading the Panama papers, let us know when you hit your 1TB quota and we'll topup 4 free #PanamaLeaks"

This was mainly because this was something, at 2.7TB, that was actually too big for our monthly allowance on the terabyte service. It shows that even 1TB is not enough for that amount of data, but it really is one of the few things that could cause you to download so much.

However, I also appreciate that we have had had some issues for some weeks now, and they are finally coming to an end with much needed capacity upgrades.

So, as a thank you for the customers that have embraced the new Home::1 Terabyte tariffs, I have actually added 3TB to this month's quota (as top-up, so carrying to end of next month). Thus honouring the tweet for all.

Thank you all - without customers we would be nothing.

Started 4 Apr 20:21:04
Previously expected 5 Apr 00:21:04

17 Apr 19:32:58
[Broadband and Ethernet] - LNS roll overs - Info
14 Apr 08:29:19

I am pleased to confirm that as we now have many more LNSs in service we can return to the normal LNS roll-over strategy http://aa.net.uk/kb-broadband-duallns.html

We moved B.gormless to A.gormless last night, and would normally do one per night to move through all of the LNSs. However, this time we are doing some hardware reconfiguration of LNSs as well, so the next roll-over (C.gormless to B.gormless) will be some time next week.

The roll-over went smoothly as usual. Do remember that you can configure a preferred time in the control pages, as we know some people are up late working or gaming, and others are up early, and we want to minimise any disruption to your connection by routine maintenance like this.

Started 14 Apr 01:00:00
Previously expected 17 Apr 23:32:48

14 Apr 16:27:06
01 Dec 2015 10:00:08
We are starting to use a new address for the Control Pages. Instead of https://clueless.aa.net.uk it'll be https://control.aa.net.uk. The reasons for this are two fold:
  1. Control.aa.net.uk is a better description of the service than clueless.aa.net.uk - some customers have been a little confused or wary of the name 'clueless'.
  2. The new name has given us the opportunity to update the SSL (security) settings.
The slight downside of this is that older clients may not have the modern ciphers needed. For example, the minimum browser versions would be:
  • Firefox 27
  • Chrome 22
  • IE 11
  • Opera 14
  • Safari 7
  • Android 4.4
  • Java 8

We would recommend that customers update their browsers if they are using the above.

As always, we're happy to receive feedback.

Resolution Launched
Started 01 Dec 2015 09:04:43

14 Apr 16:26:42
21 Dec 2015 08:46:32
Please see our most recent newsletter: aa.net.uk/news-2016-01.html
Started 21 Dec 2015 08:45:00

14 Apr 16:25:00
02 Aug 2014 12:02:18
We are planning an expansion of our VoIP services which will require the use of more IP addresses than we have told customers to allow through their firewalls, so have added additional IP ranges to the list. We are not using these new IP ranges yet, but are giving advanced notice of the change to give customers time to update their firewall rules. The list is at http://wiki.aa.org.uk/VoIP_Firewall
Started 02 Aug 2014 11:50:35
Previously expected 02 Aug 2014 11:50:35

14 Apr 16:24:41
02 Jul 2015 10:44:33
Some customers have advised that they are unable to call some numbers which they were able to call last week. This is due to new rules which mandate that we charge a service charge set by the range holder plus our access fee (2p/min) for such numbers. OFCOM have failed to provide means for telcos to determine the service charge that applies. Their only suggestion was to use the Bt Carrier Price list. We have loaded this (and are doing so every day) but it seems to be lacking a number for 084 and 087 number ranges. Until those appear in the BT Carrier Price list, or OFCOM actually come up with a means for telcos to comply with the new regulations, the numbers cannot be called as we have no way to comply with OFCOM rules. Feel free to complain to OFCOM over the stupidity of these new rules.
07 Jul 2015 11:14:47
We have managed a one-off load of current codes, but there still remains no official source of the codes and rates due to OFCOM's failure to define who is responsible for compiling and maintaining this information. As such there is still no sensible way to ensure this data remains correct and complete.
Started 01 Jul 2015
Previously expected 01 Aug 2015

14 Apr 16:24:06
[VoIP and SIMs] - 999 location data - Info
12 Mar 2015 18:24:06
We allow users to provide address/location data to pass to emergency services in the event of a 999 calls. This goes via BT and BT are being a pain at present. As we have explained the data is simply a matter of what customers provide. VoIP has no inherent way to provide an accurate geographic location of callers. But BT are unhappy with some of the data and seem to have blocked us sending updates this week. They even suggested that they have to send OFCOM reports of incomplete and incorrect data as some sort of threat. We know a lot of people have multiple locations from which they make calls. We do this purely as a good will gesture and in the spirit of GC4, because we don't have location data in the first place. So if BT do not stop buggering about soon we'll be reporting BT to OFCOM instead. Anyway, the upshot is that a few customers that have provided location data over that last few days are not yet updated to BT and hence the emergency services. We are working to resolve this and hope to have it working again soon. The config pages for your VoIP set up show if there is an issue and you get an email once it is all confirmed. In the mean time, we do apologise for any concern this may cause. None of our services are provided to be used in any "safety of life" circumstances, as per our standard terms.
16 Mar 2015 16:49:34
We are hoping to have this sorted, perhaps today. It is likely that anyone with existing data registered may get a new email at some point confirming the data as we are planning to re-send everything from scratch just to be sure. We would ask that customers review what they have entered and ensure it is as accurate as possible to assist emergency services in the event of a 999 call.
18 Mar 2015 17:40:37
Data is updating again to BT. We would recommend customers ensure accurate data on the control pages for 999 calls. We will review this over the next few days, and customers may still get update emails in due course.
Started 08 Mar 2015
Previously expected 16 Mar 2015

11 Apr 12:49:31
11 Apr 11:56:02
We are investigating an issue with one of our call servers (A.Voiceless). DNS has been updated to route via B, which should ensure service is restored for those that are affected but it is worth forcing phones to re-register if you are having issues (some VoIP devices do not cache DNS properly).
11 Apr 12:09:53
Engineers on on their way to the datacentre. Calls and registrations are all being handled via our 'B' call server.
11 Apr 12:34:29
Engineers are onsite.
11 Apr 12:38:00
Hardware is being replaced.
11 Apr 12:44:39
Hardware replaced, and server online.
Resolution The faulty server has been repaired and back on line.
VoIP and SIMs Users Affected 50%
Started 11 Apr 11:50:00
Closed 11 Apr 12:49:31
Previously expected 11 Apr 12:50:00

4 Apr 15:31:58
[Broadband and Ethernet] - Bogus uplink speeds from BT - Closed
4 Apr 14:44:27
Since our software update on our LNSs on Sunday we are now able to pick up the uplink speeds for DSL lines from the carrier.

However, some customers are seeing slow upload speeds and also a red line on the bottom of our monitoring graphs indicating the link is full. The problem is that BT are sending bogus uplink speeds for some lines, and therefore we are limiting the throughput.

We are working on a fix for this now.

4 Apr 15:32:16
This has now been resolved.
Started 4 Apr 14:42:19
Closed 4 Apr 15:31:58

17 Apr 19:32:20
[Broadband and Ethernet] - At risk: Routing changes - Completed
3 Apr 17:44:24
We have a number of things to do over the coming few weeks with changes to the peering and transit links we use. This should not impact customers directly, and is some minor routing changes, but it is the final step in managing the capacity upgrades. So there is a small risk of some side effects. We'll aim to keep these outside office hours as much as possible.
4 Apr 18:13:56
We have sorted one of the links to LONAP today, but later in the week we will be moving more transit and peering links, step by step, carefully. This should not impact service, but will allow more capacity and future proofing.
13 Apr 10:37:38
The next stage of work is scheduled for Wednesday 20th, again we don't expect any customers to be affected by this as it is minor configuration changes at our side.
Started 4 Apr
Closed 17 Apr 19:32:20