Order posts by limited to posts

12 Jun 11:06:49
12 Jun 11:00:23
Our office connectivity blipped. We have internet again, but our phones are down. We're investigating. This is not customer affecting.
12 Jun 11:06:49
Phones are working again.
Previously expected 12 Jun 14:57:07

26 May 17:55:57
26 May 17:55:57
We have had some issues with printing of SIM cards of late. We have two printers but they are struggling with the card material, and this making the print quality somewhat poorer than we would expect.

In light of this, we will be happy to try and print cards as requested, even if it takes us a few attempts, but we are also prepared to offer the cards at a discount if the print quality is not up to scratch. If you order cards, please feel free to pick artwork or load your own as usual, but staff may contact you if there are issues and offer them at a discount.

We are looking in to a new make of card printer which should address this, but we are being careful to evaluate the printer before we go ahead, so it may be a few more weeks.

Sorry for any inconvenience or disappointment this may cause.

Started 1 May

13 Apr
9 Apr 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr

12 Mar 09:48:01
12 Mar 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.

19 Jan 16:08:37
17 Jul 2014 10:08:44
Our email services can learn spam/non-spam messages. This feature is currently down for maintenance as we work on the back-end systems. This means that if you move email in to the various 'learn' folders they will stay there and will not be processed at the moment. For the moment, we advise customers not to use this feature. Will will post updates in the next week or so as we may well be changing how this feature works. This should not affect any spam scores etc, but do contact support if needed.
29 Jul 2014 11:42:12
This project is still ongoing. This should not be causing too many problems though, as the spam checking system has many many other ways to determine if a message is spam or not. However, for now, if customers have email that is miss-classified by the spam checking system then please email the headers in to support and we can make some suggestions.
19 Jan 16:08:37
We are working on rebuilding the spam learning system. We expect to make this live in the next couple of weeks.
Started 17 Jul 2014 10:00:00
Update was expected 29 Jan 13:00:00

8 Jan 12:39:06
8 Jan 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
9 Jan 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
12 Jan 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 8 Jan 12:39:06 by AA Staff
Previously expected 15 Jan 17:00:00

03 Jun 2014 17:00:00
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

6 Jul 12:49:42
We have been advised by Three of works on their network (22:30 8th July to 04:50 9th July 2015)

Which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some partners may experience dropped connections. Any device reconnecting will automatically be routed via a new path. We apologise for any inconvenience this may cause and the short notice of this advisory.

12 Feb 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

16 Jul
5 Aug 12:36:08
Due to an issue our our carriers, billing for mobile data SIM usage for 26th June to 16th July has been delayed. We expect this to be correctly included on future bills. We apologise for the delay raising these charges.
1 Oct 09:17:40
The charges have still not been received, but should be this month (October) so they can be billed correctly.
Started 26 Jun
Closed 16 Jul

22 Sep 11:35:00
21 Sep 13:26:39
There is some sort of routing problem affecting customers. We are still investigating
21 Sep 13:35:40
Routing/traffic is back to normal, we continue to monitor...
22 Sep 10:24:49
The burst of traffic that caused this disruption has not returned. We do apologise for the inconvenience this caused.
Started 21 Sep 13:20:00
Closed 22 Sep 11:35:00

10 Sep 10:05:21
25 May 22:24:55
We're seeing peak time (evening) congestion on lines at BERMONDSEY exchange again. It started on evening of 29th April. We've reported this previously, on 14 Jan and was fixed then on the 15 Jan. We'll update this post shortly.
2 Jun 13:51:26
No updates as yet. We are chasing TT again today.
5 Jun 09:18:13
Sadly no update, we'll chase this via alternate channels!
10 Sep 10:05:34
It appears that the issue is resolved.
Started 25 May 22:23:31
Closed 10 Sep 10:05:21

10 Sep 09:54:03
4 May 21:41:59
We are are seeing congestion on HORNDEAN and WATERLOOVILLE exchanges (Hampshire). This is usually noticeable in the evenings. This will be reported to BT, and we'll update this post with updates.
10 Sep 09:54:17
Congestion issues appear to be resolved at both locations.
Started 4 May 21:38:34
Closed 10 Sep 09:54:03

10 Sep 09:53:10
19 Jun 11:28:43
We are seeing around 1-2% packet loss on the MAIDA VALE Exchange. This has been reported to the TSO team within BT Wholesale.
10 Sep 09:53:19
Packet loss appears to be gone
Broadband Users Affected 0.50%
Started 19 Jun 11:26:46
Closed 10 Sep 09:53:10
Previously expected 21 Jun 13:00:00

10 Sep 09:50:07
11 Aug 11:35:34
We have noticed congestion one some of our TalkTalk circuits. customers affected are seeing an increase of latency and some packet loss. We have reported this to TalkTalk and as soon as we have an update we will update this post.
11 Aug 11:42:09
Here is an example graph
11 Aug 14:49:09
Update from TT We lost multiple 10G circuits to our transmission network due to a BT fibre break in the Ipswich area - Openreach have applied for a MBORC on the issue. There was congestion in the Ipswich and Cambridgeshire wider area as a result which was resolved shortly after midnight.
Started 10 Aug 17:32:43
Closed 10 Sep 09:50:07

10 Sep 09:49:30
14 Aug 11:50:09
We have been seeing increased packet loss and latency on certain line on the Forest hill exchange. It appears one VLAN is currently running hot. BT are aware and are investigating.
18 Aug 10:31:33
BT believe the SVLAN has been upgraded. We will monitor the circuits tonight.
10 Sep 09:49:48
Issues appear to be resolved.
Started 14 Aug 11:46:37 by AA Staff
Closed 10 Sep 09:49:30
Previously expected 17 Aug 09:00:00

9 Sep 10:16:19
[Email and Web Hosting] - SSL Certificates Updated - Info
9 Sep 10:16:19
We're updating SSL certificates for our email servers today, including webmail. The old serial number is 106E03. The new serial number is 11207E. Users who don't have the CAcert root certificate installed may see errors. Details on http://aa.net.uk/cacert.html

26 Aug 23:30:00
26 Aug 00:16:15
We've seen a couple of incidents this evening where customers have seen packet loss and some DSL sessions have blipped. It seems to have been caused by a traffic flood, and we're investigating the cause. This status is posted as Minor as things look OK at the moment.
26 Aug 20:33:37
We've had another set of blips affecting internet access for some customers this evening (20:15ish). We are looking in to the cause of this.
26 Aug 21:07:47
This is problem is continuing to cause disruption, please bear with us as we work to get to the bottom of this.
26 Aug 21:37:02
Traffic is has been normal for the past 30 mins or so, we've not yet tracked down the cause and are still looking on to this.
26 Aug 22:37:34
We believe we have found the cause of the network issues. More details to follow in the morning.
26 Aug 23:05:49
Unfortunately we are still having problems, we're still working on this.
26 Aug 23:33:38
We've done further work on this, traffic is back to normal at the moment and we are monitoring the situation.
Resolution The intermittent problems on the evenings of 25th and 26th were caused by a distributed denial of service attack against one of our customers. This type of attack does happen from time to time and we have systems which usually stop an attack in its tracks very quickly. However, in this case the attack was not automatically blocked. Due to the nature of the attack it did take longer than we would have liked to pin point the target of the attack, and this was eventually done at around 23:30 on the 26th. Moving on, we will be looking in to why this was not automatically blocked and how we can improve other systems with a view to prevent this from being a problem in the future. We do apologise for the inconvenience this caused.
Closed 26 Aug 23:30:00

23 Aug 17:55:28
22 Aug 11:26:44
We've got at least 5 examples where SIM2SIM isn't working for calls at all. It's a carrier problem outside our network. Mobile data is working. It's a known problem, and it's being investigated.
22 Aug 12:05:41
We don't have an estimated time for a fix yet, but we'll post updates as soon as we can.
23 Aug 17:57:30
Apologies for the late update! This was fixed around 17:30 yesterday. It was some sort of signalling problem, and alternative routing was used as a workaround to get it up and running again.
Closed 23 Aug 17:55:28

28 Jul 13:00:00
27 Jul 16:17:35
We have a problem at the moment activating data SIMs. This has been escalated to the carrier who are investigating.
28 Jul 10:30:41
No update as yet, but we are chasing the carrier on this.
28 Jul 13:45:21
SIM activation has now been restored.
Resolution Problem at the carrier has been fixed.
Started 27 Jul 09:00:00
Closed 28 Jul 13:00:00

30 Jun 16:26:40
30 Jun 16:27:00
We're investigating some sort of issue affecting lots of broadband showing some packet loss.
30 Jun 16:34:37
Seems to be recovering, and we are trying to find cause now.
Resolution Further issues during the night allowed us to track down the source of the problem which was a compromised customer hosted machine. Steps are being taken to address this in future.
Started 30 Jun 16:16:40
Closed 30 Jun 16:26:40

27 Jul 10:36:06
16 Jun 11:32:55
As some of you may know, the new system for migrations of broadband lines comes in this weekend. See http://aa.net.uk/news-20150601-not.html

We expect to enable the new notice emails during this week, maybe even today. These will be sent for customers migrating away from us, and migrating to us (when order commits) and include the options for cancelling the migration.

We are also taking this opportunity to include notices for services ceasing - either because you have requested a cease, or unsolicited (e.g. if ceasing phone line causes cease of broadband).

In any case, if leaving us, the notice details any early termination charges.

If any issues, concerns, or questions, please let us know.

16 Jun 18:24:44
We are now sending NoT emails. Please let us know any issues.
Started 16 Jun
Previously expected 20 Jun

20 Jul 19:00:00
18 Jul 12:40:39
We have had some reports this week of poor call quality on SIP2SIM some of the time, and it seems to be getting worse. We are looking in to it, and have had reports that it is particularly bad today. It seems to be something in the link between us and the mobile carriers.
20 Jul 15:38:24
Detailed logs have been sent to the carrier and they are investigating the cause of this.
20 Jul 18:00:48
The carrier is making changes soon that may address this issue.
21 Jul 14:31:52
Our carrier implemented a change at their end at around 7pm on Monday which should fix call problems. We are monitoring this ourselves and welcome feedback from customers.
Resolution The carriers have made a change which they believe has addressed the issue.
Started 11 Jul
Closed 20 Jul 19:00:00
Previously expected 22 Jul

9 Jul 17:15:00
9 Jul 16:35:04
We are reloading one of our route reflectors - this should have little of no impact as routing should automatically fall over to the other.

If this does not resolve the issues we may have to do more work.

Started 9 Jul 17:00:00
Closed 9 Jul 17:15:00
Previously expected 9 Jul 17:10:00

2 Jul 10:44:33
2 Jul 10:44:33
Some customers have advised that they are unable to call some numbers which they were able to call last week.

This is due to new rules which mandate that we charge a service charge set by the range holder plus our access fee (2p/min) for such numbers.

OFCOM have failed to provide means for telcos to determine the service charge that applies. Their only suggestion was to use the Bt Carrier Price list. We have loaded this (and are doing so every day) but it seems to be lacking a number for 084 and 087 number ranges.

Until those appear in the BT Carrier Price list, or OFCOM actually come up with a means for telcos to comply with the new regulations, the numbers cannot be called as we have no way to comply with OFCOM rules.

Feel free to complain to OFCOM over the stupidity of these new rules.

7 Jul 11:14:47
We have managed a one-off load of current codes, but there still remains no official source of the codes and rates due to OFCOM's failure to define who is responsible for compiling and maintaining this information. As such there is still no sensible way to ensure this data remains correct and complete.
Started 1 Jul
Previously expected 1 Aug

1 Jul 04:34:32
1 Jul 03:11:16
From 2:18 today, we are seeing packet loss and some lines dropping their connection.
Resolution We've identified the caused of this, and traffic is back to normal.
Started 1 Jul 02:18:00
Closed 1 Jul 04:34:32

30 Jun 18:42:13
30 Jun 10:22:49
From tomorrow our rates of 084, 087 and 09 numbers are changing as per OFCOM rules. These will now be the advertised rate for the number plus an access charge. Our access charge is 2p/min inc VAT. This means if someone lists a number at 10p/min plus access charge, we will be charging 12p/minute for the number.

This compares well to many operators working on 10p/min access charges.

Note that SIP2SIM is a separate mobile access service and has its own access charge for all calls either way regardless of the VoIP gateway to which it is connected.

Note that we do not provide access to 09 premium rate numbers at present and most customers have a limit on call rates set which prohibits calls to expensive 084/087 numbers.

Started 1 Jul

28 Jun 16:29:57
28 Jun 14:02:49
Access to accounts and placing orders will be delayed / unavailable for some minutes. Sorry for any inconvenience.
28 Jun 15:24:19
Taking somewhat longer than expected, sorry.
Started 28 Jun 14:00:00
Closed 28 Jun 16:29:57
Previously expected 28 Jun 16:30:00

24 Jun 16:27:39
24 Jun 16:27:39
When Home::1 reaches quota you can opt to have the service stop, slow down, or auto top-up.

Slow mode was around 250kb/s. However, we have changed this so that the speed depends on the tariff. For 100GB users the slow mode will now be around 330kb/s and for 200GB users it will be around 660kb/s. The idea is that at that speed you could not use the whole of the coming month's quote even flat out.

Started 24 Jun 16:25:00

19 Jun 16:28:46
31 May 15:28:31
With the up-coming changes to broadband migrations and the abolishing of Migration Authorisation Codes (MACs), AAISP has launched a new "anti-slamming" service to allow customers to "lock" their line against unwanted migrations to another provider.

This extra service, which is completely free, works on much the same principle as "domain locking" where domains can be locked against migration.

The process is simple and allows each line to be locked against migrations for broadband or the underlying copper pair "phone line" part of the service. It is just a "standing order" from the customer to AAISP to reject all migrations. From the 20th, any migration request that then comes in is automatically rejected if anti-slamming is enabled. An email is sent to advise the customer of what happened, including a simple link to turn off the anti-slamming if they do wish to migrate after all.

The anti-slamming service has been provided by popular demand after many customers expressed concerns that their lines could be "slammed" (maliciously taken over by other telcos) or that mistakes could lead to unwanted migrations, and it would be very easy to miss the notice of transfer that is sent before the migration goes ahead. OFCOM do not seem to have created any "fast correct" of mistakes, so an unwanted migration could mean waiting another 10 working days to fix the situation.

Slamming is just one of the many concerns over the new migration process. There may still be ways LLU providers can take over lines, as can happen now without a MAC, but this new service should avoid mistakes and give customers peace of mind.

The control pages for the line include a simple link to enable or disable the anti-slamming service.

Started 31 May 15:27:25
Previously expected 20 Jun

19 Jun 16:28:30
14 May 15:42:00

We are please to inform customers that we are changing the entry-level router that we supply to our customers. From next week we'll be shipping the ZyXEL VMG1312 by default instead of the Technicolor.

Since around 2012 we have been providing the Technicolor TG852, which was the first consumer level router to support IPv6. With the advent of wires-only FTTC and the need for a more flexible and easy to use router we have been looking for a replacement. The ZyXEL VMG1312 is able to do ADSL, FTTC and PPPoE and is flexible enough to be used on most of our lines. We have been working with ZyXEL over the past few months to iron out bugs that we have found. There are still some bugs to be fixed and these are detailed on our Support site. The biggest bug is the lack of 1500 byte MTU when running in bridge mode, however, ZyXEL expect to have this fixed soon and in the meantime FTTC installations will be installed with the Openreach modem.

More information about the router: https://support.aa.net.uk/Category:ZyXEL_VMG1312

Started 14 May 13:12:00

19 Jun 16:27:57
19 Jun 16:27:57
You no longer need to obtain a Migration Authorisation Code to migrate broadband. Simple contact the new provider with your details to arrange the migration.

If you wish to migrate to us, please complete the order, and you will not be asked for a MAC.

If you wish to migrate away, simply contact the new provider (from tomorrow) and they can arrange the migration.

The lead time has increased from 5 working days to 10 - a significant added delay for which you can thank OFCOM.

Started 19 Jun 16:25:59
Previously expected 20 Jun

18 Jun 15:00:00
[Broadband] - BT Blip - Closed
18 Jun 14:50:08
A number of BT lines dropped out at about 14:44, this appears to be an issue on the BT side, all lines have now recovered. We are chasing Bt for an explanation.
18 Jun 15:03:46
Some lines are still blipping, we are investigating further.
Resolution The blip seems to have gone away - we may move some lines that ended up on the wrong LNS back later. This is not the first time this link to BT had blipped and we are chasing with BT.
Started 18 Jun 14:47:59
Closed 18 Jun 15:00:00

16 Jun 18:25:11
13 Feb 14:29:06
We currently supply the Technicolor TG582 for most ADSL services, but we are considering switching to a new router, the ZyXEL VMG1312-B

It is very comprehensive and does both ADSL and VDSL as well as bridging and wifi. It means we can have one router for all service types. As some of you may know, BT will be changing FTTC to be "wires only" next year, and so a VDSL router will be needed.

We have a small number available now for people to trial - we want to test the routers, our "standard config" and the provisioning process.

Please contact trial@aa.net.uk or #trial on the irc server for more information.

P.S. Obviously it does Internet Protocol, the current one, IPv6 and the old one IPv4

Obviously this initial trial is limited number of routers which we are sending out at no charge to try different scenarios. However, we expect to be shipping these as standard later in the month, and they will be available to purchase on the web site.

13 Feb 15:49:08
Thanks for all the emails and IRC messages about trialling the new routers. We will contact customers back on Monday to arrange shipping of the units.
16 Feb 10:43:54
We now have enough trialists for the new router, we will contact a selection of customers today to arrange delivery of the routers. Thanks
Started 13 Feb 14:25:34

16 Jun 18:24:58
3 Jun 11:30:45

We are now processing FTTC orders as 'wires only'. This means:

  • An Openreach engineer appointment is not required if the phone line already exists
  • Openreach will not provide a modem.
This makes the installation easier to arrange as no onsite appointment is required. There are a number of FTTC (VDSL) modems and routers available on the market, we are providing the ZyXEL VMG1312 router which can act as an all-in-one modem/router or just as a modem in a similar fashion to the original Openreach modem.

Do contact us for more information

Started 3 Jun 11:00:00

10 Jun 02:21:49
10 Jun 10:26:47
It seems that one of our BT links had an issue over night. It was only one of the four links we have and only BT services that were affected. Lines reconnected. If you have not reconnected, try power cycling your router. We don't have any explanation from BT as yet.
Started 10 Jun 02:03:19
Closed 10 Jun 02:21:49

4 Jun 22:52:32
4 Jun 16:53:51
We tend to do server maintenance on Thursday evenings. We'll be doing some security updates on servers this evening which will involve some reboots. These tend to be quick, and so service should only be affected for a few minutes. We'll be rebooting the following servers: Control Pages, Email servers, Web servers as well as a few servers that are not accessed by customers directly.
Resolution This work has been completed.
Started 4 Jun 22:00:00
Closed 4 Jun 22:52:32

1 Jun 23:30:51
1 Jun 23:11:35
We're seeing odd routing blips across our network. We're investigating.
1 Jun 23:31:30
Traffic is back to normal. We'll investigate this further and will update this post in the morning.
Resolution This looked like it was some sort of DOS that affected our routing, possibly other ISPs too.
Started 1 Jun 22:55:00
Closed 1 Jun 23:30:51

12 May 10:27:29
5 May 10:21:17
BT have had trouble with this exchange (http://aastatus.net/2052) but we are now seeing evening congestion on TalkTalk connected lines. We have reported this and will update this post accordingly.
5 May 12:38:47
TalkTalk have fixed a misconfiguration at their end. This should now be resolved. We'll check again tomorrow.
7 May 08:55:48
Lines still seem to be congested, this is being looked in to by TalkTalk.
13 May 12:13:36
Update from TT 'This should now properly be resolved. A faulty interface meant that we were running at 2/3 of capacity - only a few of your subscribers would have suffered last night hopefully but should now all be good.'
Started 5 May 10:00:00
Closed 12 May 10:27:29

2 Jun 11:18:33
2 Jun 11:15:34
At 11:07 one of our links to BT blipped - this caused a number of BT DSL lines to drop. Lines are reconnecting at the moment.
2 Jun 11:18:56
Most lines are now back online.
Started 2 Jun 11:07:00
Closed 2 Jun 11:18:33

27 May 14:48:58
27 May 10:37:50
One of our routers is misbehaving - the one we planned to reload shortly as it happens, and this seems to be impacting mobile data SIMs.
27 May 10:46:58
This appears to be a LINX issue
27 May 10:53:20
We have managed to shut down the port to LINX while this is resolved.
27 May 11:01:41
This looks to be impacting lots of ISPs, but traffic will go via alternative peering and transit whilst LINX have issues and so this should have minimal impact on customers at present.
27 May 12:09:13
LINX are clearly having some major issues on one of the LANs, and we remain disconnected until this is resolved. Once it is resolved, we may reload one of our routers so as to add more logging and allow us to more carefully monitor further issues.
27 May 14:39:14
LINX have rectified the issue, so we will be re-enabling LINX ports shortly.
Resolution All working now.
Started 27 May 10:30:00
Closed 27 May 14:48:58

27 May 10:53:48
27 May 10:12:22
We have been encountering issues with LINX which result in some routes being "Black-holed" (i.e. going nowhere).

The system normally handles this without any problem, identifying an infeasible route and removing from the routing table to allow alternative routes a chance. However we have identified an edge case where this does not happen.

We plan to do a router upgrade today, as this issue is causing customers problems at present. This is one router and should be pretty seamless in that routing should go via other routers whilst it happens.

However, there is a small risk of some routing blips during the process.

Resolution Upgrade completed
Started 27 May 10:00:00
Closed 27 May 10:53:48
Previously expected 27 May 17:00:00

25 May 21:30:00
25 May 21:45:20
From around 6:30pm there was IPv6 routing problems to some hosts via our Linx peering - notably to Google.
Resolution A temporary workaround was applied at 9:30pm and routing has been restored. Staff were alerted to this via customers using the 'MSO' SMS facility, however, do to individual staff circumstances it was not until 9pm that staff were able to respond. We do apologise for the time this took, it is very rare for all the staff that are alerted in this way to all be unavailable at the same time. We'll consider what can be done to improve this.
Started 25 May 18:30:00
Closed 25 May 21:30:00

15 May 14:00:00
15 May 15:29:53
We have had reports of some one way audio calls today. We have not quite got to the bottom of it, and it seems to have gone away. This happened earlier in the week when one carrier had network issues, but we are not convinced that this is the same effect.
Started 15 May 12:00:00
Closed 15 May 14:00:00

13 May 12:16:45
26 Mar 09:53:31

Over the past couple of weeks we have seen FTTC lines drop and reconnect with in increase in latency of around 15ms. This is seen on the monitoring graphs as a thicker blue line.

Upon first glance it looks as if interleaving has been enabled, but a line test shows that this is not the case.

We've been in contact with BT and it does look like BT are rolling out a new profile on to their Huawei DSLAMs in the local green cabinets. It has been expected that BT would be rolling out this new profile, but we didn't expect such an increase in latency.

The profile adds 'Physical Retransmission (ReTX) technology (G.INP / ITU G.998.4)' which helps with spikes of electromagnetic interference and can make lines more stable.

We would hope to have control over the enabling and disabling of this profile, but we don't. Line profiles with FTTC is managed by BT Openreach and are tricky for us and even BT Wholesale to get adjusted.

We're still discussing this with BT and will update this post with news as we have it.

26 Mar 09:55:29
26 Mar 10:48:37
This has been escalated to the head of fibre deployment within BT Wholesale and we are expecting an update by the end of the day.
26 Mar 11:12:08
Further information about G.INP:
  • http://www.ispreview.co.uk/index.php/2015/01/bt-enables-physical-retransmission-g-inp-fttc-broadband-lines.html
  • http://www.thinkbroadband.com/news/6789-impulse-noise-protection-rolling-out-on-openreach-vdsl2.html
  • http://forum.kitz.co.uk/index.php?topic=15099.0
...among others.
27 Mar 16:46:22
BT have asked us for further information which we have provided to them, we don't expect an update now until Monday
9 Apr 14:26:19
This is still ongoing with BT Wholesale and BT Openreach.
16 Apr 15:58:53
This has been escalated to a very senior level within BT and we are expecting a proper update in the next few days.
24 Apr 13:01:45
We have just received the below back from BT on this

Following communications from a small number of BTWholesale FTTC comms providers regarding Openreach’s implementation of Retransmission and the identification of some your customers who have seen increased latency on some lines for some applications since retransmission was applied. Over the last 4 weeks I have been pushing Openreach to investigate,

feedback and provide answers and options related to this issue. As a result attached is a copy of a briefing from Openreach; sent to their CPs today, on how RetX works and what may have caused this increase latency.

This info is being briefed to all BTWholesale customers via our briefing on Saturday morning 25/4/15 but as you have contacted me direct I’m sending this direct as well as providing ann opportunity to participate in a trial.

Openreach have also advise me this afternoon that they intend to run a trial next week (w/c 25/4/15) on a small set of lines; where devices aren’t retransmission compatible in the upstream to see if changing certain parameters removes the latency and maintains the other benefits of retransmission. The exact date lines will be trialled has yet to be confirmed.

However, they have asked if I have any end users who would like to be included in this trail. To that end if you have particular lines you’d like to participate in this trial please can you provide the DN for the service by 17:00 on Monday 28th April so I can get them included.

This is a trial of a solution and should improve latency performance but there is a risk that there may be changes to the headline rate.

5 May 22:28:25
Update to Trial here: https://aastatus.net/2127
Started 26 Mar 09:00:00
Closed 13 May 12:16:45

13 May 12:16:18
12 May 12:50:17
One of our upstream VOIP providers are having some network issues (see below) this may be affecting some VOIP calls and 3G data SIMS.

Affected Systems: Connectivity to some aql core services Affected Customers: Some customers across aql core services Expected Resolution Time:

Dear Customers, We are aware of a connectivity issue in Leeds which engineers are investigating as their top priority. Voice customers may have seen calls drop Customers colocating in Leeds may be experiencing connectivity issues. Resiliency in some services is reduced.

13 May 12:15:45
This issue has now been resolved and below is information from AQL relating to the issue.

It's never a good thing to have to write an email like this. aql operate a resilient core network with an agile edge, allowing changes to be made to accommodate the addition of customers without making any fundamental (or risky) changes to the core network.

Last week, we had an outage related to instability within our core network, due to what we believe is a vendor / O/S bug within some core switch fabric and had started works with the vendor to address the issue with as little impact as possible.

aql have strong procedures and processes to minimise operational and network impact by way of planning core works or changes to core network to have little or no disruption, to be performed during silent hours and to be announced well in advance.

In this instance, regrettably, a senior member of staff did not follow these procedures and that is now under investigation. Please be assured we take such matters seriously and will take all necessary measures to ensure that there cannot be a repeat incident.

As with all incidents, aql prepares a full RFO "Reason for Outage", within 48 hours of any incident. If you require a copy of this, please make contact with your account manager.

Both personally and on behalf of aql, my sincere apologies - You should just be able to rely on us to do our part and you can then concentrate on your business.

Closed 13 May 12:16:18