Order posts by limited to posts

Yesterday 13:25:38
11 Mar 11:39:17
We are seeing some evening time congestion on all BT 21CN lines that connect through BRAS's 21CN-BRAS-RED1-MR-DH up to 21CN-BRAS-RED13-MR-DH I suspect one of the BT nodes is hitting limits some evenings as we don't see the higher latency every night. This has been reported into BT and we will update this past as soon as they respond back.
11 Mar 11:44:06
Here is an example graph
12 Mar 12:00:45
This has been escalated further to the BT networK guys and we can expect an update within the next few hours.
17 Mar 15:41:18
Work was done on this overnight so I will check again tomorrow morming and post another update.
18 Mar 11:38:25
The changes BT made over night have made a significant difference to the latency, still seeing it slightly higher than we would like so we will go back to then again.
19 Mar 14:54:44
Unfortunately the latency has increased again so whatever BT did two nights ago has not really helped. We are chasing again now.
23 Mar 14:07:53
BT have still not pinpointed the issue so it has been escalated further.
27 Mar 13:03:38
Latency is hardly noticeable now but we are still chasing BT on sorting the actual issue, update will be mOnday now.
30 Mar 10:04:14
BT have advised that they are aware of the congestion issue at Manchester, and the solution they have in place is to install some additional edge routers, they are already escalating on this to bring the date in early, currently the date is May. Obviously May is just not acceptable and we are doing all we can to get BT to bring this date forward.
2 Apr 12:28:39
We have requested a further escalation within BT, the time scales they have given for a fix is just not acceptable.
13 Apr 15:12:23
The last update from BT was 'is latency issue has been escalated to high level. BT TSO are currently working on a resolution and are hoping to move into the testing phase soon. We will keep you updated as we get more information' I am chasing for another update now.
16 Apr 16:01:15
We are still chasing BT up on bringing the 'fix' forward. Hopefully we will have another response by the morning.
Yesterday 13:25:21
The latest update from BT: We have identified a solution to the capacity issue identified and are looking to put in a solution this Friday night...
Started 11 Mar 01:35:37
Update expected Friday 14:25:25

16 Apr 15:59:17
26 Mar 09:53:31

Over the past couple of weeks we have seen FTTC lines drop and reconnect with in increase in latency of around 15ms. This is seen on the monitoring graphs as a thicker blue line.

Upon first glance it looks as if interleaving has been enabled, but a line test shows that this is not the case.

We've been in contact with BT and it does look like BT are rolling out a new profile on to their Huawei DSLAMs in the local green cabinets. It has been expected that BT would be rolling out this new profile, but we didn't expect such an increase in latency.

The profile adds 'Physical Retransmission (ReTX) technology (G.INP / ITU G.998.4)' which helps with spikes of electromagnetic interference and can make lines more stable.

We would hope to have control over the enabling and disabling of this profile, but we don't. Line profiles with FTTC is managed by BT Openreach and are tricky for us and even BT Wholesale to get adjusted.

We're still discussing this with BT and will update this post with news as we have it.

26 Mar 09:55:29
26 Mar 10:48:37
This has been escalated to the head of fibre deployment within BT Wholesale and we are expecting an update by the end of the day.
26 Mar 11:12:08
Further information about G.INP:
  • http://www.ispreview.co.uk/index.php/2015/01/bt-enables-physical-retransmission-g-inp-fttc-broadband-lines.html
  • http://www.thinkbroadband.com/news/6789-impulse-noise-protection-rolling-out-on-openreach-vdsl2.html
  • http://forum.kitz.co.uk/index.php?topic=15099.0
...among others.
27 Mar 16:46:22
BT have asked us for further information which we have provided to them, we don't expect an update now until Monday
9 Apr 14:26:19
This is still ongoing with BT Wholesale and BT Openreach.
16 Apr 15:58:53
This has been escalated to a very senior level within BT and we are expecting a proper update in the next few days.
Started 26 Mar 09:00:00
Update was expected 19 Apr 09:00:01

13 Apr
9 Apr 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr

3 Apr
2 Apr 15:48:08
We expect to do some router upgrades, including normal rolling LNS upgrades over the next week as a new release of the FireBrick is expected to be released shortly. This should have little or no disruption, as usual.
9 Apr 12:52:05
This was a bit delayed and should start tonight, and be ongoing in to the weekend.
Started 3 Apr
Previously expected 13 Apr

2 Apr 13:16:30
09 Dec 2014 11:20:04
Some lines on the LOWER HOLLOWAY exchange are experiencing peak time packet loss. We have reported this to BT and they are investigating the issue.
11 Dec 2014 10:46:42
BT have passed this to TSO for investigation. We are waiting for a further update.
12 Dec 2014 14:23:56
BT's Tso are currently investigating the issue.
16 Dec 2014 12:07:31
Other ISPs are seeing the same problem. The BT Capacity team are now looking in to this.
17 Dec 2014 16:21:04
No update to report yet, we're still chasing BT...
18 Dec 2014 11:09:46
The latest update from this morning is: "The BT capacity team have investigated and confirmed that the port is not being over utilized, tech services have been engaged and are currently investigating from their side."
19 Dec 2014 15:47:47
BT are looking to move our affected circuits on to other ports.
13 Jan 10:28:52
This is being escalated further with BT now, update to follow
19 Jan 12:04:34
This has been raised as a new reference as the old one was closed. Update due by tomorrow AM
20 Jan 12:07:53
BT will be checking this further this evening so we should have more of an update by tomorrow morning
22 Jan 09:44:47
An update is due by the end of the day
22 Jan 16:02:24
This has been escalated further with BT, update probably tomorrow now
23 Jan 09:31:23
we are still waiting for a PEW to be relayed to us. BT will be chasing this for us later on in the day.
26 Jan 09:46:03
BT are doing a 'test move' this evening where they will be moving a line onto another VLAN to see if that helps with the load, if that works then they will move the other affected lines onto this VLAN. Probably Wednesday night.
26 Jan 10:37:45
there will be an SVLAN migration to resolve this issue on Wednesday 28th Jan.
30 Jan 09:33:57
Network rearrangement is happening on Sunday so we will check again on Monday
2 Feb 14:23:12
Network rearrangement was done at 2AM this morning, we will check for paclet loss and report back tomorrow.
3 Feb 09:46:49
We are still seeing loss on a few lines - I am not at all happy that BT have not yet resolved this. A further escalation has been raised with BT and an update will follow shortly.
4 Feb 10:39:03
Escalated futher with an update due at lunch time
11 Feb 14:14:58
We are getting extremly irritated with BT on this one, it should not take this long to add extra capaity in the affected area. Rocket on it's way to them now ......
24 Feb 12:59:54
escalated further with BT, update due by the end of the day.
2 Mar 09:57:59
We only have a few customers left showing peak time packet loss and for now the fix will be to move them onto another MSAN, I am hoping this will be done in the next few days. We really have been pushing BT hard on this and other areas where we are seeing congestion. I am please that there are now only a handful of affected customers left.
17 Mar 11:21:33
We have just put a boot up BT on this, update to follow.
2 Apr 13:16:10
BT have still not fixed the fault so we have moved some of the affected circuits over to TalkTalk and I am pleased to say that we are not seieng loss on those lines. 100% this is a BT issue and I am struggling to understand why they have still not tracked the fault down.
Update was expected 6 Apr 14:16:15
Previously expected 1 Feb 09:34:04 (Last Estimated Resolution Time from AAISP)

31 Mar 17:00:00
31 Mar 16:24:26
Over the next few days we are working on some minor changes to the way we handle passwords on the control pages (clueless).

At these first stages you should see no impact, but there is a risk of issues, and we would ask anyone with problems logging in to control pages, changing passwords, or logging in to DSL, SIMs, etc, to let us know.

Each stage is being tested on our test system and then deployed, with the first stage expected to be updated tonight.

The final stage will mean a change on where passwords are visible, and the processes for issuing and changing passwords. We'll post more details closer to the time.

This is all part of ongoing work to improve security. Thank you for your understanding.

2 Apr 15:53:01
The first stage seems to have gone well - our test/monitoring has been working well to help us check any anomalies and ensure consistency.

The next stage should be equally harmless as it means changing over various systems to use the new password hashes. We plan to work on this over Easter.

We will then go on to change the way passwords are issued when ordering and updated when customers wish to change them.

This work is all part of general review and update of security for passwords on our various systems. Thank you for your understanding.

3 Apr 12:55:34
We are progressing with updates - the login to the control pages is now switched over to the new hashes - any issues, please let me know on irc, but all looks good from here.

The RADIUS logins have changed over as well, to use line based passwords (which are same as control pages login passwords at present). Again, please let us know any issues, but so far all looks well.

The next step later today will be a change to how you change the password on the control pages - this will move to the same system we use on the accounts pages - an emailed link that offers a new passwords via https. This is safer than plain text emailed passwords.

Once that is complete, we plan to update he way passwords are issued when ordering new services, which will hopefully be done later today.

There will then be more testing and cleaning up to be done later.

4 Apr 06:53:08
The first side effect that has been noted is that passwords on the control pages are now case-sensitive. Sorry for any confusion this may have caused.
4 Apr 11:19:58
We expect the work for there weekend to stop now - with more later in the week or next weekend. We are at a stage now that we need to provide some clear documentation on the different levels of passwords and what levels of protection are provided for these in our systems.
5 Apr 13:06:22
We are going ahead with more of the work this weekend now, and expect to separate control page login passwords from Line/DSL login passwords today or tomorrow. We'll post more details once the work is complete. We are currently running tests on out tests systems now.
5 Apr 13:50:44
We have now separated login passwords and line passwords.

Any issues, please let us know.

Started 31 Mar 17:00:00
Previously expected 7 Apr

12 Mar 09:48:01
12 Mar 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.

19 Jan 16:08:37
17 Jul 2014 10:08:44
Our email services can learn spam/non-spam messages. This feature is currently down for maintenance as we work on the back-end systems. This means that if you move email in to the various 'learn' folders they will stay there and will not be processed at the moment. For the moment, we advise customers not to use this feature. Will will post updates in the next week or so as we may well be changing how this feature works. This should not affect any spam scores etc, but do contact support if needed.
29 Jul 2014 11:42:12
This project is still ongoing. This should not be causing too many problems though, as the spam checking system has many many other ways to determine if a message is spam or not. However, for now, if customers have email that is miss-classified by the spam checking system then please email the headers in to support and we can make some suggestions.
19 Jan 16:08:37
We are working on rebuilding the spam learning system. We expect to make this live in the next couple of weeks.
Started 17 Jul 2014 10:00:00
Update was expected 29 Jan 13:00:00

8 Jan 12:39:06
8 Jan 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
9 Jan 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
12 Jan 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 8 Jan 12:39:06 by AA Staff
Previously expected 15 Jan 17:00:00

03 Jun 2014 17:00:00
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

12 Feb 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

Monday 09:54:30
Monday 09:54:30
Customers will have received an email from us. Apologies for not PGP signing it. It asks you to go to a secure link on our control pages and confirm (one click) that you consent to receive notices via email.

Yes, I know it is crazy, and it is already part of our terms, and you already know we email notices, and that this email is a notice we have emailed you... Sorry but OFCOM insist we get *explicit* consent to send some notices we send.

We'd appreciate it if you just click the link and then the confirm button.

We'll email you again if you don't, sorry. If you are not happy about this, please do complain to OFCOM. Thank you.

Yesterday 18:23:01
We have resent the email to all of those that have not followed the link and confirmed. This time, PGP signed. Sorry for any concern the previous email caused.
Yesterday 18:29:59
I'd also like to thank the *thousands* of people that have confirmed their consent so far.
Started 19 Apr
Expected close 1 Jun

17 Apr 15:54:16
15 Apr 13:16:15
Some customers on the Bradwell Abbey exchange are currently experiencing an outage. We have received reports from FTTP customers, however this may also affect customers using other services. BT have advised that they are currently awaiting a delivery for a new card at this exchange. We will chase BT for updates and provide them as we receive them.
15 Apr 15:47:41
I have requested a further update from BT.
16 Apr 08:07:15
Openreach AOC and PTO are investigating further at this time. We will reach out for an update later today.
16 Apr 10:32:55
BT have advised that a Cable down is the root cause at this time.
16 Apr 15:51:50
PTO are still onsite. I have asked for an ECD, however OpenReach are not supplying that information, due to being fibre work.
17 Apr 10:19:33
OpenReach have stated, they are hoping for a completion on the fibre today and resource is being tasked out. OpenReach have stated this is only an estimate and not set in stone.
17 Apr 14:49:46
Some customers are reporting a restored service. BT advise that teams are still on site to resolve this P1 issue.
17 Apr 15:55:25
The cable down issue affecting customers using the Bradwell Abbey exchange has now been resolved.
Started 15 Apr 12:55:00 by AAISP Staff
Closed 17 Apr 15:54:16
Cause BT

16 Apr 16:00:24
27 Mar 14:03:52
We are seeing packet loss on all lines connected through 21cn-BRAS-RED8-SL the loss is all through the day/night started 10:08 on the 25th. This has been reported to BT
27 Mar 14:07:22
Here is an example graph:
30 Mar 14:37:04
BT claimed to have fixed this but our monotoring is still seeing the loss, BT chased further
Broadband Users Affected 0.01%
Closed 16 Apr 16:00:24

16 Apr 15:59:33
2 Feb 10:10:46
We are seeing low level packet loss on BT lines connected to the Wapping exchange - approx 6pm to 11pm every night. Reported to BT...
2 Feb 10:13:57
Here is an example graph:
3 Feb 15:55:40
Thsi has been escalated further with BT
4 Feb 10:27:37
Escalated further with BT, update due after lunch
11 Feb 14:18:00
Still not fixed, we are arming yet another rocket to fire at BT smiley
24 Feb 12:58:51
escalated further with BT, update due by the end of the day.
2 Mar 10:00:11
Again the last few users seeing packet loss will be moved onto another MSAN in the next few days.
12 Mar 12:02:57
Updatew expected in the next few hours
17 Mar 11:19:48
A further escalation has been raised on this, update by the end of the day
30 Mar 15:35:32
This has been escalated to the next level
Broadband Users Affected 0.09%
Started 2 Feb 10:09:12 by AAISP automated checking
Closed 16 Apr 15:59:33

13 Apr 15:01:38
13 Apr 14:51:55
There was an issue with two of our routers - a few lines dropped, and are reconnecting. Routing was affected for a minute or two. We're investigating.
Resolution Service has recovered as expected. We'll see if we can find the underlying cause. Sorry for any inconvenience.
Started 13 Apr 14:46:46
Closed 13 Apr 15:01:38
Previously expected 13 Apr 14:50:00

2 Apr 16:02:22
17 Mar 12:38:27
We are seeing higher than normal evening time latency on the Wrexham exchange, it is not every night but it does suggest BT are running another congested link. This has been reported to them and we will update thia as and when they get back to us.
17 Mar 12:41:51
Here is an example graph:
20 Mar 14:36:18
It has looked better the last two eveings but it's still being investigated as the BT links were probably less busy.
Broadband Users Affected 0.01%
Started 15 Mar 12:36:07 by AAISP Staff
Closed 2 Apr 16:02:22

2 Apr 11:57:32
1 Apr 10:00:06
Some customers connected through Gloucestershire are affected by an ongoing TalkTalk major service outage. Details below: Summary

Network monitoring initially identified total loss of service to all customers connected to 2 exchanges in the Gloucester area. Our NOC engineers re-routed impacted traffic whilst virgin media engineers carried out preliminary investigations. Virgin media restoration work subsequently resulted in several major circuits in the Gloucester area to fail.

This has resulted in a variety of issues for multiple customers connected to multiple exchanges. Our NOC engineers have completed re-routing procedures to restore service for some customers with other customers continuing to experience total loss of service due to capacity limitations. Impact: Tigworth, Witcombe and Painswick exchanges

Hardwicke and Barnwood exchanges – experiencing congestion related issues.

Cheltenham and Churchdown – experiencing congestion related issues.

experiencing congestion related issues Stroud, Stonehouse, Whitecroft, Blakeney, Lydney, Bishops Cleeve, Winchcombe, Tewkesbury, Bredon exchanges.

1 Apr 10:31:30
TT have advised that splicing of the affected fibre is still ongoing. There is no further progressive updates at this time. Further updates will be sent out shortly.
2 Apr 11:57:26
Root cause analysis identified a major Virgin Media fibre break due to third party contractor damage as being the cause of this incident. Service was fully restored when Virgin Media Fibre engineers spliced new fibre. Following this we received confirmation that service had returned as BAU. TalkTalk customers would have been automatically rerouted and would have experienced only a momentary Loss of Service. An observation period has been carried out verifying network stability and as no further issues have been reported this incident will be closed with further investigations into the cause being tracked via the Problem Management process
Closed 2 Apr 11:57:32

1 Apr 09:01:39
1 Apr 09:01:39

We have extended our support hours, which are now 8am to 6pm, Mon-Fri, except (English) public holidays.

Previously we worked 9am to 5pm, and sales/accounts still do. However the support staff can usually address simple/urgent queries in those areas if necessary.

Occasionally we do have people ask why we only work office hours, and it is worth trying to explain this. Many ISPs do, indeed, have 24 hour telephone support, for example.

For most of our services, there are faults that come in two flavours. Either there is some big issue (a major outage), in which case we have staff, getting involved in fixing things whatever time it is, or an individual line fault for DSL. It is pretty rare to have individual faults for VoIP, SIMs, etc, but you can, of course, get line faults for DSL.

When it comes to individual DSL faults, there are a load of things people can do at home/office to eliminate equipment and test for themselves, and we offer various on-line tests via our control pages.[1] This can help resolve things. But the issues that don't just go away, and would require support staff to do something, are almost always something that needs a BT engineer to go out.

With very few exceptions, BT engineers are not going to be going our any quicker if we book them next working day at 9am. So having support staff take calls in the middle of the night would not usually be any help. We also have no intention of farming support out to call centres following scripts.

However, we have decided to extend the hours a bit. The reasons being :-

  • BT engineers work 8am to 6pm normally, and so we can help address any issues that come up with an engineer visit, and talk to the engineer or our customer about it at the time. This has already been seen with some 8am visits by engineers who are confused by the notes and need us to explain.
  • Starting at 8am gives customers a chance to resolve issues that can be resolved by talking to support before the usual working day for most people. If it is a line fault, that is not much help, but if it is a matter of swap a router or reboot something, we can offer the necessary advice before you have an office full of people that cannot work.
  • Starting at 8am and finishing a 6pm allows a lot of people that work during the day to contact us from home where they have an issue with their home broadband. We know some customers appreciate that.
  • We have increased the number of support staff, allowing some staggered working hours so that we can offer this.
But please do bear in mind, we do have irc[2], with a simple web front end, which can offer various help and advice by staff and other customers at all sorts of times. It is informal support from staff outside normal hours, but is usually available. We are thinking of perhaps extending this to be more formal evening irc support at some point maybe, with a rota of some sort.

Obviously we're interested in feedback on how the new support hours work for customers.

Started 1 Apr 09:00:00

31 Mar 22:12:13
31 Mar 22:05:19
The Control Pages are currently offline and affecting some other services, eg SIP2SIM registrations. This is being looked in to at the moment.
31 Mar 22:12:32
Service is back to normal.
Started 31 Mar 21:50:00
Closed 31 Mar 22:12:13

27 Mar 09:00:00
25 Mar 21:48:13
Since the 24th March we have been seeing congestion on TalkTalk lines on the Shepherds Bush exchange. This has been reported to TalkTalk. Example graph:
26 Mar 10:51:48
TalkTalk say they have fixed this. We'll be checking overnight to be sure smiley
26 Mar 22:27:33
Lines are looking good.

We had this feedback from TalkTalk regarding this congestion issue:

Shepherds Bush has three GigE backhauls to two different BNG's - there was some a software process failure and restart on one of these devices on Tuesday morning which had two of the three backhauls homed to it. As a result all customers redialled to the one 'working' BNG in the exchange - normally when this happens we will calculate whether or not the backhaul can handle that number of customers and if not manually intervene, in this case however a secondary knock on issue meant that our DHCP based customers (FTTC subs) were sent through the same backhaul and the calculation was inaccurate.

If the PPP session was restarted they would have reconnected on their normal BNG and everything should be OK - we've just made this change manually moving subscribers over - still have a couple of lines on the backup BNG so will monitor if there are any issues and take any necessary actions to resolve.

Started 24 Mar 17:00:00
Closed 27 Mar 09:00:00

26 Mar 14:16:54
4 Feb 10:55:10
One of our carriers (AQL) will be doing some maintenance on their SMS platform Wednesday 11th February between 10:00 - 11:00. This is to load new firmware on some routers but no loss of service is expected. This is advisary only.
Started 4 Feb 10:51:46
Previously expected 12 Feb 11:00:00

25 Mar 15:10:21
[Email and Web Hosting] - SSL Certificates Updated - Info
25 Mar 15:08:57
We're updating SSL certificates for our email servers this afternoon, including webmail. The old serial number is 0FA016. The new serial number is 106E03. Users who don't have the CAcert root certificate installed may see errors. Details on http://aa.net.uk/cacert.html

23 Mar 10:38:00
23 Mar 09:51:01
We are investigating a problem with some incoming calls, our engineers are busy investigating this now. Apologise for the inconvenience.
23 Mar 10:01:55
We have raised this with the carrier in question, they are investigating.
23 Mar 10:04:39
The carrier have confirmed that they have a fault.
23 Mar 10:09:20
Customers can call us on:
  • Sales: 05555 400 000
  • Support 05555 400 999
for the time being.
23 Mar 10:12:32
Carrier say "Our engineers are working on this as a top priority"
23 Mar 10:33:16
Carrier says: "initial investigations point to memory issues and we are investigating that further. As a temporary solution we are attempting to update our internal routing to mitigate the impacts."
23 Mar 10:37:35
Incoming calls seem to be working ok now.
Resolution Calls working again. Fault was caused by upstream carrier and affected may VoIP services across the UK. The carrier has provided a 'Reason for Outage' document: http://www.aql.com/downloads/RFO_20150323_aql.pdf
Closed 23 Mar 10:38:00

17 Mar 11:18:55
20 Jan 12:53:37
We are seeing low level packet loss on some BT circuits connected to the EUSTON exchange, this has been raised with BT and as soon as we have an update we will post an update here.
20 Jan 12:57:32
Here is an example graph:
22 Jan 09:02:48
We are due an update on this one later this PM
23 Jan 09:36:21
BT are chasing this and we are due an update at around 1:30PM.
26 Jan 09:41:39
Work was done over night on the BT side to move load onto other parts of the network, we will check this again this evening and report back.
27 Jan 10:33:05
We are still seeing lines with evening packet loss but BT don't appear to understand this and after spending the morning arguing with them they have agreed to investigate further. Update to follow.
28 Jan 09:35:28
Update from BT due this PM
29 Jan 10:33:57
Bt are again working on this but no further updates will be given until tomorrow morning
3 Feb 16:19:06
This one has also been escalated further with BT
4 Feb 10:18:11
BT have identified a fault within their network and we have been advised that an update will be given after lunch today
11 Feb 14:16:56
Yet another rocket on it's way to BT
24 Feb 12:59:20
escalated further with BT, update due by the end of the day.
2 Mar 09:59:19
STill waiting for BT to raise an emergency PEW, the PEW (planned engineering work) will sort the last few lines where we are seeing peak time packet loss)
12 Mar 12:03:57
I need to check this tonight as bT think it is fixed, I will post an update back tomorrow
Broadband Users Affected 0.07%
Started 10 Jan 12:51:26 by AAISP automated checking
Closed 17 Mar 11:18:55
Previously expected 21 Jan 16:51:26

17 Mar 10:08:05
17 Mar 10:08:05

For the past year or so we have been offering 2 'CSS styles' for the Control Pages. Today we are changing the default style to the new one. This means that most customers will now be using the new style.

You can see the two styles on this page: http://wiki.aa.org.uk/CSS

Feedback is welcome, pop us an email to webmaster@aa.net.uk

Started 17 Mar 09:55:00

13 Mar 22:57:53
13 Mar 22:44:00
TalkTalk lines lost connection at about 10:30, and are reconnecting at the moment.
13 Mar 22:47:49
About 80% of lines have now reconnected.
13 Mar 22:58:06
Most lines are back.
Resolution Confirmed as a fault within the TalkTalk network that affected us and other ISPs.
Started 13 Mar 22:30:00
Closed 13 Mar 22:57:53

13 Mar 13:12:51
13 Mar 12:44:14
Sorry for the short notice, the Control Pages will be offline for a few minutes this afternoon whilst we carry out some work on the hardware. (no, we are not installing black boxes!). Services will be unaffected, but customer won't be able to access the Control Pages during this time.
13 Mar 13:06:06
This work has started, we expect the Control Pages to be back in 10-15 minutes.
13 Mar 13:13:16
Work has completed successfully
Started 13 Mar 13:00:00
Closed 13 Mar 13:12:51

12 Mar 18:30:05
12 Mar 18:24:06
We allow users to provide address/location data to pass to emergency services in the event of a 999 calls.

This goes via BT and BT are being a pain at present. As we have explained the data is simply a matter of what customers provide. VoIP has no inherent way to provide an accurate geographic location of callers. But BT are unhappy with some of the data and seem to have blocked us sending updates this week.

They even suggested that they have to send OFCOM reports of incomplete and incorrect data as some sort of threat. We know a lot of people have multiple locations from which they make calls. We do this purely as a good will gesture and in the spirit of GC4, because we don't have location data in the first place. So if BT do not stop buggering about soon we'll be reporting BT to OFCOM instead.

Anyway, the upshot is that a few customers that have provided location data over that last few days are not yet updated to BT and hence the emergency services. We are working to resolve this and hope to have it working again soon.

The config pages for your VoIP set up show if there is an issue and you get an email once it is all confirmed. In the mean time, we do apologise for any concern this may cause.

None of our services are provided to be used in any "safety of life" circumstances, as per our standard terms.

16 Mar 16:49:34
We are hoping to have this sorted, perhaps today.

It is likely that anyone with existing data registered may get a new email at some point confirming the data as we are planning to re-send everything from scratch just to be sure.

We would ask that customers review what they have entered and ensure it is as accurate as possible to assist emergency services in the event of a 999 call.

18 Mar 17:40:37
Data is updating again to BT. We would recommend customers ensure accurate data on the control pages for 999 calls. We will review this over the next few days, and customers may still get update emails in due course.
Started 8 Mar
Previously expected 16 Mar

12 Mar 12:01:00
12 Mar 09:50:23
TalkTalk have a a fibre outage affecting EoFTTC lines in the East Midlands and East Anglia area. Lines will be in sync, but will not have any further connectivity. TalkTalk are aware and have engineers investigating. They have their own status post here: http://solutions.opal.co.uk/network-status-report.php?reportid=6291
12 Mar 10:18:51
Not further updates as yet.
12 Mar 10:20:37
Update from TT: Our NOC and Network Support Engineers have confirmed that corrective work to restore services to all impacted exchanges remains on-going.
12 Mar 11:44:30
Update from TalkTalk: "Our NOC Engineers have confirmed that all network alarms have cleared and normal working service is now fully restored to all impacted exchanges and customers. There are continued reports of some B2B circuits continuing to experience issues and our IP Engineers are currently investigating these reports. Incident Management will continue to liaise with our NOC Engineers and further updates will be provided upon receipt."
12 Mar 14:20:48
Service has been restored.
Started 12 Mar 00:10:00
Closed 12 Mar 12:01:00

6 Mar 13:00:00
5 Mar 10:54:01
We are seeing quite a few lines on the Durham and NEW BRANCEPETH exchange with a connection problem. Customers may have no internet access and their router constantly logging in and out.

We have reported this fault to BT and they are investigating. It looks like the BRAS has a fault.

5 Mar 16:25:28
We have been chasing this with BT throughout the day; their tech team are still investigating.
6 Mar 09:16:19
We still have a few lines off and we are on the phone to BT now chasing this. Update to follow.
6 Mar 13:48:38
BT appear to have a broken LAG (link aggregation group) and as a work around we have had to change one of our end point IP addresses and the affected customers are back on line. This is just a work around and hopefully BT will shortly fix their end.
Started 5 Mar 02:00:00
Closed 6 Mar 13:00:00

7 Mar 16:33:26
7 Mar 16:33:26
We have been updating the systems that handle advising BT of location data for 999 calls, and this has resulted in a number of emails yesterday confirming previously entered details.

Sorry for any confusion caused.

Started 6 Mar
Previously expected 7 Mar

6 Mar 18:14:21
6 Mar 16:05:00
We have identified that Ethernet customers are not seeing archived graphs for last few days and the cause is part of the recent router upgrade code. We plan to carry out a further upgrade this evening which should have little or no disruption, and should rectify this issue.
6 Mar 16:53:29
There may be a blip on some TalkTalk lines during this upgrade.
Resolution Work completed for now - seems to have gone as planned. There was a small TalkTalk line blip as expected.
Started 6 Mar 17:00:00
Closed 6 Mar 18:14:21
Previously expected 6 Mar 19:00:00

5 Mar 18:15:01
4 Mar 17:44:38
Work this evening went pretty much as planned - we added extra diagnostics to try and identify something unexpected in routing. We are going to do some more investigation tomorrow. We expect there to be little or no impact on services once again.
Resolution Work completed, again with minimal disruption.
Started 5 Mar 17:00:00
Closed 5 Mar 18:15:01
Previously expected 5 Mar 18:00:00

28 Feb 14:11:01
27 Feb 12:30:57
We're investigating a problem with VoIP audio problems, calls breaking up etc. This looks to be some packetloss somewhere between us and our carriers. We'll update this post again shortly.
27 Feb 13:04:21
We have identified the cause of this packetloss and are looking in to fixing it.
27 Feb 14:55:30
We're working closely with a 3rd party that is involved in a BGP traffic problem between us and them. This is taking longer to get to the bottom of that we first thought.
27 Feb 15:22:14
As we and the other BGP peer have not been able to get to the root cause of the problem we have put in a temporary fix. This has brought traffic levels back down to normal.
27 Feb 16:26:46
Surprisingly, the problem has come back even though peering has been disabled! Needless to say, we are investigating again!
27 Feb 17:00:05
The problem has gone away again whilst it was being looked in to.
27 Feb 17:04:04

It's worth us explaining the problem... We have a peer at LINX that is sending us lots of traffic. This traffic is not for us, but for someone completely different - a different country even. Even through we have stopped the peering to this 3rd party, the traffic is still being sent, intermittently. This is causing our links to be filled, and hence causes packet loss.

We have been in direct contact with the 3rd party all afternoon, and we and they are confused as to how this is happening. At the point in time, we suspect some kind of router memory corruption which is causing the router to send traffic to the wrong peer. This type of problem is difficult to prove, and so it is taking time to get to the bottom of it.

We are still in contact with the 3rd party and will work to resolve this with them.

Resolution We were able to stop the floods of traffic yesterday afternoon, as a temporary measure, but the underlying problem remained until 10am Saturday when the LINX facing card at the peer was reset after the issue was reported by other LINX members. It is a shame that this was not done yesterday. This does confirm that it was to just AAISP that was affected by this. We will be working on contingency plans to allow us to react more efficiently for something like this in future. Thank you all for your understanding.
Started 27 Feb 12:15:00
Closed 28 Feb 14:11:01

4 Mar 17:41:52
4 Mar 12:48:36
We plan to upgrade a couple more routers this evening. It should have no impact for most of these, but one of them is likely to cause a slight blip on TalkTalk lines.
Resolution The work has been completed and without the TalkTalk blip. Some routing was a bit odd to/from Maidenhead, and we are looking in to that, but the changes seem to have worked so far. We expect we'll do something similar tomorrow.
Started 4 Mar 17:00:00
Closed 4 Mar 17:41:52
Previously expected 4 Mar 18:00:00

4 Mar 10:03:07
2 Mar 06:17:18
Many of our customer broadband lines suffered a blip just after 2am. We're still investigating that, but it seems our RADIUS accounting got behind due to the high number of lines flapping. It looks like accounting has caught up now, but it will mean that we sent out some delayed notifications over night. This could result in, for example, line down/up notification emails delayed by several hours. The time stamp in the notification should show if this is the case.
Closed 4 Mar 10:03:07

3 Mar 19:46:10
3 Mar 19:29:41
We are planning a minor upgrade on a router this evening which should have no impact on normal working.
Resolution Updates done
Started 3 Mar 19:29:03
Closed 3 Mar 19:46:10
Previously expected 3 Mar 23:29:03

18 Mar 17:39:00
26 Feb 18:07:10
We have a couple of minor changes to VoIP servers planned. At present we are testing these.

The changes are subtle but we hope will assist in working with customers with asterisk boxes. We have also updated the screen setting on Remote-Party-Id to reflect trust in the CLI.

The actual update is likely to be over the weekend after some testing tomorrow. Any issues, please let support know.

27 Feb 13:57:41
Due to a number of other incidents today we have decided to delay testing of new features until next week with a roll out of the new features after that. Anyone wanting to test with asterisk, etc, please do let support know (ideally on irc).
9 Mar 07:20:13
Hopefully this is going to be sorted this week.
12 Mar 13:28:12
The "a" voiceless server is being upgraded later today. Any issues, please let us know.
13 Mar 13:43:57
Planning update "b" voiceless this evening, as no issues reported yet...
17 Mar 14:01:03
The upgrade worked with no problems in what was actually being changed (a feature to assist when working with asterisk boxes) but it caused an slight unexpected side effect. As such we expect to update again this evening.
Resolution Things seem fine, closing this status.
Started 28 Feb
Closed 18 Mar 17:39:00
Previously expected 14 Mar

27 Feb 09:51:04
[Email and Web Hosting] - Problems with our webserver - Closed
27 Feb 09:06:37
We are investigating a problem with some of our web hosting customers with a .co.uk domain seeing a 404 Page not found message. Engineers are investigating this now.
27 Feb 09:27:46
We are restoring missing files from our nightly backup, these should be restored in the next hour or so. We're not sure yet what has caused some websites to be removed. We are still investigating and will update this post shortly.
27 Feb 09:33:21
This looks to have been caused by one of our nightly jobs that tidies web site files of ceased domains. This script looks to have been over zealous! We do apologise.
Resolution Missing web sites have been restored. We are aware of what caused this problem - our nightly maintenance scripts, which we will be fixing! Again, we do apologise for this.
Started 27 Feb 09:05:19
Closed 27 Feb 09:51:04

24 Feb 13:00:22
2 Feb 11:21:40

Below is a list of exchanges that BT plan to WBC enable between April - June this year.

This is for information only and we will attempt to bulk migrate customers to the 21CN network as and when they are enabled.


24 Feb 12:57:31
11 Feb 10:17:36
We are seeing evening congestion on the Wrexham exchange, also off that two other BRAS's are affected. They are: 21CN-BRAS-RED6-SF 21CN-BRAS-RED7-SF Customers can check which BRAS/exchange they are connected to from our control pages
11 Feb 10:27:08
Here is an example graph:
13 Feb 11:39:14
We are chasing BT for an update and as soon as we have further news we wil update this post.
16 Feb 10:20:43
It looks like the peak time latency just went away Thursday evening with no report from BT that they actually changed something. We will continue monitoring for the next few days to ensure it really has gone away.
Broadband Users Affected 0.05%
Started 11 Feb 10:12:08 by AA Staff
Closed 24 Feb 12:57:31

13 Feb 15:31:16
13 Feb 14:29:06
We currently supply the Technicolor TG582 for most ADSL services, but we are considering switching to a new router, the ZyXEL VMG1312-B

It is very comprehensive and does both ADSL and VDSL as well as bridging and wifi. It means we can have one router for all service types. As some of you may know, BT will be changing FTTC to be "wires only" next year, and so a VDSL router will be needed.

We have a small number available now for people to trial - we want to test the routers, our "standard config" and the provisioning process.

Please contact trial@aa.net.uk or #trial on the irc server for more information.

P.S. Obviously it does Internet Protocol, the current one, IPv6 and the old one IPv4

Obviously this initial trial is limited number of routers which we are sending out at no charge to try different scenarios. However, we expect to be shipping these as standard later in the month, and they will be available to purchase on the web site.

13 Feb 15:49:08
Thanks for all the emails and IRC messages about trialling the new routers. We will contact customers back on Monday to arrange shipping of the units.
16 Feb 10:43:54
We now have enough trialists for the new router, we will contact a selection of customers today to arrange delivery of the routers. Thanks
Started 13 Feb 14:25:34

15 Feb 12:24:53
12 Feb 18:00:36
The Technicolor routers we supply have a factory default config which connects to us and operates a default setup. This is applied if someone uses the RESET button on the routers.

We have identified that a few dozen of these routers are in this state, which is not correct.

As part of work we are doing for some new routers we plan to start shipping soon, we have made it that any router logging in using the factory default will automatically be updated to have the correct config.

This will likely cause the graph to reset and the LNS in use depends on the login, but it will also set the WiFi SSID correctly, and other parameters. So users may see a change.

If you have any issues, do contact support.

Resolution The change in process has meant a number of routers have been auto-provisioned as expected, the remainder will when they next connect. Any issues, do contact support.
Started 12 Feb 17:00:00
Closed 15 Feb 12:24:53
Previously expected 14 Feb

5 Feb 13:07:52
8 Jan 15:44:04
We are seeing some levels of congestion in the evening on the following exchanges: BT COWBRIDGE, BT MORRISTON, BT WEST (Bristol area), BT CARDIFF EMPIRE, BT THORNBURY, BT EASTON, BT WINTERBOURNE, BT FISHPONDS, BT LLANTWIT MAJOR. These have been reported to BT and they are currently investigating.
8 Jan 15:56:59
He is an example graph:
9 Jan 15:21:53
BT have been chased further on this as they have not provided an update as promised.
9 Jan 16:19:48
We did not see any congestion over night on the affected circuits but we will continue monitoring all affected lines and post another update on Monday.
12 Jan 10:37:32
We are still seeing congestion on the Exchanges listed above between the hours of 20:00hrs and 22:30hrs. We have updated BT and are awaiting their reply.
20 Jan 12:52:05
We are now seeing congestion starting from 19:30 to 22:30 on these exchanges. We are awaiting an update from BT.
21 Jan 11:13:44
BT have sent this into the TSO team, we are to await their investigation results. We will provide another update as soon as we have a reply.
22 Jan 09:06:14
An update is expected on this tomorrow
23 Jan 09:33:48
This one is still being investigated at the moment, and may need a card or fiber cable fitting, Will chase this for an update later on in the day.
Broadband Users Affected 0.30%
Started 8 Jan 15:40:15 by AA Staff
Closed 5 Feb 13:07:52

5 Feb 10:28:06
[Email and Web Hosting] - RoundCube webmail updated - Info
5 Feb 10:28:06
We have updated the RoundCube webmail today. This is only a minor release so it should all look and work the same as before.
Started 5 Feb 10:00:00