Order posts by limited to posts

Friday 15:58:58
Details
Friday 15:39:43
We have had a slight issue with on of our routers which has caused a few seconds of routing blips to some destinations, on a couple of occasions. We're working on this now.
Started 28 Aug

21 Aug 12:50:32
Details
21 Aug 12:50:32
Over the past week or so we have been missing data on some monitoring graphs, this is shown as purple for the first hour in the morning. This is being caused by delays in collecting the data. This is being looked in to.

18 Aug 10:00:00
Details
18 Aug 10:48:39

Our legacy 'C' VoIP platform will be removed from service on March 2nd 2015.

This platform is now old, tired and we have a better VoIP platform: our FireBrick-based 'Voiceless' platform.

We have created a wiki page with details for customers needing to move platforms: http://wiki.aa.org.uk/VoIP_-_Moving_Platform

We will be contacting customers individually by email later in the year, but we'd recommend that customers start moving now. The wiki page above explains how to move, and in most cases it is simply changing the server details in your VoIP device. Please do contact Support for help though.

Started 18 Aug 10:00:00 by AAISP Staff
Update expected 02 Mar 2015 11:00:00
Expected close 02 Mar 2015 10:00:00

29 Jul 11:42:12
Details
17 Jul 10:08:44
Our email services can learn spam/non-spam messages. This feature is currently down for maintenance as we work on the back-end systems. This means that if you move email in to the various 'learn' folders they will stay there and will not be processed at the moment. For the moment, we advise customers not to use this feature. Will will post updates in the next week or so as we may well be changing how this feature works. This should not affect any spam scores etc, but do contact support if needed.
Update
29 Jul 11:42:12
This project is still ongoing. This should not be causing too many problems though, as the spam checking system has many many other ways to determine if a message is spam or not. However, for now, if customers have email that is miss-classified by the spam checking system then please email the headers in to support and we can make some suggestions.
Started 17 Jul 10:00:00

29 Jul 07:18:08
Details
29 Jul 07:19:26
We'll be moving some lines form "C" to "D" tonight after an issue early this morning. Later in the week we expect to do a rolling LNS upgrade over several nights. As usual this will be a PPP restart. You can set preferred time of night on the control pages.
Update
29 Jul 17:16:37
It is likely that the automated system to move lines from "C" to "D" will not work so this may be done in one go during the night or early morning. The knock on effects of the RADIUS issues early this morning have also caused to free usage, and some unexpected "line down" emails/texts/tweets. Sorry for any inconvenience.
Started 29 Jul 07:18:08
Previously expected 4 Aug

3 Jun 17:00:00
Details
3 Jun 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
Update
3 Jun 18:47:21
First batch of updates done.
Started 3 Jun 17:00:00
Previously expected 7 Jun

23 May 12:00:00
Details
23 May 12:07:01

Our legacy 'A' VoIP platform will be removed from service on November 10th 2014.

This platform is our original Asterisk based system; it is now old, tired and we have a better VoIP platform: our FireBrick based 'Voiceless' platform.

We have created a wiki page with details for customers needing to move platforms: http://wiki.aa.org.uk/VoIP_-_Moving_Platform

We will be contacting customers individually by email in the coming weeks.

Our Asterisk platform has historically been used by customers using IAX or who have phones behind NAT. Our current 'Voiceless' platform should work just as well for phones behind NAT. This does mean that we will no longer be providing a IAX service. The wiki pages have details on configuring asterisk to use SIP instead.

The feature set on Voiceless is the same (if not better) than on 'A' (apart from the IAX support).

Please see the wiki page for more information: http://wiki.aa.org.uk/VoIP_-_Moving_Platform

Update
19 Aug 13:42:19
We are starting to email customers about this work this week.
Started 23 May 12:00:00 by AAISP Staff
Expected close 10 Nov 10:00:00

14 Apr
Details
13 Apr 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
Update
21 Apr 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

Update
22 Apr 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
Update
22 Apr 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
Update
22 Apr 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
Update
23 Apr 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
Update
25 Apr 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
Update
26 Apr 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr
Previously expected 1 May

11 Apr 15:50:28
Details
11 Apr 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 15:50:28

7 Apr 13:45:09
Details
7 Apr 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
Update
10 Apr 23:19:59
Completed earlier this evening.
Started 7 Apr 13:45:09
Previously expected 10 Apr 22:45:00

25 Sep 2013
Details
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

26 Aug 10:08:21
Details
26 Aug 10:08:21
We now support Sieve Filters on our mail servers. In short, much like the filters feature on many email programs this enables customers to set up filters on the server side to move email in to folders.
More information on: http://wiki.aa.org.uk/Sieve_Filtering
Started 26 Aug 10:00:00

26 Aug 09:15:00
Details
26 Aug 09:02:02
Yesterday's and today's line graphs are not being shown at the moment. We are working on restoring this.
Update
26 Aug 09:42:18
Today's graphs are back, yesterdays are lost though.
Started 26 Aug 08:00:00
Closed 26 Aug 09:15:00

23 Apr 10:21:03
Details
01 Nov 2013 15:05:00
We have identified an issue that appears to be affecting some customers with FTTC modems. The issue is stupidly complex, and we are still trying to pin down the exact details. The symptoms appear to be that some packets are not passing correctly, some of the time.

Unfortunately one of the types of packet that refuses to pass correctly are FireBrick FB105 tunnel packets. This means customers relying on FB105 tunnels over FTTC are seeing issues.

The work around is to remove the ethernet lead to the modem and then reconnect it. This seems to fix the issue, at least until the next PPP restart. If you have remote access to a FireBrick, e.g. via WAN IP, and need to do this you can change the Ethernet port settings to force it to re-negotiate, and this has the same effect - this only works if directly connected to the FTTC modem as the fix does need the modem Ethernet to restart.

We are asking BT about this, and we are currently assuming this is a firmware issue on the BT FTTC modems.

We have confirmed that modems re-flashed with non-BT firmware do not have the same problem, though we don't usually recommend doing this as it is a BT modem and part of the service.

Update
04 Nov 2013 16:52:49
We have been working on getting more specific information regarding this, we hope to post an update tomorrow.
Update
05 Nov 2013 09:34:14
We have reproduced this problem by sending UDP packets using 'Scapy'. We are doing further testing today, and hope to write up a more detailed report about what we are seeing and what we have tested.
Update
05 Nov 2013 14:27:26
We have some quite good demonstrations of the problem now, and it looks like it will mess up most VPNs based on UDP. We can show how a whole range of UDP ports can be blacklisted by the modem somehow on the next PPP restart. It is crazy. We hope to post a little video of our testing shortly.
Update
05 Nov 2013 15:08:16
Here is an update/overview of the situation. (from http://revk.www.me.uk/2013/11/bt-huawei-fttc-modem-bug-breaking-vpns.html )

We have confirmed that the latest code in the BT FTTC modems appears to have a serious bug that is affecting almost anyone running any sort of VPN over FTTC.

Existing modems seem to be upgrading, presumably due to a roll out of new code in BT. An older modem that has not been on-line a while is fine. A re-flashed modem with non-BT firmware is fine. A working modem on the line for a while suddenly stopped working, presumably upgraded.

The bug appears to be that the modem manages to "blacklist" some UDP packets after a PPP restart.

If we send a number of UDP packets, using various UDP ports, then cause PPP to drop and reconnect, we then find that around 254 combinations of UDP IP/ports are now blacklisted. I.e. they no longer get sent on the line. Other packets are fine.

Sending 500 different packets, around 254 of them will not work again after the PPP restart. It is not actually the first or last 254 packets, some in the middle, but it seems to be 254 combinations. They work as much as you like before the PPP restart, and then never work after it.

We can send a batch of packets, wait 5 minutes, PPP restart, and still find that packets are now blacklisted. We have tried a wide range of ports, high and low, different src and dst ports, and so on - they are all affected.

The only way to "fix" it, is to disconnect the Ethernet port on the modem and reconnect. This does not even have to be long enough to drop PPP. Then it is fine until the next PPP restart. And yes, we have been running a load of scripts to systematically test this and reproduce the fault.

The problem is that a lot of VPNs use UDP and use the same set of ports for all of the packets, so if that combination is blacklisted by the modem the VPN stops after a PPP restart. The only way to fix it is manual intervention.

The modem is meant to be an Ethernet bridge. It should not know anything about PPP restarting or UDP packets and ports. It makes no sense that it would do this. We have tested swapping working and broken modems back and forth. We have tested with a variety of different equipment doing PPPoE and IP behind the modem.

BT are working on this, but it is a serious concern that this is being rolled out.
Update
12 Nov 2013 10:20:18
Work on this in still ongoing... We have tested this on a standard BT retail FTTC 'Infinity' line, and the problem cannot be reproduced. We suspect this is because when the PPP re-establishes a different IP address is allocated each time, and whatever is session tracking does not match the new connection.
Update
12 Nov 2013 11:08:17

Here is an update with some a more specific explanation as to what the problem we are seeing is:

On WBC FTTC, we can send a UDP packet inside the PPP and then drop the PPP a few seconds later. After the PPP re-establishes, UDP packets with the same source and destination IP and ports won't pass; they do not reach the LNS at the ISP.

Further to that, it's not just one src+dst IP and port tuple which is affected. We can send 254 UDP packets using different src+dest ports before we drop the PPP. After it comes back up, all 254 port combinations will fail. It is worth noting here that this cannot be reproduced on an FTTC service which allocates a dynamic IP which changes each time PPP re-established.

If we send more than 254 packets, only 254 will be broken and the others will work. It's not always the first 254 or last 254, the broken ones move around between tests.

So it sounds like the modem (or, less likely, something in the cab or exchange) is creating state table entries for packets it is passing which tie them to a particular PPP session, and then failing to flush the table when the PPP goes down.

This is a little crazy in the first place. It's a modem. It shouldn't even be aware that it's passing PPPoE frames, let along looking inside them to see that they are UDP.

This only happens when using an Openreach Huawei HG612 modem that we suspect has been recently remotely and automatically upgraded by Openreach in the past couple of months. Further - a HG612 modem with the 'unlocked' firmware does not have this problem. A HG612 modem that has probably not been automatically/remotely upgraded does not have this problem.

Side note: One theory is that the brokenness is actually happening in the street cab and not the modem. And that the new firmware in the modem which is triggering it has enabled 'link-state forwarding' on the modem's Ethernet interface.

Update
27 Nov 2013 10:09:42
This post has been a little quiet, but we are still working with BT/Openreach regarding this issue. We hope to have some more information to post in the next day or two.
Update
27 Nov 2013 10:10:13
We have also had reports from someone outside of AAISP reproducing this problem.
Update
27 Nov 2013 14:19:19
We have spent the morning with some nice chaps from Openreach and Huawei. We have demonstrated the problem and they were able to do traffic captures at various points on their side. Huawei HQ can now reproduce the problem and will investigate the problem further.
Update
28 Nov 2013 10:39:36
Adrian has posted about this on his blog: http://revk.www.me.uk/2013/11/bt-huawei-working-with-us.html
Update
13 Jan 14:09:08
We are still chasing this with BT.
Update
3 Apr 15:47:59
We have seen this affect SIP registrations (which use 5060 as the source and target)... Customers can contact us and we'll arrange a modem swap.
Update
23 Apr 10:21:03
BT are in the process of testing an updated firmware for the modems with customers. Any customers affected by this can contact us and we can arrange a new modem to be sent out.
Resolution BT are testing a fix in the lab and will deploy in due course, but this could take months. However, if any customers are adversely affected by this bug, please let us know and we can arrange for BT to send a replacement ECI modem instead of the Huawei modem. Thank you all for your patience.

--Update--
BT do have a new firmware that they are rolling out to the modems. So far it does seem to have fixed the fault and we have not heard of any other issues as of yet. If you do still have the issue, please reboot your modem, if the problem remains, please contact support@aa.net.uk and we will try and get the firmware rolled out to you.
Started 25 Oct 2013
Closed 23 Apr 10:21:03

25 Aug 23:49:30
Details
25 Aug 22:15:51
We are seeing what looks to be routing problems within our network with traffic to/from our Maidenhead datacentre. Routes seem to be flapping and disrupting connectivity with increased latency and packet loss. This would be affecting Ethernet services from Maidenhead as well as customers accessing web and email services that we host in Maidenhead. Customers are also reporting DNS problems.
Update
25 Aug 22:19:09
Engineers are investigating...
Update
25 Aug 23:33:53
Staff are still working on this. The cause of the problem has been identified and is being worked on.
Update
25 Aug 23:50:13
The problem has been resolved, traffic is now back to normal, we apologise for this inconvenience.
Started 25 Aug 21:45:00
Closed 25 Aug 23:49:30

22 Aug 12:17:37
Details
22 Aug 11:56:10
We have added a new section clarifying engineer visits and missed appointments. The confirms the "point of no return" for rearranging appointments, and clarifies compensation either way when an appointment is missed.

We have also added two additional reasons for charging an admin fee (£5+VAT). We hope you think these are reasonable. It is a bit of a shame that such things are necessary. We think it is not fair for such costs to be part of our overheads and so affect the price for everyone else who is being reasonable.

1. If you send us a bogus invoice which we validly reject (e.g. trying to invoice us for a delayed install when we do not guarantee install dates). Also for each further exchange of correspondence on such invoices.

2. If you attempt to take us to ADR when you are not entitled to (e.g. if you have not followed our complaints procedures, or you are a company of more than 10 staff, or you are, or have said you are, a communications provider). We will also charge any fees we end up paying as a result of such an attempt if accepted by the ADR provider.

Any questions, please let us know.

Started 22 Aug

19 Aug 12:59:53
Details
19 Aug 00:36:05
Initial reports suggest one of our fibre links to TalkTalk is down. This is affecting broadband lines using TalkTalk backhaul.
Update
19 Aug 00:43:35
00:05 TT Lines drop, looked like we had a router blip and a TT fibre blip - reasons yet unknown
00:15 Lines start to log back in
However, we are getting reports in intermittent access to some sites on internet - possible MTU related.
Update
19 Aug 01:33:16
MTU is still a problem. A workaround for the moment, is to lower the MTU setting in your router to 1432. Ideally this should not be needed, but try this until the problem is resolved.
Update
19 Aug 01:58:30
Other wholesalers using TT are reporting the same problem. TT helpdesk is aware of planned work that may be causing this. We have requested that that pass this MTU report on to the team involved in the planned work.
Update
19 Aug 07:14:05
TT tell us they think the problem with MTU has been fixed. We're still unsure at this moment, and will work with customers who still have problems.
Update
19 Aug 07:55:02
This is still a problem affecting customers using TT backhaul. TT are aware and are investigating. This is a result of a router upgrade within TT which looks to have been given incorrect settings.
Where possible, customers can change the MTU on their routers to be 1432
Update
19 Aug 08:55:47
We have been in contact with the TT Service Director who will be chasing this up internally at TT.
Update
19 Aug 09:05:48
Customers with bonded lines using TT and BT can turn off their TT modem or router for the time being.
Update
19 Aug 09:20:11
We are looking at re-routing TT connections through our secondary connection to TT...
Update
19 Aug 09:30:55
Traffic is now routing via our secondary connection to TT, this looks like it is not being routed via the faulty TT router and it is looks as if lines are passing traffic as normal
Update
19 Aug 09:55:32
Some customers are working OK, some are not.
The reason being is that we have 2 interconnects to TT. We are still seeing connections from both of them, however, we have a 1600 byte path from one but only 1500 from the other. The 1500 one is the one that TT did an upgrade on last night. So it looks like TT forgot to configure jumbo frames on an interface after the upgrade.
Needless to say, we've passed this information on to people at various levels within TT
Update
19 Aug 09:57:02
We are working on only accepting connections from TT via the working interconnect.
Update
19 Aug 10:39:32
We are forcing TT lines to reconnect, this should mean they then reconnect over the working interconnect and not the one with the faulty TT router.
Update
19 Aug 11:21:53
We are blocking connections from the faulty TT router and only accepting from the working one. This means when customers connect they have a working connection. However, this does mean that logins are being rejected from customers until they are routed via the working interconnect. It may take a few attempts for customers to connect first time.
Update
19 Aug 11:24:09
Some lines are taking a long time to come back. This is because they are still coming in via the broken interconnect - that we're rejecting. Unfortunately, affected lines just have to be left until they attempt to log in via the working interconnect. So, if we appear to be rejecting your login please leave your router to keep trying and it should fix itself.
Update
19 Aug 11:32:11
TT are reverting their upgrade from last night. This looks like it's underway at the moment.
Update
19 Aug 11:35:00
Latest from TT: "The roll back has been completed and the associated equipment has been restarted. Our (TT) engineers are currently performing system checks and a retest before confirming resolution on this incident. Further information will be provided shortly. "
Update
19 Aug 11:43:32
TT have completed their downgrade. It looks like the faulty link is working OK again, we'll be testing this before we unblock the link our side.
Update
19 Aug 13:01:55
We've re-enabled the faulty link, we are now back to normality! We do apologise for this outage. We will be discussing this fault and future upgrades of these TT routers with TT staff.
Started 19 Aug 00:05:00
Closed 19 Aug 12:59:53

21 Aug 10:30:00
Details
18 Aug 13:21:06
We have an issue with printing the DATA SIM cards at present. (No issue with VOICE SIM cards).

For a few days at least we'll be shipping SIMs unprinted (plain white) instead. We also have a number of mis-prints, which we are doing for half price. Staff will call when you order to confirm if you want a mis-print.

We hope to have the printer working again soon. Sorry for any inconvenience.

Resolution Card printing is working well again, we still have some miss-printed cards, so do ask our Sales Dept if you are interested.
Started 18 Aug
Closed 21 Aug 10:30:00
Previously expected 25 Aug

13 Aug 09:15:00
Details
13 Aug 11:26:08
Due to a radius issue we were not receiving line statistics from just after midnight. As a result we needed to force lines to login again. This would have caused lines to lose their PPP connection and then reconnect at around 9AM. We apologise for this, and will be investigating the cause.
Started 13 Aug 09:00:00
Closed 13 Aug 09:15:00

15 Aug
Details
12 Aug 08:48:28
The recent router upgrades have now seen some issues (last night). This means we expect to do more upgrades (or downgrades) over the next few days. We'll know more later today. If there are further issues this may end up being done during the day even, but this looks unlikely.
Update
12 Aug 17:42:46
One of the routers showing problems (a.aimless) had a further issue today, and as part of the defensive design of our kit has automatically downgraded to the previous release. We are still investigating the cause of this issue.
Update
14 Aug 17:30:32
We have a much better handle on the problem, and it looks related to "stuff" out on the internet having an unexpected knock-on effect on our routers. We have some plans for further changes that will address this.
Started 12 Aug 08:45:30
Closed 15 Aug
Previously expected 15 Aug

16 Aug
Details
8 Aug 14:33:12
We will be doing some router upgrades over the next week or so. These will usually have little or no disruption, and LNS upgrades will be done over night as usual.
Started 9 Aug
Closed 16 Aug
Previously expected 16 Aug

8 Aug 15:25:00
Details
8 Aug 15:42:28
At 15:15 we saw customer on the 'D' LNS's lose their connection and reconnect a few moments later. The cause of this is being looked in to.
Resolution Lines quickly came back online, we apologise for the drop though. The cause will be investigated.
Started 8 Aug 15:15:00
Closed 8 Aug 15:25:00

1 Aug 10:00:00
Details
We saw what looks to be congestion on some lines on the Rugby exchange (BT lines). This shows a slight packet loss on Sunday evening. We'll report this to BT.
Update
30 Jul 11:03:08
Card replaced early hours this morning, which should have fixed the congestion problems.
Started 27 Jul 21:00:00
Closed 1 Aug 10:00:00

4 Aug 13:44:05
Details
Due to a database problem viewing things such as graphs, submitting tests and other things relating to a Line is problematic at the moment. This is being worked on and should be restored shortly.
Resolution Database problem resolved. Sorry for the inconvenience.
Started 4 Aug 13:00:00
Closed 4 Aug 13:44:05

2 Aug 12:02:18
Details
2 Aug 12:02:18
We are planning an expansion of our VoIP services which will require the use of more IP addresses than we have told customers to allow through their firewalls, so have added additional IP ranges to the list.

We are not using these new IP ranges yet, but are giving advanced notice of the change to give customers time to update their firewall rules.

The list is at http://wiki.aa.org.uk/VoIP_Firewall

Started 2 Aug 11:50:35

9 Aug
Details
1 Aug 08:01:24
Many customers have regular payments on 1st of the month and so do not get a separate Direct Debit notice each month.

Unfortunately, this month, the system did not run correctly meaning that Direct Debits were not collected today. As such they are being re-notified with the agreed 5 working days notice for collection on 8th or 9th.

This should return to normal next month.

Sorry for any confusion that may be caused by this.

Started 1 Aug
Closed 9 Aug

25 Jul 21:00:00
Details
7 Jul 15:34:10
There is a problem activating some new data SIMs which being is caused by a problem at Three. This has been escalated within Three and we expect an update by the end of the day. Please note that this only affects the activation of new Three data SIMs. Existing data SIMs are not affected, nor is the data on our O2 based voice SIMs.
Update
8 Jul 09:15:12
Three are still working on this problem as a Priority 1 case. We should have updates every 4 hours and will post them here.
Update
8 Jul 16:19:30
An update: "Three have advised that the interface card that they believe was causing the fault has been replaced as of 01:00 this morning. We are continuing to see API failures, however these have lessened since 11:00. We have asked Three to investigate further and feed back."
Update
9 Jul 13:12:12
This seems to be fixed for new SIMs being activated now. Customers with SIMs which attempted to activate SIMs over the past few days may be stuck in a broken partially-activated state. We are getting these reset by the carrier so we can activate them again.
Update
14 Jul 16:11:18
Activating seems broken again. We've reported it and are awaiting an update.
Update
15 Jul 17:18:11
We do have a small number of SIMs that are in a 'stuck' state, and we're waiting for the carrier to clear them.
Update
17 Jul 15:48:25
Three are still having further problems with activating SIMS. This has been happening for the last couple of days, and is on going. This has been raised with Three.
Update
21 Jul 15:33:17
This is still an ongoing problem, the latest from our supplier is:

"The current understanding is that some of Three's load balanced platforms are returning an inconsistent response when the requests are submitted. This is in turn causing our platform to lock the SIMs from further activation attempts so that they can be investigated."

Update
24 Jul 11:27:28
This is still ongoing, but is more of an intermittent problem. We have been able to activate most SIMs.
Resolution Problems with upstream carrier now resolved. Activating SIMs is now working ok, we apologise for the inconvenience this caused.
Started 6 Jul 12:00:00
Closed 25 Jul 21:00:00

28 Jul 11:00:00
Details
28 Jul 09:20:03
Customers may have seen a drop and reconnect of their broadband lines this morning. Due to a problem with our RADIUS accounting on Sunday we have needed to restart our customer database server, Clueless. This has been done, and Clueless is back online. Due to the initial problem with RADIUS accounting most DSL lines have had to be restarted.
Update
28 Jul 10:02:13
We are also sending out order update messages in error - eg, emails about orders that have already completed. We apologise for this confusing and are investigating this.
Started 28 Jul 09:00:00
Closed 28 Jul 11:00:00

29 Jul 13:30:58
Details
29 Jul 13:30:58
The special offer price on the O2/EU SIMs, i.e. free apart from postage, ends on Thursday. After this they go back to the £5+VAT per SIM.
Started 29 Jul
Previously expected 1 Aug

29 Jul 12:22:25
Details
8 Jul 13:19:27
We have the SIMs and should be able to ship today with any luck. We still have some checks to do. However, I have enabled ordering. See http://aa.net.uk/telecoms-sip2sim.html

We have an umber of people wanting to upgrade the UK only O2 SIMs to roaming. For this month these are being supplied at zero buy cost with postage added at cost. After this month we will be back to £5+VAT.

Update
8 Jul 16:33:21
We have not been able to finish testing yet - just waiting on a minor change in the mobile network. We have the SIMs and hope to be able to ship tomorrow.
Update
8 Jul 17:48:31
The good news is that we now have the keys for the SIMs for provisioning now, and are all ready to go on that front. I see several orders waiting to ship.

However we have just been advised that there is a snag in the mobile operator tariffing systems which may take a couple of days to sort. Until that is sorted we cannot really send SIMs out as the call rates will be all screwed up.

So please bear with us.

Update
9 Jul 12:55:43
We are still waiting on an update from the mobile carrier. We have checked, and the tariffing is way out at present which means we still can't ship the SIMs. It could be a couple more days by the look of it.
Update
11 Jul 08:18:04
I am sorry to say that we are being told it may be another week before the tariff is sorted. Please bear with us.
Update
18 Jul 07:04:21
Still waiting on the mobile operator, sorry for delay.
Update
18 Jul 13:14:18
We should be posting SIMs Monday - or possibly today - we are just going through testing with the mobile carrier now to confirm everything is set up right. Thank you all for your patience.
Update
18 Jul 17:07:44
Still testing, I'm afraid - going well but not quite there yet, so we are looking at Monday for the SIMs to be shipped.
Update
21 Jul 15:30:37
Still not sorted - not sure if they will ship today. I appreciatet hat this is very frustrating.
Update
21 Jul 17:47:58
The good news is that testing has finally got close enough to ship the SIMs, in that the main part of the tariffing code is right. We have a couple of small snags to sort and texts from the mobile while roaming are not working yet but hopefully that can be sorted tomorrow. It is not a reason to avoid shipping them I feel.
Update
21 Jul 18:59:16
OK SIMs will go out tomorrow. At present texts from the mobile while on EU profile not working, but we expect that to be sorted very soon.
Update
22 Jul 13:51:24
SIMs are shipping today - at present outgoing texts are not working on the EU profile but we expect that to be fixed very soon.
Update
22 Jul 18:49:07
All of the back orders were shipped today. We expect to put prices back to £5 to buy the SIM from next month.
Update
29 Jul 12:22:25
The outgoing SMS issue when roaming is fixed.

Also, many SIMs went out without an operator name set in the SIM, but SIMs are now going out with the selected operator name. Customers can ask support to send a SIM update to change operator name.

Started 8 Jul 13:00:00

29 Jul 01:17:44
Details
28 Jul 21:38:18
We are having reports this evening of some lines being unable to log in, but are in sync. We are investigating.
Update
28 Jul 22:00:52
We believe we have identified the problem and are working on a fix.
Update
28 Jul 22:17:51
Lines are logging in successfully now. If you are still off, please keep trying.
Resolution An issue with authentication on the "C" LNS, and then on the "D" LNS. We have found the issue, and lines are connecting to "D" cleanly now. The underlying issue causing this is being investigated.
Started 28 Jul 21:37:18
Closed 29 Jul 01:17:44
Cause BT

21 Jul 16:21:17
Details
21 Jul 15:49:07
We now have a new official URL for our Status Pages: https://aastatus.net The reason for the change is to make the status pages completely independent of any AAISP infrastructure. They were already hosted on a server in Amsterdam our side of our network but now the DNS is independent too. Anyone using status.aa.net.uk should update to use aastatus.net
Started 21 Jul 15:45:00

19 Jul 13:05:20
Details
19 Jul 12:21:16
There are some ordering issues which we expect to be resolved in a couple of hours, sorry for any inconvenience. Please try later.
Update
19 Jul 12:58:10
If you are trying to do a top-up on Home::1, for now, we suggest changing to auto-topup. This will allow the line to work whilst waiting for top-up to be working again. Once the top up then happens you can change back.
Resolution Work completed
Started 19 Jul 11:20:00
Closed 19 Jul 13:05:20
Previously expected 19 Jul 14:00:00

17 Jul 17:45:00
Details
17 Jul 16:23:15
We have a few reports from customers, and a vague Incident report from BT that looks like there may be PPP problem within the BT network which is affecting customers logging in to us. Customers may see their ADSL router in sync, but not able to log in (no PPP).
Update
17 Jul 16:40:31
This looks to be affecting BT ADSL and FTTC circuits. A line which tries to log in may well fail.
Update
17 Jul 16:42:34
Some lines are logging in successfully now.
Update
17 Jul 16:54:15
Not all lines are back yet, but lines are still logging back in, so if you are still offline it may take a little more time.
Resolution This was a BT incident, reference IMT26151/14. This was closed by BT at 17:45 without giving us further details about what the problem was or what they did to restore service.
Started 17 Jul 16:00:00
Closed 17 Jul 17:45:00

16 Jul 18:00:14
Details
16 Jul 10:13:15
Just to be clear on our policy here - when DRIPA comes in to force, and if A&A become subject to a retention notice for all customers, we aim to work on all practical legal means to minimise the amount of data retained under that legislation - making full use of the bad wording in the Schedule in the 2009 regulations where possible. We also aim to clearly publish what is retained under such a notice and what steps we have taken to minimise such data. Such steps may mean separate companies running email or other services, or even hosting some servers outside the UK, if those are practical steps we can take.

Why? Because blanket mass surveillance is illegal under EU law as it is against our basic human right to privacy as decided by a court, that's why!

Feel free to comment on my blog.

Started 15 Jul

28 Jul 12:10:28
Details
15 Jul 10:41:58
We are reworking the SMS/twitter/email line up/down notifications and hope to have the new system launched later this week. There may be slightly different wording of the messages.
Update
15 Jul 18:06:51
We're looking to do this in stages. i.e. switch over emails then texts then tweets or something like that. So please bear with us. Ideally the changes should not lose any messages.
Update
17 Jul 09:21:13
We have switched over to the new system - the most noticable change is that SMS and Tweets are now independant. You can have either or both if you require - settings are on the control pages. SMS still has a back off if you have lots of line flaps, but tweets and emails do not delay. Do let us have any feedback on the new system.
Started 16 Jul
Closed 28 Jul 12:10:28
Previously expected 20 Jul

15 Jul 12:52:51
Details
15 Jul 12:52:51
The usage reports sent on 15th of the month for customers that have requested it have apparently not all worked. Some were blank.

These are being resent now, so apologies if you get two of them.

Started 15 Jul

11 Jul 11:03:55
Details
11 Jul 17:00:48
The "B" LNS restarted today, unexpectedly. All lines reconnected within minutes (however fast the model retries). We'll clear some traffic off the "D" server back to the "B" server later this evening.
Resolution We're investigating the cause of this.
Broadband Users Affected 33%
Started 11 Jul 11:03:52
Closed 11 Jul 11:03:55

10 Jul 20:10:00
Details
10 Jul 19:18:35
We are seeing a problem with BT 21CN ADSL and FTTC circuits being unable to log in since approximately 18:00 today. Existing sessions are working fine but are failing to reconnect when they drop. 20CN ADSL and TalkTalk backhaul circuits are working fine.

BT have raised incident IMT25152/14 which looks to be related, but just says they are investigating a problem.

Update
10 Jul 22:16:28
BT have reported that service should have been restored as of 20:10 this evening.

Customers who are still having problems should attempt to re-connect as they may be stuck on a BT holding session.

Anyone still having problems after doing that should contact tech support.

Started 10 Jul 17:15:00
Closed 10 Jul 20:10:00
Cause BT

9 Jul 15:50:00
Details
9 Jul 15:27:03
We are experiencing problem with VoIP platform, this is affecting calls for customers. We are investigating.
Update
9 Jul 15:38:20
Outgoing and 'internal' calls are OK - the problem is with inbound calls not working.
Update
9 Jul 15:42:38
Actually - outbound and internal calls over IPv6 were working, but over IP4 failing.
Update
9 Jul 15:43:10
The problem has been found. Calls should start working again shortly.
Resolution VoIP has been a bit off for a couple of days with two unrelated incidents. Not good. It looks like in both cases many incoming calls were failing. Yesterday was an issue with the code that steers calls - we made a change done in an emergency for a customer (who was trying to abuse SIP and not have to pay for lots of licence fees on his VoIP switch). The change worked for him, but broke many other customers (not all, and not our office). The testing done before this was deployed did not pick up the issue as the test calls were the type that worked (like calls to our office). It was spotted and fixed very quickly. We're trying to work out how we can create more comprehensive tests in future - we want to be agile and responsive to customer needs, but we need any changes to be robust and not cause problems. Today was very different. We noticed an issue on one called server impacting some SIP2SIM customers, so took it out of service to investigate. We have multiple call servers, and switching the active servers is a routine task that can be done in cases just like this to ensure service continues. Switching a call server out of service is done using a function on our control pages which has been tested many times in the past. Unfortuntely it pushes out the zone files for one of the domains as part of the process as it adjusts SRV records for the call servers. This is, again, normally quite safe. What has caught us out is that somehow the zone database was broken, missing some key records, so when it was pushed out the call servers vanished on IPv4. This caused yet more confusion as all of our test calls worked as they come from IPv6 equipment. This took several minutes to pin down and fix. We're now checking archives to try and find when and how the DNS records vanished. We're still investigating why the one call server is playing up, and hope to put it back in service later this evening.
Started 9 Jul 15:20:00
Closed 9 Jul 15:50:00

4 Jul 11:00:06
Details
4 Jul 11:00:06
Just to update - we have the physical SIM cards now, and we have pricing agreed. They are not yet provisioned on the network and that will hopefully be start of next week at which point we'll be able to start selling them. Thank you all for your patience.
Started 4 Jul
Previously expected 8 Jul

2 Jul 09:03:10
Details
1 Jul 16:31:19
SIP2SIM currently appears not to be working. We are investigating and will post an update as soon as possible.
Update
1 Jul 18:26:56
Working again now. We'll post an explanation as soon as possible.
Update
1 Jul 18:44:53
It was a fault in an upstream carrier's network. We don't have any further information at this point.
Started 1 Jul 16:30:27
Closed 2 Jul 09:03:10

26 Jun 10:48:19
Details
26 Jun 10:08:26
An after-the-fact status post, since the status pages were unreachable during the outage.

The authoritative DNS servers which, amongst other things, serve aa.net.uk and reverse DNS zones were unable to serve records due to a database problem from about 09:10 to 10:00 this morning.

This affected reverse DNS records and services under aa.net.uk, including email, voip and the clueless control panel. Customer hosted zones were not affected by this outage.

A fuller post-mortem will be posted here in due time.

Update
26 Jun 10:20:20
The current status as of 10:15 is that we have moved the delegation from Nominet for aa.net.uk from the affected DNS servers to alternative servers as of 09:40. Nominet's servers are serving these new records but resolvers will have cached the old records and will take some time to update.

Doing this has reduced the load on the affected servers, and has allowed them to begin working again, so all DNS services should now be working again.

There may be ongoing issues with customer facing email access which uses hostnames of the format yourdomain.com.mail.aa.net.uk to reach our mail servers. This is being worked on.

Update
26 Jun 10:54:52
We believe we have fixed the problem with serving records for yourdomain.com.mail.aa.net.uk style zones, and have fixed the delegation for mail.aa.net.uk to go back to the previous servers which understand how to serve these zones.

This again will take a short while for DNS updates to propagate, but should mean that all services are back to normal now.

Started 26 Jun 09:09:00
Closed 26 Jun 10:48:19

1 Jul 23:25:00
Details
1 Jul 20:50:32
We have identified some TalkTalk back haul lines with congestion starting around 16:20 and now 100ms with 2% loss. This affects around 3% of our TT lines.

We have techies in TalkTalk on the case and hope to have it resolved soon.

Update
1 Jul 20:56:19
"On call engineers are being scrambled now - we have an issue in the wider Oxford area and you should see an incident coming through shortly."
Resolution Engineers fixed the issue last night.
Started 1 Jul 16:20:00
Closed 1 Jul 23:25:00
Previously expected 2 Jul

1 Jul 17:44:07
Details
1 Jul 17:44:07
We have been looking again in to CDRs. We have some new logic that should show correct I/C logs for calls to your number. The exception is where that is done as part of an also ring from another number that was called. It should also show calls to another A&A number (as free) with the exception of where this is an also ring from another number. I.e. the intermediate legs are not longer showing (they were not showing properly before). We believe the charging is correct.

As a separate issue, customers billed for use of BT lines we provide (which is only a few customers that have outgoing calls) have seen a lack of call bills for a few weeks. This should catch up (hopefully today) with any previously unbilled calls being billed.

Started 1 Jul 17:30:00

30 Jun 17:14:37
Details
30 Jun 17:14:37
The CDR timestamps for SIP2SIM were in UTC. As we do normal call CDRs in local time, this was somewhat confusing.

We have now changed CDRs for SIP2SIM in local UK time. This will affect new call records for calls/text/data on SIP2SIM from now.

Started 30 Jun 17:10:00

28 Jun 09:25:33
Details
28 Jun 09:25:33
We hope to have our first roaming SIMs late next week. These are the same price to get as the existing UK O2 only SIMs (£5 to buy, £2/month).

They have a dual personality in that they can be O2 to work in the UK on O2 at the same prices as the O2 only SIMs (2p/min, 2p/text, 2p/MB), or can switch to an EU profile which has the same price in UK and throughout the EU at 10p/min, 5p/text, 10p/MB.

The EU profile allows use on other UK networks so allowing you much more coverage than a normal UK only SIM.

These SIMs do allow roaming outside EU at a higher price, which we'll confirm in due course.

For a limited period, we are planning to offer these free (pay for postage only) to customers that need roaming and have got one of our O2 only SIMs. Once we have the SIMs in stock, we'll work out the details.

Started 28 Jun

27 Jun 10:07:44
Details
27 Jun 10:08:52
It looks like we have not been recording logs of incoming calls that have been answered in the CDRs on our VoIP service. These have no call cost implications obviously. This has now been fixed and the logs should now show such calls. Sorry for any concern this may have caused.
Started 23 Jun 15:00:00
Closed 27 Jun 10:07:44

25 Jun 06:28:18
Details
25 Jun 05:39:05
The minor change last night is still having some issues with some servers being slow, and this is currently impacting VoIP. This is still being working on.
Update
25 Jun 06:07:24
VoIP services starting to work properly again, still not 100% right.
Update
25 Jun 06:15:04
VoIP looking a lot healthier now. Still working on tracking down the poor performance issues.
Update
25 Jun 08:35:02
For more details on what happened, please see https://www.facebook.com/AAISP/posts/673421826086051
Resolution We'll be monitoring this closely during the day.
Started 25 Jun 05:38:24
Closed 25 Jun 06:28:18

24 Jun 20:37:53
Details
24 Jun 18:22:57
We are making a very minor change to database formats, but this has resulted in some unexepected impact on some servers using lots of CPU to effect the change. The upshot is some DNS lookups are stalling causing some disruption especially to access to our servers.

The tests we did on the backup/test system did not show this.

However the work is almost over and will be back to normal shortly.

Update
24 Jun 19:22:28
The database update really did take a surprisingly long time to complete, but seems to have finished on the clueless now. The slave servers are completing the updates and we took one of the DNS servers that was being sluggish out of replication until this is finished. Sorry for any inconvenience.
Update
24 Jun 20:11:33
One of the commands in the update has replicated and is taking over an hour on some of the servers. This is not normally an issue, but it looks like the older "C" call server may have problems with registered SIP connections as this has now exceeded the normal SIP registration time. The new call server is not reliant on the central database in the same way so should be fine.

We don't expect the command to take much longer to complete now, but if you need a work around, register on the new call server (voiceless) and that should be fine.

Resolution The servers have caught up.
Started 24 Jun 18:00:00
Closed 24 Jun 20:37:53
Previously expected 24 Jun 19:00:00