Order posts by limited to posts

31 Mar 17:00:00
Details
31 Mar 16:24:26
Over the next few days we are working on some minor changes to the way we handle passwords on the control pages (clueless).

At these first stages you should see no impact, but there is a risk of issues, and we would ask anyone with problems logging in to control pages, changing passwords, or logging in to DSL, SIMs, etc, to let us know.

Each stage is being tested on our test system and then deployed, with the first stage expected to be updated tonight.

The final stage will mean a change on where passwords are visible, and the processes for issuing and changing passwords. We'll post more details closer to the time.

This is all part of ongoing work to improve security. Thank you for your understanding.

Update
2 Apr 15:53:01
The first stage seems to have gone well - our test/monitoring has been working well to help us check any anomalies and ensure consistency.

The next stage should be equally harmless as it means changing over various systems to use the new password hashes. We plan to work on this over Easter.

We will then go on to change the way passwords are issued when ordering and updated when customers wish to change them.

This work is all part of general review and update of security for passwords on our various systems. Thank you for your understanding.

Update
3 Apr 12:55:34
We are progressing with updates - the login to the control pages is now switched over to the new hashes - any issues, please let me know on irc, but all looks good from here.

The RADIUS logins have changed over as well, to use line based passwords (which are same as control pages login passwords at present). Again, please let us know any issues, but so far all looks well.

The next step later today will be a change to how you change the password on the control pages - this will move to the same system we use on the accounts pages - an emailed link that offers a new passwords via https. This is safer than plain text emailed passwords.

Once that is complete, we plan to update he way passwords are issued when ordering new services, which will hopefully be done later today.

There will then be more testing and cleaning up to be done later.

Update
4 Apr 06:53:08
The first side effect that has been noted is that passwords on the control pages are now case-sensitive. Sorry for any confusion this may have caused.
Update
4 Apr 11:19:58
We expect the work for there weekend to stop now - with more later in the week or next weekend. We are at a stage now that we need to provide some clear documentation on the different levels of passwords and what levels of protection are provided for these in our systems.
Update
5 Apr 13:06:22
We are going ahead with more of the work this weekend now, and expect to separate control page login passwords from Line/DSL login passwords today or tomorrow. We'll post more details once the work is complete. We are currently running tests on out tests systems now.
Update
5 Apr 13:50:44
We have now separated login passwords and line passwords.

Any issues, please let us know.

Started 31 Mar 17:00:00
Previously expected 7 Apr

12 Mar 09:48:01
Details
12 Mar 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.

03 Jun 2014 17:00:00
Details
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
Update
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

1 Apr 09:01:39
Details
1 Apr 09:01:39

We have extended our support hours, which are now 8am to 6pm, Mon-Fri, except (English) public holidays.

Previously we worked 9am to 5pm, and sales/accounts still do. However the support staff can usually address simple/urgent queries in those areas if necessary.

Occasionally we do have people ask why we only work office hours, and it is worth trying to explain this. Many ISPs do, indeed, have 24 hour telephone support, for example.

For most of our services, there are faults that come in two flavours. Either there is some big issue (a major outage), in which case we have staff, getting involved in fixing things whatever time it is, or an individual line fault for DSL. It is pretty rare to have individual faults for VoIP, SIMs, etc, but you can, of course, get line faults for DSL.

When it comes to individual DSL faults, there are a load of things people can do at home/office to eliminate equipment and test for themselves, and we offer various on-line tests via our control pages.[1] This can help resolve things. But the issues that don't just go away, and would require support staff to do something, are almost always something that needs a BT engineer to go out.

With very few exceptions, BT engineers are not going to be going our any quicker if we book them next working day at 9am. So having support staff take calls in the middle of the night would not usually be any help. We also have no intention of farming support out to call centres following scripts.

However, we have decided to extend the hours a bit. The reasons being :-

  • BT engineers work 8am to 6pm normally, and so we can help address any issues that come up with an engineer visit, and talk to the engineer or our customer about it at the time. This has already been seen with some 8am visits by engineers who are confused by the notes and need us to explain.
  • Starting at 8am gives customers a chance to resolve issues that can be resolved by talking to support before the usual working day for most people. If it is a line fault, that is not much help, but if it is a matter of swap a router or reboot something, we can offer the necessary advice before you have an office full of people that cannot work.
  • Starting at 8am and finishing a 6pm allows a lot of people that work during the day to contact us from home where they have an issue with their home broadband. We know some customers appreciate that.
  • We have increased the number of support staff, allowing some staggered working hours so that we can offer this.
But please do bear in mind, we do have irc[2], with a simple web front end, which can offer various help and advice by staff and other customers at all sorts of times. It is informal support from staff outside normal hours, but is usually available. We are thinking of perhaps extending this to be more formal evening irc support at some point maybe, with a rota of some sort.

Obviously we're interested in feedback on how the new support hours work for customers.

Started 1 Apr 09:00:00

3 Feb 21:56:57
Details
3 Feb 21:54:51

We have received a few reports from a customers about a popup window claiming to be from us, and encouraging the user to fill in a survey...

This is not from us and we have no connection with it. We wouldn't undertake this kind of activity. We have more information on this wiki page:

http://wiki.aa.org.uk/Mystery_Popups

Started 3 Feb 20:00:00

8 Jan 12:51:58
Details
8 Jan 12:51:58
We are looking for someone to join our Technical Support team in Bracknell, info here: aa.net.uk/job.pdf
Started 8 Jan 12:50:00

04 Dec 2014 10:18:06
Details
21 Jul 2014 15:49:07
We now have a new official URL for our Status Pages: https://aastatus.net The reason for the change is to make the status pages completely independent of any AAISP infrastructure. They were already hosted on a server in Amsterdam out side of our network, and now the DNS is independent too. Anyone using status.aa.net.uk should update to use aastatus.net
Started 21 Jul 2014 15:45:00

10 Dec 2014 15:25:48
Details
03 Dec 2014 08:26:55
Having sorted the BGP issue this week we will be rolling out upgrades over the next few days with LNS upgrades over night, and some router upgrades early in the mornings.
Started 03 Dec 2014
Closed 10 Dec 2014 15:25:48
Previously expected 08 Dec 2014

02 Dec 2014 08:52:25
Details
20 Nov 2014 09:37:01
We have had a couple of incidents over the last few weeks with some external routes vanishing from our network. Whilst this may seem quite minor it simply should not happen. As such we are working on some investigation over the next few days. This may mean re-loading some routers to add additional diagnostics. In general this is a pretty seamless operation as packets are re-routed around the equipment that is being reloaded. However, there is a small risk of issues.
Update
25 Nov 2014 10:25:38
Investigations are going well and have not needed any changes yet. We may be reloading two routers later today (Tuesday) which should have little or no impact, but will help us with diagnostics.
Update
01 Dec 2014 09:22:04
We think we may have found the cause of the routing issue, and plan to upgrade some routers during the week. This should be relatively seamless.
Update
01 Dec 2014 17:11:04
Some router upgrades this evening (Monday). We have seen this cause a blip on TalkTalk lines before, but hopefully that will not happen this time. In any case we expect any disruption to be a few seconds at most, and for most people none at all.
Resolution The changes seem to have worked. We will also be upgrading again later in the week. Thank you all for your patience.
Started 21 Nov 2014
Closed 02 Dec 2014 08:52:25
Previously expected 08 Dec 2014

10 Nov 2014 09:00:00
Details
05 Nov 2014 13:30:50
Some routers will be updated tomorrow morning, this should have little or no impact. We are also doing LNS upgrades over the next 3 nights anyway.
Update
06 Nov 2014 06:09:29
The upgrades went as planned, but we are extending this PEW to mornings over next few days.
Started 06 Nov 2014
Closed 10 Nov 2014 09:00:00
Previously expected 10 Nov 2014 08:00:00

24 Oct 2014 18:13:53
Details
01 Sep 2014 09:40:46
Once again, the Direct Debits have not gone through on the 1st and so have caused an emailed notice for collection and hence they are going out on the 8th. Obviously they are going out on the date notified in the email, but I appreciate that a few extra days credit may be inconvenient for some people expecting the DD on the 1st. We are working on this. The problem is that the system has been desigend very "defensively" so that any doubt at all on the emailed advance notice will result in a new emailed 5 working days notice to be absolutely sure we are meeting the DD rules.
Resolution This is resolved now
Started 01 Sep 2014
Closed 24 Oct 2014 18:13:53

28 Oct 2014 12:14:13
Details
23 Oct 2014 08:16:55
Over the next few days we expect to do some minor updates. Previous work has made it so that these are seamless, but there is always a risk of some impact. Plans are to do work before 8am though some backup routers may be updated at otehr times (and expect to be no risk). We will also be doing some overnight rolling LNS updates As ever, we may update test routers/LNS at any time.
Update
24 Oct 2014 18:15:46
Updates this morning were around 5am with one extra at 8am, but they show as virtually no disruption to any traffic. Several "B" routers have been done today. We are doing an LNS roll over tonight and more routers in the morning. The testing over the last few weeks has been very good, and allowed us to track down some minor issues that simply did not show on the bench test systems.
Resolution Updates were all completed over the weekend.
Started 24 Oct 2014
Closed 28 Oct 2014 12:14:13
Previously expected 30 Oct 2014

02 Oct 2014 11:05:01
Details
02 Oct 2014 11:05:01
We're updating SSL certificates for our customer facing servers this morning (email, webmail). Users who don't have the CAcert root certificate installed may see errors. Details on http://aa.net.uk/cacert.html
Started 02 Oct 2014 11:04:16

25 Sep 2014 12:07:57
Details
25 Sep 2014 11:48:00
We are investigating a network problem affecting our offices.
Update
25 Sep 2014 11:51:31
This is affecting our telephones.
Update
25 Sep 2014 12:08:45
The office is back online. We had lost IPv4, we're looking in to the cause of this.
Started 25 Sep 2014 11:47:00
Closed 25 Sep 2014 12:07:57

29 Aug 2014 09:00:00
Details
29 Aug 2014 15:39:43
We have had a slight issue with on of our routers which has caused a few seconds of routing blips to some destinations, on a couple of occasions. We're working on this now.
Started 28 Aug 2014
Closed 29 Aug 2014 09:00:00

25 Aug 2014 23:49:30
Details
25 Aug 2014 22:15:51
We are seeing what looks to be routing problems within our network with traffic to/from our Maidenhead datacentre. Routes seem to be flapping and disrupting connectivity with increased latency and packet loss. This would be affecting Ethernet services from Maidenhead as well as customers accessing web and email services that we host in Maidenhead. Customers are also reporting DNS problems.
Update
25 Aug 2014 22:19:09
Engineers are investigating...
Update
25 Aug 2014 23:33:53
Staff are still working on this. The cause of the problem has been identified and is being worked on.
Update
25 Aug 2014 23:50:13
The problem has been resolved, traffic is now back to normal, we apologise for this inconvenience.
Started 25 Aug 2014 21:45:00
Closed 25 Aug 2014 23:49:30

22 Aug 2014 12:17:37
Details
22 Aug 2014 11:56:10
We have added a new section clarifying engineer visits and missed appointments. The confirms the "point of no return" for rearranging appointments, and clarifies compensation either way when an appointment is missed.

We have also added two additional reasons for charging an admin fee (£5+VAT). We hope you think these are reasonable. It is a bit of a shame that such things are necessary. We think it is not fair for such costs to be part of our overheads and so affect the price for everyone else who is being reasonable.

1. If you send us a bogus invoice which we validly reject (e.g. trying to invoice us for a delayed install when we do not guarantee install dates). Also for each further exchange of correspondence on such invoices.

2. If you attempt to take us to ADR when you are not entitled to (e.g. if you have not followed our complaints procedures, or you are a company of more than 10 staff, or you are, or have said you are, a communications provider). We will also charge any fees we end up paying as a result of such an attempt if accepted by the ADR provider.

Any questions, please let us know.

Started 22 Aug 2014

15 Aug 2014
Details
12 Aug 2014 08:48:28
The recent router upgrades have now seen some issues (last night). This means we expect to do more upgrades (or downgrades) over the next few days. We'll know more later today. If there are further issues this may end up being done during the day even, but this looks unlikely.
Update
12 Aug 2014 17:42:46
One of the routers showing problems (a.aimless) had a further issue today, and as part of the defensive design of our kit has automatically downgraded to the previous release. We are still investigating the cause of this issue.
Update
14 Aug 2014 17:30:32
We have a much better handle on the problem, and it looks related to "stuff" out on the internet having an unexpected knock-on effect on our routers. We have some plans for further changes that will address this.
Started 12 Aug 2014 08:45:30
Closed 15 Aug 2014
Previously expected 15 Aug 2014

16 Aug 2014
Details
08 Aug 2014 14:33:12
We will be doing some router upgrades over the next week or so. These will usually have little or no disruption, and LNS upgrades will be done over night as usual.
Started 09 Aug 2014
Closed 16 Aug 2014
Previously expected 16 Aug 2014

09 Aug 2014
Details
01 Aug 2014 08:01:24
Many customers have regular payments on 1st of the month and so do not get a separate Direct Debit notice each month.

Unfortunately, this month, the system did not run correctly meaning that Direct Debits were not collected today. As such they are being re-notified with the agreed 5 working days notice for collection on 8th or 9th.

This should return to normal next month.

Sorry for any confusion that may be caused by this.

Started 01 Aug 2014
Closed 09 Aug 2014

19 Jul 2014 13:05:20
Details
19 Jul 2014 12:21:16
There are some ordering issues which we expect to be resolved in a couple of hours, sorry for any inconvenience. Please try later.
Update
19 Jul 2014 12:58:10
If you are trying to do a top-up on Home::1, for now, we suggest changing to auto-topup. This will allow the line to work whilst waiting for top-up to be working again. Once the top up then happens you can change back.
Resolution Work completed
Started 19 Jul 2014 11:20:00
Closed 19 Jul 2014 13:05:20
Previously expected 19 Jul 2014 14:00:00

16 Jul 2014 18:00:14
[General] - DRIPA - Info
Details
16 Jul 2014 10:13:15
Just to be clear on our policy here - when DRIPA comes in to force, and if A&A become subject to a retention notice for all customers, we aim to work on all practical legal means to minimise the amount of data retained under that legislation - making full use of the bad wording in the Schedule in the 2009 regulations where possible. We also aim to clearly publish what is retained under such a notice and what steps we have taken to minimise such data. Such steps may mean separate companies running email or other services, or even hosting some servers outside the UK, if those are practical steps we can take.

Why? Because blanket mass surveillance is illegal under EU law as it is against our basic human right to privacy as decided by a court, that's why!

Feel free to comment on my blog.

Started 15 Jul 2014

25 Jun 2014 06:28:18
Details
25 Jun 2014 05:39:05
The minor change last night is still having some issues with some servers being slow, and this is currently impacting VoIP. This is still being working on.
Update
25 Jun 2014 06:07:24
VoIP services starting to work properly again, still not 100% right.
Update
25 Jun 2014 06:15:04
VoIP looking a lot healthier now. Still working on tracking down the poor performance issues.
Update
25 Jun 2014 08:35:02
For more details on what happened, please see https://www.facebook.com/AAISP/posts/673421826086051
Resolution We'll be monitoring this closely during the day.
Started 25 Jun 2014 05:38:24
Closed 25 Jun 2014 06:28:18

24 Jun 2014 20:37:53
Details
24 Jun 2014 18:22:57
We are making a very minor change to database formats, but this has resulted in some unexepected impact on some servers using lots of CPU to effect the change. The upshot is some DNS lookups are stalling causing some disruption especially to access to our servers.

The tests we did on the backup/test system did not show this.

However the work is almost over and will be back to normal shortly.

Update
24 Jun 2014 19:22:28
The database update really did take a surprisingly long time to complete, but seems to have finished on the clueless now. The slave servers are completing the updates and we took one of the DNS servers that was being sluggish out of replication until this is finished. Sorry for any inconvenience.
Update
24 Jun 2014 20:11:33
One of the commands in the update has replicated and is taking over an hour on some of the servers. This is not normally an issue, but it looks like the older "C" call server may have problems with registered SIP connections as this has now exceeded the normal SIP registration time. The new call server is not reliant on the central database in the same way so should be fine.

We don't expect the command to take much longer to complete now, but if you need a work around, register on the new call server (voiceless) and that should be fine.

Resolution The servers have caught up.
Started 24 Jun 2014 18:00:00
Closed 24 Jun 2014 20:37:53
Previously expected 24 Jun 2014 19:00:00

08 Jun 2014 18:53:59
Details
08 Jun 2014 18:53:59
Customers with UK domains registered through us will see a new option on the control pages to "Check .uk". This checks if you are entitled to the corresponding domain directly under .uk (i.e. a shorter domain). From Tuesday customers who are entitled to register the .uk domains can then click a button to copy their existing domain and register the new domain. These new domains and corresponding services are the same price as other normal UK domains.

In most cases the new domain is available if you have a domain ending .co.uk, but in a few cases customers with other second level domains are entitled if there was no .co.uk version.

Started 08 Jun 2014 18:00:00
Previously expected 10 Jun 2014 08:00:00

30 Jun 2014 13:49:06
Details
28 May 2014 17:48:23
We're doing some routine router upgrades, and LNS upgrades over the next few days. This should have minimal impact on services.
Started 28 May 2014 17:00:00
Closed 30 Jun 2014 13:49:06
Previously expected 02 Jun 2014

04 Jun 2014 11:00:00
Details
28 May 2014 16:11:35

One of our interconnects in Maidenhead has planned work scheduled for Wednesday 4th June. The interconnect will be taken out of service for around 4 hours. During this time will will routing traffic over our backup link.

Details:

What is at risk: Connectivity to services hosted in Maidenhead. (Hosted servers, web, email, VoIP, control page access, Ethernet customers in Maidenhead). General internet connectivity for ADSL, FTTC and Ethernet customers in London will not be affected.

What is the risk? The risk is that during this time access to some services will be affected. That said, we will have backup link in place which will minimise this risk.

Time this is happening: 10pm Wednesday 4th June through to 6am Thursday 5th June, with downtime estimated of 4 hours.

Started 28 May 2014 15:51:03 by AAISP Staff
Closed 04 Jun 2014 11:00:00
Previously expected 04 Jun 2014 22:00:00 (Last Estimated Resolution Time from Datacentre Staff)

16 May 2014 11:21:56
Details
16 May 2014 11:21:56
The status page is once again posting incidents to Facebook and Twitter: https://www.facebook.com/AAISPStatus https://twitter.com/aaisp
Started 16 May 2014 11:18:55

07 May 2014 14:04:43
Details
07 May 2014 13:50:53
Unfortunately our free SMS 'Keyword' service which uses the number 01444666444 has been withdrawn. We would recommend that customers wanting to have an incoming SMS to email or HTTP post to order a number from our ordering system. You can see a list of area codes which support incoming SMS on http://wiki.aa.org.uk/VoIP_-_SMS
Started 07 May 2014 13:32:00
Closed 07 May 2014 14:04:43

26 Oct 2013 19:00:00
Details
11 Oct 2013 17:12:54
We've fixed a problem with line up/down SMS/email/tweet notifications. This may result in customers receiving old notifications, so please disregard any such historical notifications that you receive.
Update
12 Oct 2013 13:24:43
It looks like the notification system is still a bit broken, with some customers receiving many duplicate notices. We've disabled notifications entirely until we've fixed this. Sorry for any inconvenience.
Update
14 Oct 2013 10:52:12
The notification problem should be fixed now, but please do let support know if you receive any erroneous updates.
Started 11 Oct 2013 17:09:34 by AAISP Staff
Closed 26 Oct 2013 19:00:00

09 Apr 2014 23:00:00
Details
09 Apr 2014 20:02:43

Question: Does the Heartbleed bug affect any AAISP servers?

The answer is that no servers are affected that hold customer data or our aa.net.uk SSL certificate secret key. The control and billing pages, email servers and our ticketing system are all running an unaffected version of openssl.

This doesn't mean that we're running out of date software; we still apply backported security patches to those boxes and plan suitable upgrades in the long term.

Unfortunately, however, we had a single test box that was both affected by the bug and held the CAcert signed certificate that we use for our email services. We are therefore going to revoke that certificate and replace the secret key.

The chances of the key having been leaked are tiny, but we think it is worth this measure as a precaution.

Customers who do not have the CAcert root cert installed may see warnings when they connect to our email services. There is more information here: http://aa.net.uk/cacert.html

Please contact support if you have any questions.

Started 09 Apr 2014 19:40:20 by AAISP Staff
Closed 09 Apr 2014 23:00:00

13 Feb 2014 16:00:00
Details
13 Feb 2014 15:14:54
We are investigating a potential problem with our Maidenhead routers. This should be considered an "at risk" period. Fibre links based in Maidenhead will already have blipped briefly at about 14:55, but we will try to ensure that they do not lose connectivity again.
Started 13 Feb 2014 14:55:00
Closed 13 Feb 2014 16:00:00
Previously expected 13 Feb 2014 16:00:00

03 Apr 2014 15:04:47
Details
01 Apr 2014 08:59:54
As you may be aware the office suffered a break-in in mid March, and as a result we've installed new computers. In a change of direction for A&A we have opted for Windows 8, and are pleased to report that going forward we will only be able to support Windows based devices. Linux and Apple support will be dropped from today. We are also replacing all of our routers with Cisco IOS 14.04.01 based devices so we can deploy Carrier Grade NAT.
Update
01 Apr 2014 13:00:00
Happy April 1st! smiley
Started 01 Apr 2014

07 Apr 2014 13:38:49
Details
31 Mar 2014 15:10:58
There are some router upgrades during this week. We'll roll out changes gradually and disruption should be minimal (few seconds maybe but ideally no outage at all). We'll roll out LNS upgrades as well over night.
Started 31 Mar 2014 17:00:00
Closed 07 Apr 2014 13:38:49
Previously expected 07 Apr 2014

14 Mar 2014 13:45:03
Details
14 Mar 2014 08:25:52
Our offices suffered a break-in last night. Because of this, we may be slow to answer phones, etc, this morning. If it is taking a long time to get through by phone, please email instead.
Update
14 Mar 2014 13:45:42
We're back in the office and mostly back to normal! Thank you for your patience this morning.
Closed 14 Mar 2014 13:45:03

22 Feb 2014 13:00:00
Details
22 Feb 2014 08:03:44
A disk server has failed, it is impacting all web sites we host and email. Engineers are working on this now.
Update
22 Feb 2014 10:08:38
There is a major issue with one of the disk servers, and we are planning to switch to a backup, but that is likely to involve an engineer visit to the data centre.
Update
22 Feb 2014 10:16:44
Engineer is on his way to the data centre now.
Update
22 Feb 2014 11:15:40
This is looking more complex than expected - we have switched the secondary controller, but there are issues with one of the disk arrays as well. Engineer still on site.
Update
22 Feb 2014 11:17:48
Disk array is rebuilding now. We should have email working shortly and then web pages once the disk array rebuilds.
Update
22 Feb 2014 11:57:48
Web space up, and mail servers being reconnected to disk array now.
Update
22 Feb 2014 12:11:56
Issues with web pages again, investigating.
Update
22 Feb 2014 12:14:17
The secondary disk server is now showing problems too. We are working on it.
Update
22 Feb 2014 12:32:03
This is proving to be quite a serious issue - we appear to have issues with two separate disk controllers and with some of the RAID disks and with the file system on one of the disks. This is a very odd multiple failure, especially given that all of this is monitored constantly and was not showing any issues yesterday. We do have daily backups, so if all else fails there are ways to get service restored with backups and some loss of recent emails or changes. At this stage we are working to repair the failed file systems before considering that move.
Update
22 Feb 2014 12:35:22
It looks like we have the mail file store repaired and mail should be back on line shortly.
Update
22 Feb 2014 12:38:27
Web pages back.
Update
22 Feb 2014 12:40:20
Incoming email should now be working again.
Update
22 Feb 2014 12:44:44
We are checking all mail and web servers now to confirm all is well again.
Resolution Obviously this sort of multiple failure is somewhat unexpected. We do have plans for new disk servers anyway, and this type of failure will be considered as part of that system design.
Started 22 Feb 2014 00:39:00
Closed 22 Feb 2014 13:00:00
Previously expected 22 Feb 2014 13:00:00

25 Sep 2013 16:20:00
Details
25 Sep 2013 16:09:44
We're currently investigating a problem with our disk storage server that runs our email and web space. We are investigating.
Update
25 Sep 2013 16:18:26
The server is being restarted at the moment.
Update
25 Sep 2013 16:20:41
Ser server is back on line, we are doing post-boot up checks etc to being the services back online.
Update
25 Sep 2013 16:20:58
Web services are now back online.
Started 25 Sep 2013 16:00:00
Closed 25 Sep 2013 16:20:00

13 Jan 2014 11:39:45
Details
13 Jan 2014 11:39:45
We've updated SSL certificates for our customer facing servers this morning (email, webmail). Users who don't have the CAcert root certificate installed may see errors. Details on http://aa.net.uk/cacert.html
Started 13 Jan 2014 11:38:46

13 Jan 2014 09:00:00
Details
09 Jan 2014 14:41:09
We'll be doing some router upgrades over the weekend - this should have little or no impact.
Started 11 Jan 2014
Closed 13 Jan 2014 09:00:00
Previously expected 12 Jan 2014 23:59:59

03 Dec 2013
Details
02 Dec 2013 16:46:51
We are suffering some unexplained blips with some ports flapping occasionally during the day - and this has been most noticable in effects on Ethernet customers and VoIP services. We believe it is not affecting normal broadband servcies. We have been trying to track this down all day and continue to work on this.
Update
03 Dec 2013 09:27:41
The main issues stopped around 7pm, but we are still looking in to this.
Started 02 Dec 2013 10:00:00
Closed 03 Dec 2013
Previously expected 03 Dec 2013

24 Dec 2013 11:59:17
Details
24 Dec 2013 11:59:17
Just in case any one is concerned, we have renewed the ssl certificate for *.aa.net.uk today. This is used on various systems.

17 Dec 2013 14:54:33
Details
17 Dec 2013 11:14:24

FUD (Fear Uncertainty and Doubt) seems to abound at the moment with a report claiming that there is a secret back-door in BT FTTC modems allowing US DoD / NSA / GCHQ access.

The BT FTTC modems do have a management VLAN for TR069 allowing them to be upgraded, and allowing BT to do things like line tests. This is pretty standard practice and no surprise to anyone.

The hype appears to stem from the fact that BT have used an un-routed block of IP addresses for that management LAN, a block that is assigned to US Department of Defence, 30.x.x.x. This is not uncommon for private management LANs in large networks and does not mean there is any link to the US DoD. It is just a number they picked, and being a block that does not clash with anything in their network or any normal private addresses makes this a reasonably sensible choice.

The paper in question presents no evidence that BT modems have secret spy back doors. Yes, BT could change the firmware in the modem, and could add all sorts if they wanted, just as they could add something to the DSLAM or BRAS or any part of their network. If they did, then they would most likely be found out and a real paper on the issue with actual evidence would be published.

We see no evidence of any deliberate tampering with any packets across the BTW back-haul network, either in the modem or elsewhere.

Happy for comments - this was originally posted on our Facebook page.

Update
17 Dec 2013 14:54:21
Some more links:
  • http://www.ispreview.co.uk/index.php/2013/12/confusion-alleged-gchq-nsa-backdoor-bt-fttc-modems.html
  • http://revk.www.me.uk/2013/12/paraniod-ravings.html

  • 10 Dec 2013 11:10:02
    Details
    10 Dec 2013 10:19:02

    Here are the opening times for the Christmas holiday period:

    • Tuesday 24th December - Open with fewer staff
    • Wednesday 25th December - Closed ***
    • Thursday 26th December - Closed ***
    • Friday 27th December - Skeleton Staff, use email/IRC/SMS to contact us *
    • Saturday 28th December - Closed **
    • Sunday 29th December - Closed **
    • Monday 30th December - Skeleton Staff, use email/IRC/SMS to contact us *
    • Tuesday 31st December - Skeleton Staff, use email/IRC/SMS to contact us *
    • Wednesday 1st January - Closed ***
    • Thursday 2nd January - Open as usual (***)

    Special Holiday Rates

    We are pleased to confirm that customers on our standard units based tariffs will again benefit from a Christmas special rate between Christmas and New Year:

    * The three normal working days between Christmas and New Year will be a Christmas special rate which consumes no units regardless of the amount of usage. That is 27th Dec, 30th Dec, and 31st Dec.
    ** The 28th Dec and 29th Dec being normal Weekend rates.
    *** The 25th Dec, 26th Dec, 1st Jan (and for lines in Scotland, 2nd Jan) are also at Holiday rate.
    The special 'Holiday' rate is the same as weekend rate.


    Merry Christmas and a Happy New Year from all at A&A
    Started 10 Dec 2013 10:00:00

    06 Dec 2013 07:26:39
    Details
    04 Dec 2013 08:49:13
    We will be shutting down 'Clueless' on Friday morning for some maintenance. It will be offline for about 15 minutes.
    During this time access to the control pages won't be possible.
    Update
    06 Dec 2013 07:09:59
    This work will be starting shortly.
    Update
    06 Dec 2013 07:18:34
    Clueless is being shut down now, a few minutes earlier than planned. We expect it to be back online in about 15 minutes or so.
    Update
    06 Dec 2013 07:26:26
    Works completed.
    Resolution Works completed. RAM upgraded and a new Disk for the CQM Graphs archive added.
    Started 06 Dec 2013 07:30:00
    Closed 06 Dec 2013 07:26:39

    01 Dec 2013 09:20:39
    Details
    01 Dec 2013 08:10:32
    Our main database/control server known as clueless is suffering some issues this morning. This could start to have some knock on effects on other services and delay some billing this morning. We hope to have this resolved soon.
    Update
    01 Dec 2013 09:20:55
    Clueless is now back to normal.
    Started 01 Dec 2013 05:00:00
    Closed 01 Dec 2013 09:20:39
    Previously expected 01 Dec 2013 09:00:00

    01 Nov 2013 08:43:14
    Details
    05 Jul 2013 15:42:28

    We have added a feature to our Control Pages which will list IPs that we have found to reply to DNS queries. This is to aid customers in securing any open DNS servers that they may have.

    We will rescan IPs on a weekly basis, and we will be contacting customers by email when we find vulnerable DNS servers shortly.

    There is more information about Open DNS resolvers on our wiki:
      http://wiki.aa.org.uk/Category:Open_DNS_Resolvers

    Our control pages can be accessed at:
      http://aa.net.uk/login-clueless.html

    We are happy for customers to run their own recursive or authoritative DNS resolvers, this tool should help where there are DNS servers that are potentially miss-configured and vulnerable to amplification attacks.

    Started 05 Jul 2013 15:39:00

    24 Sep 2013 16:04:14
    Details
    24 Sep 2013 16:04:14
    Customer who have a PSTN line provided by us are now able to run an Openreach 'Copper Integrated Demand Test' on their line. This can help diagnose a PSTN problem before speaking to AAISP Support. More information on the wiki at: http://wiki.aa.org.uk/Line_Diagnostics#Run_PSTN_Test
    Started 24 Sep 2013 16:00:00

    08 Sep 2013 20:20:24
    Details
    06 Sep 2013 11:49:48
    We will be doing some router upgrades over the weekend. This is a prelude to some minor internal network reconfiguration. This plan is to work through these changes very carefully and fall back if any issues. Generally, a router upgrade is likely to have almost no impact on traffic, but been seen to cause a blip for a few seconds some times.
    Update
    08 Sep 2013 20:22:54
    Router upgrades complete, and new transit feed on-line, including more IPv6 transit. We will be rolling out upgrades to LNSs over night during the week. There was minimal disruption to most customers during this work but it was not quite 100% seamless. Well done to the Paul and Cliff and other staff working on this over the weekend.
    Started 06 Sep 2013 17:00:00
    Closed 08 Sep 2013 20:20:24
    Previously expected 09 Sep 2013

    05 Sep 2013 19:04:46
    Details
    05 Sep 2013 18:07:46
    We are seeing some internal routing issues which are being looked in to now.
    Update
    05 Sep 2013 18:46:07
    This is still being worked on.
    Update
    05 Sep 2013 18:58:21
    Things are getting back to normal now.
    Update
    05 Sep 2013 19:21:46
    Customers may need to reboot their DSL routers if they are still having problems with connectivity.
    Started 05 Sep 2013 18:05:00
    Closed 05 Sep 2013 19:04:46

    17 Aug 2013 17:30:46
    Details
    17 Aug 2013 17:30:46
    Customers who have UK domains with us are getting emails from Nominet, confirming the domains they have, and very much worded as if you have just signed up for a domain. This appears to be a side effect of some housekeeping we are doing to bring our domain management pages back up to date. The domains have not been re-registered or anything and have the same registration date, etc. Once again our control pages for UK domains allows contact details to be updated, NS records to be updated and DNS sec DS records to be updated to Nominet.
    Started 17 Aug 2013 17:27:06