Order posts by limited to posts

20 Sep 11:26:07
Details
20 Sep 11:31:39
Our upstream provider have advised us that they will be carrying out firmware upgrades on core 3G infrastructure on 23rd September 2016 between 00:10 and 04:30 BST.
During this period data SIMs may briefly disconnect as sessions are migrated to other nodes in the upstream provider's network to facilitate the upgrades.
VoIP and SIMs Users Affected 25%
Started 20 Sep 11:26:07 by Carrier
Update was expected 23 Sep 11:30:00
Previously expected 23 Sep 04:30:00 (Last Estimated Resolution Time from Carrier)

1 Sep 17:18:26
Details
1 Sep 17:26:51
Our SMS gateway has always supported HTTPS but it still allows using HTTP only. As using HTTP only is probably a mistake these days we are going to change our gateway to redirect all HTTP requests to HTTPS next week. We don't expect anything to break as curl, etc., should "just work" but we're posting a PEW just in case anyone has a legacy script they think might need adjusting!
Update
9 Sep 10:06:35
Unfortunately we've had to back out of making this change this week as it caused some unforeseen problems! We'll resolve those problems and make the change again soon.
Started 1 Sep 17:18:26
Previously expected 7 Sep 17:18:26

20 Jul 09:58:43
[Maidenhead Colocation] - Web and email outage - Open
Details
04 Apr 2013 15:47:25

This is ongoing. We're investigating.

Update
04 Apr 2013 16:07:41

This should now be fixed. Please let support know if you see any problems, or have any questions.

Update
05 Apr 2013 09:15:04

This was resolved yesterday afternoon.

Update
20 Jul 09:58:43
This may be related to a wider issue on the internet with a power outage at a major london data centre. Thos routing problems are still ongoing.
Started 04 Apr 2013 15:46:11

12 Jun 2015 11:06:49
Details
12 Jun 2015 11:00:23
Our office connectivity blipped. We have internet again, but our phones are down. We're investigating. This is not customer affecting.
Update
12 Jun 2015 11:06:49
Phones are working again.
Previously expected 12 Jun 2015 14:57:07

13 Apr 2015
Details
09 Apr 2015 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr 2015

12 Mar 2015 09:48:01
Details
12 Mar 2015 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.

08 Jan 2015 12:39:06
Details
08 Jan 2015 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
Update
09 Jan 2015 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
Update
12 Jan 2015 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 08 Jan 2015 12:39:06 by AA Staff
Previously expected 15 Jan 2015 17:00:00

03 Jun 2014 17:00:00
Details
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
Update
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
Details
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
Update
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

Update
22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
Update
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
Update
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
Update
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
Update
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
Update
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
Details
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
Details
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
Update
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
Details
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

Details
06 Jul 2015 12:49:42
We have been advised by Three of works on their network (22:30 8th July to 04:50 9th July 2015) Which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some partners may experience dropped connections. Any device reconnecting will automatically be routed via a new path. We apologise for any inconvenience this may cause and the short notice of this advisory.

Details
12 Feb 2015 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

Yesterday 11:40:47
Details
6 Dec 13:33:53

We are working on our systems to add two factor authentication to web site logins.

At present this has been added to the accounts system only. You will see an option to set up 2FA when you log in. Simply follow the instructions.

In due course we plan to also add this to our control pages.

See http://aa.net.uk/kb-broadband-2fa.html for more information.

Started 6 Dec

28 Nov 10:27:07
Details
28 Nov 10:16:14
We have an issue with SMS and are investigating now.
Resolution This has been fixed, sorry for any inconvenience
Closed 28 Nov 10:27:07

25 Nov 10:44:38
Details
24 Nov 13:22:50

Home::1 Quota Increase: We're pleased to announce a 50GB quota increase for customers on the Home:1 tariffs with no change in the monthly costs. For example, those on the Home::1 100GB tariff will have 150GB, those on 200GB will have 250GB etc... This will start from your next bill - which for most people will be December 1st. (The Home::1TB tariff is unchanged and stays at 1TB)

New SoHo::1 Tariffs: We are launching new SoHo::1 tariffs! These are aimed at businesses and will have 150GB, 250GB 350GB and 2TB quotas. These will have a few extra features that the existing Home::1 services don't have such as blocks of IPv4, VAT invoices, optional proactive line monitoring during working hours. This will be available on the order form next week (28th November)

We'll have new web pages explaining the Home::1 and SoHo::1 tariffs in more detail over the coming days.

Update
25 Nov 10:45:23
We have updated all Home::1 lines (not on 1TB) to have the extra 50GB quota from the start of your next month now, and updated the ordering system to allow orders and regrades using the new quota levels.
Started 24 Nov 12:00:00 by AA Staff
Closed 25 Nov 10:44:38

24 Nov 23:00:25
Details
22 Nov 13:57:03
On Thursday night we'll be bringing up new email servers that make up smtp.aa.net.uk - the outgoing email servers that customers use when sending email. These servers will have new SSL certificates. We'll be using certificates from "Let's Encrypt" which should be trusted by email clients unlike the original "CACert" which will mean fewer (or no) warnings about needing to manually trust the certificate.
Update
24 Nov 21:05:38
This work will involves a short period of time where the mail servers may not be available - we will work to make this as short as possible.
Update
24 Nov 21:58:33
This work is about to start...
Update
24 Nov 22:40:16
This work is nearing completion, the new email servers are now online.
Update
25 Nov 10:05:09
If customers are having problems sending email, then please check the following settings:

Set the outgoing smtp server to: smtp.aa.net.uk

Set the encryption options to TLS or STARTTLS

Resolution This planned work has been completed.
Started 24 Nov 22:00:00 by AA Staff
Closed 24 Nov 23:00:25

24 Nov 20:27:00
Details
24 Nov 17:03:53
One of our VoIP carriers has reported that they are experiencing connectivity issues, which are manifesting themselves as call drops, and poor quality calls. We have received additional reports from customers that some inbound calls via the affected carrier are not routing.
Further updates will be posted once we have received them from the carrier.
Update
24 Nov 17:07:13
Call routing for outbound calls has been temporarily overridden to bypass the affected carrier.
This affects only outbound calls, inbound calls via the affected carrier will still be impacted.
Resolution Call quality was returned to normal at 17:48 when the carrier was able to get traffic routing via backup routes. Their primary routes were back online at 20:27.
Started 24 Nov 16:14:03 by Carrier
Closed 24 Nov 20:27:00

23 Nov 14:45:26
[DNS, Email and Web Hosting] - Unable to send email - Closed
Details
23 Nov 14:06:53
We're investigating a problem with customers sending email via smtp.aa.net.uk
Resolution Customers are now able to send email again - we do apologise for the inconvenience this caused. Email would not have been lost due to this.
Started 23 Nov 13:51:16
Closed 23 Nov 14:45:26

19 Nov 12:07:55
Details
19 Nov 12:07:55
For most of this year we've been open on Saturdays, with staff available on phone/irc/email/etc. Due to the very low number of enquiries in the afternoon, from December we're slightly reducing the hours we are open. The new Saturday hours will be from 10AM until 2PM.
Started 19 Nov 12:00:00

18 Nov 09:35:00
[Broadband and Ethernet] - Blip - Closed
Details
18 Nov 09:42:12
A number of lines dropped and reconnected at 9:30 - we're investigating the cause.
Update
18 Nov 12:11:52
The blip affected a couple of our Wholesale customers and a small number of A&A customers. This was caused by a routing change that took place at the time which was done to prefer some directly attached routes rather than our transit routes. This change should have been seamless and not affect anything. We're still trying to understand why it did cause a blip and we apologise to those that were affected.
Started 18 Nov 09:30:00
Closed 18 Nov 09:35:00

14 Nov 14:22:26
Details
14 Nov 14:22:26
Our NAT64 gateway has historically allowed access from outside our customer base for those wishing to experiment with IPv6 only networks. Sadly, due to abuse, we are now locking this down to customer IP addresses only. See http://aa.net.uk/kb-broadband-ipv6-nat64.html for further information on the NAT64 gateway.

28 Oct 19:16:26
[Broadband and Ethernet] - L2TP issues - Closed
Details
28 Oct 18:08:03
Customers using L2TP are seeing some issues. We are working on these now.
Update
28 Oct 18:27:54
We think we see the issue and will be applying a fix shortly which will drop sessions one more time.
Update
28 Oct 19:09:52
Sorry for ongoing issues with L2TP service, we are still working on it.
Update
29 Oct 16:56:01
This also caused the quota image on the main A&A page that shows for Home::1 customers to vanish - that should be back now.
Resolution L2TP service is now on a backup LNS, sorry for any inconvenience. We will resolve the underlying issue in due course.
Started 28 Oct 18:00:00
Closed 28 Oct 19:16:26
Previously expected 28 Oct 18:30:00

8 Nov 13:29:47
Details
9 Oct 18:42:58
We now have some aerial photography services, see http://aa.net.uk/drone.html
Started 9 Oct 18:42:21

31 Oct 17:00:00
Details
31 Oct 08:10:37
There was an issue with one of our call servers today which would have resulted in some calls failing and some SIP2SIM not working at all. Service has been restored, and we will be installing a new call server this morning to replace the faulty unit.
Started 31 Oct 06:00:00
Closed 31 Oct 17:00:00

3 Nov 17:08:43
[Broadband and Ethernet] - L2TP services - Completed
Details
3 Nov 09:08:27

The replacement server for L2TP was installed this morning, which meant sessions all cleared and moved over automatically back to the main server.

We expect to complete an upgrade as part of our ongoing router upgrades this evening - causing an L2TP reconnect at that time.

Resolution Upgrade completed, all looks good.
Started 3 Nov 17:00:00
Closed 3 Nov 17:08:43
Previously expected 3 Nov 19:00:00

3 Nov 09:49:30
Details
3 Nov 08:39:14
We've had a few reports that customers are having some problems with incoming calls on our VoIP service. Staff are looking in to this at the moment
Resolution This post has been superceded by https://aastatus.net/2313
Started 3 Nov 08:00:00
Closed 3 Nov 09:49:30

3 Nov 09:55:30
Details
3 Nov 09:35:32
We have identified a serious issue where some calls are totally failing to route. This will mean a reset of call servers causing calls to drop.
Resolution We sorry for the inconvenience and we are investigating exactly how this happened, but VoIP services are restored now.
Started 3 Nov 09:34:16
Closed 3 Nov 09:55:30

4 Nov 21:30:00
Details
27 Oct 14:57:00

We don't yet have dates for this, but will update this status when we do.

However, we are planning to change over the TT links to new 10G capacity soon. The plan is to move traffic between links to do this as seamlessly as possible. But we know things do not always go to plan. Ideally we would do this on a Sunday or early hours of the day.

At present we are expecting two separate changes, each of which will be for two separate links, and may be on different days.

More to follow as soon as we have details. This will create a massive increase to our capacity in to Talk Talk back-haul.

Update
27 Oct 16:05:20
I am pleased to say that TT Business are actually now being very helpful over this upgrade, and working with us to sort the exact date, which involves some planning on both sides.
Update
28 Oct 14:09:56
The current plan is some time next week, one fibre each day over two days. The idea is to do this in the morning, but exact time is not yet set. It will mean running at some reduced capacity for up to an hour each time - but the idea is to pick a time when this will not be a problem. More exact details to follow.
Update
28 Oct 16:05:39
Sadly the engineers not the TT side will not be able to do this until the following week. We'll try and sort exact dates as soon as possible.
Update
1 Nov 14:56:56
Good news - the first stage is planned 6am to 8am on Thursday 3rd. We hope to make this as seamless as possible, and once done TT links will have much higher capacity. We will plan a second day once this is all done and working for the other link.
Update
1 Nov 16:48:07
We are also doing some preparation work Wednesday morning which should have no impact, but is a risk.
Update
1 Nov 16:48:50
Support staff are planning to open phone lines at 7am on Thursday in case there are any unexpected issues.
Update
2 Nov 06:12:19
The Wednesday (2nd Nov) morning tests worked successfully.
Update
3 Nov 06:03:25
The work scheduled for this morning has started.
Update
3 Nov 07:06:41
TalkTalk traffic has been moved over to the other Datacentre and we're just awaiting the port change to happen.
Update
3 Nov 07:23:34
The new 10G TT link is now live and passing traffic. Staff are continuing to monitor it.
Update
3 Nov 14:30:22
The second fibre will now be changed over on Friday 4th November during the morning from 10AM. Again, this should not impact customer services in any way.
Update
4 Nov 10:25:55
This work has started, like yesterday this should not impact customers
Update
4 Nov 12:19:38
The work has overrun, but is not impacting customers. We'll post more once this is all sorted.
Update
4 Nov 21:17:44
Eventually... we've just been able to bring the second 10G port in to use. Traffic is starting to be shared across the two new links.
Resolution The 2nd 10G port was brought in to service at 21:15. Looking good!
Started 2 Nov 07:00:00
Closed 4 Nov 21:30:00
Previously expected 14 Nov

31 Oct 23:12:09
Details
31 Oct 16:47:08
Sorry for the short notice, but we're carrying out a PEW tonight (the 31st of October) at 22:00 which will affect access to the control pages and to email systems. The window is an hour, but we expect the work to complete much more quickly than that.
Update
31 Oct 22:22:00
This work is starting now.
Resolution This has been completed successfully.
Started 31 Oct 16:41:31
Closed 31 Oct 23:12:09
Previously expected 31 Oct 23:00:00

31 Oct 10:21:19
Details
31 Oct 07:24:52
One of our 'Voiceless' VoIP servers is currently offline. We are running on a single server at the moment. In and outbound calls should still be working OK. However, customers with problems may need to force their phone to re-register as we have removed the faulty server from DNS.
Resolution The fault is resolved now, and we are looking in to the cause. We think simple equipment failure.
Started 31 Oct 07:00:00
Closed 31 Oct 10:21:19

11 Nov 10:20:15
[Broadband and Ethernet] - Router upgrades - Completed
Details
28 Oct 14:11:02
Over the next week or so we expect to upgrade firmware in various routers, this is normally done with little or no disruption. We also expect to do over night rolling upgrades to LNSs.
Update
28 Oct 17:45:44
L2TP customers will be reset this evening for a few seconds.
Update
28 Oct 18:02:19
L2TP customers are seeing some issues this evening, we are working on it.
Update
28 Oct 19:36:04
Router upgrades may be delayed whilst we investigate an issue.
Update
1 Nov 09:53:24
Router upgrades are expected to resume this week, having identified the issue.
Update
2 Nov 17:48:08
Some routers being upgrades tonight (Wednesday 2nd)
Update
3 Nov 07:46:06
We expect more router upgrades tonight (Thursday) and tomorrow, and rolling LNS upgrades to start tomorrow as well.
Update
4 Nov 17:58:45
Tonights upgrades have been completed and rolling LNS upgrades will start tonight
Update
11 Nov 10:19:59
LNS roll overs are now complete.
Resolution Work complete
Started 31 Oct
Closed 11 Nov 10:20:15
Previously expected 11 Nov

28 Oct 19:07:34
[Broadband and Ethernet] - Talk Talk upgrades - Closed
Details
22 Oct 10:13:47

Over the last month or so we have noticed some issues with capacity on some of our Talk Talk back-haul. Staff have been working on efforts to rebalance this manually, but there are some signs of latency on some lines at some peak times. The is obviously not up to our usual standards, and we have ordered major upgrades to 10Gb/s links which are expected in a few weeks. We hope to announce more concrete dates in a planned work announcement soon.

At the end of last year we planned for these upgrades making our network ready for 10Gb/s links. We do forecasts on usage regularly and started trying to get these upgrades around 3 months ago, but it has taken a lot longer than expected.

There is more generally on this on my blog http://www.revk.uk/2016/10/to-ten-gigabits-and-beyond.html>/p>
Resolution See planned work notice on upgrades
Started 1 Oct
Closed 28 Oct 19:07:34
Previously expected 1 Dec

26 Oct 17:15:32
[Broadband and Ethernet] - Checking routers - Info
Details
26 Oct 16:08:14

There are occasionally issues that crop up in the internet and security industry, one of the latest being the use of millions of consumer devices to launch attacks on parts of the Internet.

Naturally we do not want our customers to be unwittingly helping such attacks. We want to help, and advise customers if they have an issue with their equipment. We already did this some time ago with a system to check for open DNS resolvers (for which there is an opt-out on our control pages).

However, we are concerned that anything more complex, such as a port/IP scan or default password check could be considered, in some cases, to be computer misuse, even though our intentions are good.

We have now updated our standard terms to cover this (making what we would be doing "authorised" and hence legal if it was not already), and expect to be adding an opt-out for this on the control pages. Always bear in mind that anyone on the Internet can be doing the same to your network at any time, for malicious purposes, so ensuring firewalls and other security systems are in place is important.

See http://aa.net.uk/legal-terms-services.html under Firewalls for more details

P.S. obviously happy to discuss this on irc with people. RevK.

Started 26 Oct 15:58:24

20 Oct 18:09:29
Details
20 Oct 18:09:29

We are able to offer 07 UK Mobile numbers on our Voice over IP platform. This means a conventional 07 number, charged by operators like any other UK mobile number, routed to you using VoIP/SIP.

For more information http://aa.net.uk/kb-telecoms-07.html

Started 20 Oct 18:08:50

20 Oct 04:37:58
Details
14 Oct 08:51:10
We are needing to update the software on our core switches in both of our London datacentres.

This work will be carried out between 1AM and 4AM on Thursday 20th October

This will affect Broadband and Ethernet customers as there will be routing changes and possible PPP drops and reconnects.

Staff will be available on IRC during this period.

Update
20 Oct 04:12:35
The maintenance window is unfortunately overruning, due to an issue with spanning tree after a switch reload. We're currently working on the issue, and will post further updates when they are available
Update
20 Oct 04:37:01
The maintenance window is now complete. All services should be back to normal at this time.
Started 20 Oct 01:00:00
Closed 20 Oct 04:37:58
Previously expected 20 Oct 04:00:00

12 Oct 15:21:33
Details
12 Oct 15:17:00
We're currently investigating a problem with users logging in to the Control Pages.
Update
12 Oct 15:22:29
The Control Pages are working again.
Started 12 Oct 15:05:00
Closed 12 Oct 15:21:33

4 Oct 19:00:00
Details
4 Oct 08:59:06
We are investigating a problem with some users accessing email. This looks to be affecting IMAP users and email older than today. We will update his post shortly with more details as we investigate this.
Update
4 Oct 09:24:05

It looks like email and folders are missing from customers mailboxes from before this morning. Customers will only be seeing new email that has arrived today.

Customers email and folders have not been lost and we are working on restoring access to email and folders.

The cause of the missing email and folders is due to an regular overnight archiving program that has moved these by mistake.

Update
4 Oct 10:07:36
Mail is being restored at the moment.
Update
4 Oct 11:50:24
This is still ongoing. Mail is still being restored.
Resolution The restoration completed yesterday evening. We do apologise for the inconvenience that this caused.
DNS, Email and Web Hosting Users Affected 10%
Started 4 Oct 08:45:00
Closed 4 Oct 19:00:00

4 Oct 12:07:32
Details
4 Oct 12:07:32

The way the mobile operator handles the data SIMs has changed - work has been ongoing for nearly a year and a lot of testing has been done and a lot of changes behind the scenes, but last week the last of the old LACs were removed leaving only the new ones.

Even after all of the testing, this has led to a handful of edge cases where SIMs did not work. The main issues are around the use of usernames and passwords with SIMs.

In general these do not matter as the SIM is authenticated by its ID, but cases where end devices sent a blank username or password rather than simply not authenticating, have caused issues.

As a workaround, we have today implemented some defaults for blank username and password. This will not affect SIMs using our IP, but if we are relaying a connection to you then you may see some new authentication details. The old LACs did this too, so it should not cause problems for any customers. The default username passed on is the ICCID, and the default password is the word "password".

Obviously customers are welcome to use actual username and password authentication if you wish, but we would usually recommend authentication on the calling station ID (the ICCID).

Started 4 Oct 11:00:00

2 Oct 09:06:05
[Broadband and Ethernet] - LNS roll-over upgrades - Info
Details
9 Sep 15:41:27
We do upgrades on our L2TP Network Servers over night one at a time as a rolling upgrade from time to time. It has been around 6 months since the last upgrades, and new versions are now available. We are moving lines overnight as normal, using the preferred time as set on the control pages as a basis. This will be ongoing over the next couple of weeks, impacting a small proportion of customers each night with a PPP drop and reconnect. Depending on your equipment this could be anything from a fraction of a second outage to a few minutes. Thank you for your patience.
Update
9 Sep 15:46:48
Note that careless, used for L2TP service, will be reloaded this evening.
Update
13 Sep 16:26:24
Wholesale customers of those with L2TP relay may start to notice changes in LCP echo handling. Any issues please do let us know. We are planning some config options for this but the default for now is to provide us with loss/latency graphs on relayed connections which will be visible on the control pages. Other customers are not affected, so if this makes no sense, don't worry.
Update
15 Sep 13:00:52
This is taking longer than expected and customers on the "G" LNS will be moved back to the "F" LNS this evening. We believe we have found an issue that was affecting half a dozen lines briefly this morning and should have that resolved to continue the upgrade over the weekend.
Update
18 Sep 08:36:11
LNS roll out now progressing again as planned - we are aiming at one LNS every two days (over night, as usual).
Resolution The main upgrades were all completed by end of last week. We are doing further minor updates this week over night.
Started 10 Sep
Previously expected 30 Sep

30 Sep 18:01:35
[VoIP and SIMs] - Mobile call rates - Info
Details
30 Sep 17:57:57
Our rates for calls to the main UK mobiles have been reduced from 5p+VAT/min to 2.5p+VAT/min.
Started 1 Oct

30 Sep 14:58:39
[Broadband and Ethernet] - Some TT lines blipped - Closed
Details
30 Sep 14:48:23
At around 14:40 some TalkTalk connected lines blipped. Some are still offline at the moment. We are investigating.
Update
30 Sep 14:52:38
Lines are reconnecting now.
Resolution Lines have reconnected. In the worst cases some customers saw an outage for around 12 minutes. This was a result of some routing changes we have been making this afternoon to better balance our TalkTalk connections. We do apologise to the customers this affected, it was unexpected.
Started 30 Sep 14:40:00
Closed 30 Sep 14:58:39

29 Sep 16:06:57
Details
29 Sep 16:06:57

We're looking for a new member of staff for our front line technical support team, and another to join our sales/order processing team here in Bracknell

Please do send an email to jobs@aa.net.uk for further information if you are interested.

Started 29 Sep 16:00:00

29 Sep 11:48:52
Details
29 Sep 11:18:28
There seems to be a UK wide problem in calling 03 numbers from mobiles at the moment. This is a problem outside of our and our carriers networks. It is affecting most 03 numbers in the UK. Customers wanting to call our offices can use 01344 400 000
Update
29 Sep 11:49:06
This has been resolved, and calls are now working again.
Started 29 Sep 10:00:00
Closed 29 Sep 11:48:52

26 Sep 18:47:27
[Broadband and Ethernet] - Some sort of routing issues - Closed
Details
26 Sep 18:21:20
May be an attack, not sure yet - we are looking in to it - no reports from customers yet and a lot of traffic flowing normally but also some rather odd traffic.
Update
26 Sep 18:31:23
Looks to be impacting Talk Talk backhaul customers more. We think target is a TT connected line.
Update
26 Sep 18:40:35
We think we see the target but over 3Gb/s is hitting us in this attack right now.
Update
26 Sep 18:44:18
Looking better now.
Resolution Seems to be resolved, we are double checking the blackhole routing logic as it seems at least one transit was flooding with the attack after we detected it. Sorry for any inconvenience.
Started 26 Sep 18:00:00
Closed 26 Sep 18:47:27

28 Sep
[VoIP and SIMs] - DTMF handling - Completed
Details
23 Sep 17:23:47
We are planning some upgrades over the next few days that should improve some DTMF handling on our VoIP platform. Please do let us know if you have any issues.
Resolution The upgrade to our voice servers will be later tonight (Tuesday). We have also added an option on some of our number ranges to allow RFC2833 or in-band DTMF, on our control pages. - this is only on a few of our ranges and some ported numbers. Outgoing DTMF, which mostly worked, should now be greatly improved. Please let us have any feedback.
Started 23 Sep 18:00:00
Closed 28 Sep
Previously expected 30 Sep

23 Sep 17:24:51
[VoIP and SIMs] - New O2/EU SIMs - Info
Details
31 Aug 16:21:27

We're sorry to say that we won't don't have any more of the O2/EU SIMs which roam to other UK networks when no O2 coverage. This is sadly not for technical reasons.

From now on we have O2 only SIMs (work in UK on O2), EU only SIMs (work in UK, EU, and world, but higher price), and O2/EU SIMs which switch, but only do O2 when in the UK and so do not roam to other UK networks.

We hope soon to have some new SIMs that have sensible UK pricing and roam to other UK networks, but we do not have a date for this. It is being worked on.

The web site has been updated, but if you normally order Uk roaming O2/EU SIMs, please take not that they will not be the same as you had previously.

EXISTING SIMS ARE NOT AFFECTED

Resolution The new O2/EU SIMs (which are always O2 in the UK) do have a nano-SIM cut out. We'll announce once we get some UK roaming SIMs again, and this is expected soon.
Started 1 Sep
Closed 23 Sep 17:24:51
Previously expected 1 Sep

21 Sep 19:00:00
Details
21 Sep 20:57:09
Some customers using us for their incoming email have had some delayed or duplicated messages this afternoon. This was due to a system problem which was resolved this evening. We do apologise for this and would like to reassure customers that messages haven't been lost.
Started 21 Sep 14:00:00
Closed 21 Sep 19:00:00

21 Sep 17:58:39
Details
6 Sep 20:37:10

We've been doing a lot of expansion of our network this year. Earlier in the year we were working on increasing the capacity of our core London network by installing a 40G ring between our London sites and upgrading our core switch infrastructure.

Our aim is, and always has been, to provide low latency, uncongested Internet access for all our customers.

The bulk of our Internet traffic is carried over 'peering networks' in London such as LoNAP and LINX, where we get to peer directly with the major content providers. The rest of the traffic is carried over various 'transit' links which connect us to the rest of the Internet.

Today we made live a new transit link at our Telehouse site! This is using Level 3 and we have a second link going live at our Telecity/Equinix site in the next couple of weeks. This increases the capacity between us and the rest of the internet, and, more importantly, has reduced the number of hops to many places. In practice this means slightly lower latency to various places on the Internet - Good news for our customers.

Update
21 Sep 17:59:06
The second link is now live!
Started 6 Sep 20:00:00
Closed 21 Sep 17:58:39