Order posts by limited to posts

Today 10:22:42
[Broadband and Ethernet] - Talk Talk upgrades - Open
Today 10:13:47

Over the last month or so we have noticed some issues with capacity on some of our Talk Talk back-haul. Staff have been working on efforts to rebalance this manually, but there are some signs of latency on some lines at some peak times. The is obviously not up to our usual standards, and we have ordered major upgrades to 10Gb/s links which are expected in a few weeks. We hope to announce more concrete dates in a planned work announcement soon.

At the end of last year we planned for these upgrades making our network ready for 10Gb/s links. We do forecasts on usage regularly and started trying to get these upgrades around 3 months ago, but it has taken a lot longer than expected.

There is more generally on this on my blog http://www.revk.uk/2016/10/to-ten-gigabits-and-beyond.html>/p>
Started 1 Oct
Expected close 1 Dec

20 Sep 11:26:07
20 Sep 11:31:39
Our upstream provider have advised us that they will be carrying out firmware upgrades on core 3G infrastructure on 23rd September 2016 between 00:10 and 04:30 BST.
During this period data SIMs may briefly disconnect as sessions are migrated to other nodes in the upstream provider's network to facilitate the upgrades.
VoIP and SIMs Users Affected 25%
Started 20 Sep 11:26:07 by Carrier
Update was expected 23 Sep 11:30:00
Previously expected 23 Sep 04:30:00 (Last Estimated Resolution Time from Carrier)

1 Sep 17:18:26
1 Sep 17:26:51
Our SMS gateway has always supported HTTPS but it still allows using HTTP only. As using HTTP only is probably a mistake these days we are going to change our gateway to redirect all HTTP requests to HTTPS next week. We don't expect anything to break as curl, etc., should "just work" but we're posting a PEW just in case anyone has a legacy script they think might need adjusting!
9 Sep 10:06:35
Unfortunately we've had to back out of making this change this week as it caused some unforeseen problems! We'll resolve those problems and make the change again soon.
Started 1 Sep 17:18:26
Previously expected 7 Sep 17:18:26

20 Jul 09:58:43
[Maidenhead Colocation] - Web and email outage - Open
04 Apr 2013 15:47:25

This is ongoing. We're investigating.

04 Apr 2013 16:07:41

This should now be fixed. Please let support know if you see any problems, or have any questions.

05 Apr 2013 09:15:04

This was resolved yesterday afternoon.

20 Jul 09:58:43
This may be related to a wider issue on the internet with a power outage at a major london data centre. Thos routing problems are still ongoing.
Started 04 Apr 2013 15:46:11

12 Jun 2015 11:06:49
12 Jun 2015 11:00:23
Our office connectivity blipped. We have internet again, but our phones are down. We're investigating. This is not customer affecting.
12 Jun 2015 11:06:49
Phones are working again.
Previously expected 12 Jun 2015 14:57:07

13 Apr 2015
09 Apr 2015 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr 2015

12 Mar 2015 09:48:01
12 Mar 2015 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.

08 Jan 2015 12:39:06
08 Jan 2015 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
09 Jan 2015 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
12 Jan 2015 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 08 Jan 2015 12:39:06 by AA Staff
Previously expected 15 Jan 2015 17:00:00

03 Jun 2014 17:00:00
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

06 Jul 2015 12:49:42
We have been advised by Three of works on their network (22:30 8th July to 04:50 9th July 2015) Which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some partners may experience dropped connections. Any device reconnecting will automatically be routed via a new path. We apologise for any inconvenience this may cause and the short notice of this advisory.

12 Feb 2015 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

Thursday 18:09:29
Thursday 18:09:29

We are able to offer 07 UK Mobile numbers on our Voice over IP platform. This means a conventional 07 number, charged by operators like any other UK mobile number, routed to you using VoIP/SIP.

For more information http://aa.net.uk/kb-telecoms-07.html

Started Thursday 18:08:50

Thursday 04:37:58
14 Oct 08:51:10
We are needing to update the software on our core switches in both of our London datacentres.

This work will be carried out between 1AM and 4AM on Thursday 20th October

This will affect Broadband and Ethernet customers as there will be routing changes and possible PPP drops and reconnects.

Staff will be available on IRC during this period.

Thursday 04:12:35
The maintenance window is unfortunately overruning, due to an issue with spanning tree after a switch reload. We're currently working on the issue, and will post further updates when they are available
Thursday 04:37:01
The maintenance window is now complete. All services should be back to normal at this time.
Started Thursday 01:00:00
Closed Thursday 04:37:58
Previously expected Thursday 04:00:00

12 Oct 15:21:33
12 Oct 15:17:00
We're currently investigating a problem with users logging in to the Control Pages.
12 Oct 15:22:29
The Control Pages are working again.
Started 12 Oct 15:05:00
Closed 12 Oct 15:21:33

9 Oct 18:42:58
9 Oct 18:42:58
We now have some aerial photography services, see http://aa.net.uk/drone.html
Started 9 Oct 18:42:21

4 Oct 19:00:00
4 Oct 08:59:06
We are investigating a problem with some users accessing email. This looks to be affecting IMAP users and email older than today. We will update his post shortly with more details as we investigate this.
4 Oct 09:24:05

It looks like email and folders are missing from customers mailboxes from before this morning. Customers will only be seeing new email that has arrived today.

Customers email and folders have not been lost and we are working on restoring access to email and folders.

The cause of the missing email and folders is due to an regular overnight archiving program that has moved these by mistake.

4 Oct 10:07:36
Mail is being restored at the moment.
4 Oct 11:50:24
This is still ongoing. Mail is still being restored.
Resolution The restoration completed yesterday evening. We do apologise for the inconvenience that this caused.
DNS, Email and Web Hosting Users Affected 10%
Started 4 Oct 08:45:00
Closed 4 Oct 19:00:00

4 Oct 12:07:32
4 Oct 12:07:32

The way the mobile operator handles the data SIMs has changed - work has been ongoing for nearly a year and a lot of testing has been done and a lot of changes behind the scenes, but last week the last of the old LACs were removed leaving only the new ones.

Even after all of the testing, this has led to a handful of edge cases where SIMs did not work. The main issues are around the use of usernames and passwords with SIMs.

In general these do not matter as the SIM is authenticated by its ID, but cases where end devices sent a blank username or password rather than simply not authenticating, have caused issues.

As a workaround, we have today implemented some defaults for blank username and password. This will not affect SIMs using our IP, but if we are relaying a connection to you then you may see some new authentication details. The old LACs did this too, so it should not cause problems for any customers. The default username passed on is the ICCID, and the default password is the word "password".

Obviously customers are welcome to use actual username and password authentication if you wish, but we would usually recommend authentication on the calling station ID (the ICCID).

Started 4 Oct 11:00:00

2 Oct 09:06:05
[Broadband and Ethernet] - LNS roll-over upgrades - Info
9 Sep 15:41:27
We do upgrades on our L2TP Network Servers over night one at a time as a rolling upgrade from time to time. It has been around 6 months since the last upgrades, and new versions are now available. We are moving lines overnight as normal, using the preferred time as set on the control pages as a basis. This will be ongoing over the next couple of weeks, impacting a small proportion of customers each night with a PPP drop and reconnect. Depending on your equipment this could be anything from a fraction of a second outage to a few minutes. Thank you for your patience.
9 Sep 15:46:48
Note that careless, used for L2TP service, will be reloaded this evening.
13 Sep 16:26:24
Wholesale customers of those with L2TP relay may start to notice changes in LCP echo handling. Any issues please do let us know. We are planning some config options for this but the default for now is to provide us with loss/latency graphs on relayed connections which will be visible on the control pages. Other customers are not affected, so if this makes no sense, don't worry.
15 Sep 13:00:52
This is taking longer than expected and customers on the "G" LNS will be moved back to the "F" LNS this evening. We believe we have found an issue that was affecting half a dozen lines briefly this morning and should have that resolved to continue the upgrade over the weekend.
18 Sep 08:36:11
LNS roll out now progressing again as planned - we are aiming at one LNS every two days (over night, as usual).
Resolution The main upgrades were all completed by end of last week. We are doing further minor updates this week over night.
Started 10 Sep
Previously expected 30 Sep

30 Sep 18:01:35
[VoIP and SIMs] - Mobile call rates - Info
30 Sep 17:57:57
Our rates for calls to the main UK mobiles have been reduced from 5p+VAT/min to 2.5p+VAT/min.
Started 1 Oct

30 Sep 14:58:39
[Broadband and Ethernet] - Some TT lines blipped - Closed
30 Sep 14:48:23
At around 14:40 some TalkTalk connected lines blipped. Some are still offline at the moment. We are investigating.
30 Sep 14:52:38
Lines are reconnecting now.
Resolution Lines have reconnected. In the worst cases some customers saw an outage for around 12 minutes. This was a result of some routing changes we have been making this afternoon to better balance our TalkTalk connections. We do apologise to the customers this affected, it was unexpected.
Started 30 Sep 14:40:00
Closed 30 Sep 14:58:39

29 Sep 16:06:57
29 Sep 16:06:57

We're looking for a new member of staff for our front line technical support team, and another to join our sales/order processing team here in Bracknell

Please do send an email to jobs@aa.net.uk for further information if you are interested.

Started 29 Sep 16:00:00

29 Sep 11:48:52
29 Sep 11:18:28
There seems to be a UK wide problem in calling 03 numbers from mobiles at the moment. This is a problem outside of our and our carriers networks. It is affecting most 03 numbers in the UK. Customers wanting to call our offices can use 01344 400 000
29 Sep 11:49:06
This has been resolved, and calls are now working again.
Started 29 Sep 10:00:00
Closed 29 Sep 11:48:52

26 Sep 18:47:27
[Broadband and Ethernet] - Some sort of routing issues - Closed
26 Sep 18:21:20
May be an attack, not sure yet - we are looking in to it - no reports from customers yet and a lot of traffic flowing normally but also some rather odd traffic.
26 Sep 18:31:23
Looks to be impacting Talk Talk backhaul customers more. We think target is a TT connected line.
26 Sep 18:40:35
We think we see the target but over 3Gb/s is hitting us in this attack right now.
26 Sep 18:44:18
Looking better now.
Resolution Seems to be resolved, we are double checking the blackhole routing logic as it seems at least one transit was flooding with the attack after we detected it. Sorry for any inconvenience.
Started 26 Sep 18:00:00
Closed 26 Sep 18:47:27

28 Sep
[VoIP and SIMs] - DTMF handling - Completed
23 Sep 17:23:47
We are planning some upgrades over the next few days that should improve some DTMF handling on our VoIP platform. Please do let us know if you have any issues.
Resolution The upgrade to our voice servers will be later tonight (Tuesday). We have also added an option on some of our number ranges to allow RFC2833 or in-band DTMF, on our control pages. - this is only on a few of our ranges and some ported numbers. Outgoing DTMF, which mostly worked, should now be greatly improved. Please let us have any feedback.
Started 23 Sep 18:00:00
Closed 28 Sep
Previously expected 30 Sep

23 Sep 17:24:51
[VoIP and SIMs] - New O2/EU SIMs - Info
31 Aug 16:21:27

We're sorry to say that we won't don't have any more of the O2/EU SIMs which roam to other UK networks when no O2 coverage. This is sadly not for technical reasons.

From now on we have O2 only SIMs (work in UK on O2), EU only SIMs (work in UK, EU, and world, but higher price), and O2/EU SIMs which switch, but only do O2 when in the UK and so do not roam to other UK networks.

We hope soon to have some new SIMs that have sensible UK pricing and roam to other UK networks, but we do not have a date for this. It is being worked on.

The web site has been updated, but if you normally order Uk roaming O2/EU SIMs, please take not that they will not be the same as you had previously.


Resolution The new O2/EU SIMs (which are always O2 in the UK) do have a nano-SIM cut out. We'll announce once we get some UK roaming SIMs again, and this is expected soon.
Started 1 Sep
Closed 23 Sep 17:24:51
Previously expected 1 Sep

21 Sep 19:00:00
21 Sep 20:57:09
Some customers using us for their incoming email have had some delayed or duplicated messages this afternoon. This was due to a system problem which was resolved this evening. We do apologise for this and would like to reassure customers that messages haven't been lost.
Started 21 Sep 14:00:00
Closed 21 Sep 19:00:00

21 Sep 17:58:39
6 Sep 20:37:10

We've been doing a lot of expansion of our network this year. Earlier in the year we were working on increasing the capacity of our core London network by installing a 40G ring between our London sites and upgrading our core switch infrastructure.

Our aim is, and always has been, to provide low latency, uncongested Internet access for all our customers.

The bulk of our Internet traffic is carried over 'peering networks' in London such as LoNAP and LINX, where we get to peer directly with the major content providers. The rest of the traffic is carried over various 'transit' links which connect us to the rest of the Internet.

Today we made live a new transit link at our Telehouse site! This is using Level 3 and we have a second link going live at our Telecity/Equinix site in the next couple of weeks. This increases the capacity between us and the rest of the internet, and, more importantly, has reduced the number of hops to many places. In practice this means slightly lower latency to various places on the Internet - Good news for our customers.

21 Sep 17:59:06
The second link is now live!
Started 6 Sep 20:00:00
Closed 21 Sep 17:58:39

15 Sep 15:08:26
13 Sep 21:44:01
Apple released iOS 10 this evening, and looking at our traffic stats customers started downloading it at 6pm on the dot! Our network is coping with this increased load but it does look like it'll be the busiest evening we've ever had.

Looking back over the course of the evening BT connected lines look mostly unaffected, however some TalkTalk connected lines suffered with about an extra 30ms of latency and a decrease in download speeds. This was caused by one of our links to TalkTalk running at full capacity.

Started 13 Sep 18:00:00
Closed 15 Sep 15:08:26

12 Sep 18:12:54
12 Sep 18:12:30

The card printing system has had a few tweaks to help.

A couple of recently introduced bugs have been fixed.

The fields for VCARD can now have a default, which is useful for making VCARD entries for all the same company for example.

The fixed text field type can now have a multiple line text value - this makes it a lot easier to include standard text on a card template without having to make multiple entries, one for each line.

We'll work on any new improvements we can think of... Feel free to make suggestions.

Started 12 Sep 18:09:36
Previously expected 12 Sep 22:09:36

7 Sep 17:56:47
7 Sep 16:44:22
We are investigating a problem with some customers unable to send or receive SMSs through our service.
7 Sep 16:45:48
We have multiple SMS gateways and one of them looks to have a problem. We're taking that one out of service for the time being.
7 Sep 17:39:49
The fault has been narrowed down to a problem for SIP2SIM customers who use scripts to process outgoing SMSs from the SIM. Staff are still investigating.
7 Sep 18:42:31
This has been fixed.
Started 7 Sep 16:30:00
Closed 7 Sep 17:56:47

6 Feb 20:00:00
5 Feb 16:32:23
Some call recordings from today have been lost. We've identified the problem and have worked around it until we can fix it. We should be able to fix it with no customer impact. If you are expecting a call recording that you have not received, support can check whether the call was affected by this problem. The call recordings lost are in the minority and most calls were recorded as expected. This problem is unrelated to yesterday's problems with lack of ring tone!
Started 5 Feb 16:25:24
Closed 6 Feb 20:00:00

6 Sep 10:39:16
[Broadband and Ethernet] - New peering for gamers - Info
12 Aug 18:58:16

I am pleased to confirm we have peering with Blizzard, and are working on peering with Riot Games as well.

The Blizzard peering includes IPv6, as they have just fixed their EU realms IPv6.

Peering means we have a more direct route for traffic to and from our customers. We always have various fallbacks, but peering means the best connectivity for lowest latency in your games.

We always take latency seriously on broadband lines, and this just adds an extra edge.

6 Sep 10:39:16
Just a quick update, we are now peering with Riot Games over our LoNAP and LINX ports.
Started 12 Aug 18:55:57

30 Aug 18:15:02
30 Aug 18:15:02

Our plastic card printing service now allows multiple copies of the same card.

The original design was very much around ID cards where each card would have a different set of name, photo, barcode, other fields, and be printed once.

However, many people have used for things like business cards, wifi password cards, and all sorts where multiple prints of the same card are needed. So we have now allowed a quantity against each card.

There are also a few more standard templates now.

Started 30 Aug 18:13:24
Previously expected 30 Aug 22:13:24

12 Aug 18:21:20
[Broadband and Ethernet] - World of Warcraft IPv6 issues - Closed
24 Jul 19:42:01

It seems that once again Blizzard have some IPv6 issues that are impacting World of Warcraft.

This should be simple for them to fix. They have connectivity to their network, which seems to be very good (via LINX) with low pings, but the individual servers their end do not seem to be working on the IPv6 addresses the client is trying.

This could simply be a change of IPv6 SLAAC address following hardware changes, or could be something else, we don't know. They are not interested in investigating.

Normally a failure in one IP protocol is not a huge issue as clients fall back to the other version, e.g. trying IPv6 first then legacy IPv4. However the WoW client seems not to be doing this, simply hanging and eventually giving an error.

Obviously this has been reported to Blizzard, but I am sorry to say that they are taking the backwards step of taking no notice of IPv6 issues, effectively withdrawing support for the current version of Internet Protocol, IPv6

As such, customers may have to change their config on their game client not to try IPv6. This is obviously not something we would normally recommend as it removes the usual redundancy of having both protocols. It also gives the mistaken impression that IPv6 is somehow unreliable, rather than the more accurate impression that Blizzard are unreliable.

Hopefully they will fix it eventually. We have yet to see any official statement as to withdrawal of support for IPv6, a move that is very unusual for any company to take.

25 Jul 06:57:26
Blizzard have now said they have engineers working on this. No ETA.
26 Jul 20:19:57
An update from Blizzard suggests they have made changes to address this, but tests suggest it is not yet resolved, so still chasing.
27 Jul 10:38:54
Blizzard have made a number of changes to fix this issue. It is not quite fixed for everyone according to my tests, so they are working on it still.
7 Aug 08:54:09

Blizzard have said: "As mentioned previously, the game client is designed to take advantage of the IPV6 options simply by ticking the IPV6 option ingame and restarting the game. As IPV6 comes available, you will be automatically join it."

This is odd, as we have had IPv6 since 2002 and Blizzard have had it since 2011, so by that statement we should "join it".

At this point it is still not working in my test, still a routing loop within Blizzard's network.

As Blizzard mistakenly marked my ticket resolved, I have opened a new one. Sorry we don't have any more update on this.

7 Aug 14:33:25
There is a Blizzard EU forum post on this matter. http://eu.battle.net/forums/en/wow/topic/17612333772?page=1#post-8
Resolution Blizzard have finally resolved the issue.
Started 22 Jul
Closed 12 Aug 18:21:20

5 Aug 15:00:16
[Broadband and Ethernet] - L2TP services disrupted - Closed
5 Aug 14:43:08

L2TP services were disrupted and reconnected following a replacement of one of our LNSs that handled L2TP. The new box causes services to immediately switch from backup to live service automatically which caught us by surprise slightly.

There will be another brief disruption shortly then all should be well again.

Sorry for the inconvenience and very short notice.

Resolution The main L2TP server is fully back in service now.
Started 5 Aug 14:35:00
Closed 5 Aug 15:00:16
Previously expected 5 Aug 14:50:00

12 Aug 18:55:27
[Broadband and Ethernet] - Router issue - Closed
12 Aug 18:04:47

One of our edge routers appears to be playing up, we are investigating.

As normal traffic re-routed. We should have this sorted shortly.

12 Aug 22:51:15
Router was taken out of service until we could resolve the issue.
Resolution Router is back in service, thanks to staff going in to data centre this evening.
Started 12 Aug 18:00:00
Closed 12 Aug 18:55:27

9 Jun 22:00:00
[DNS, Email and Web Hosting] - Email problems to gmail addresses - Closed
9 Jun 17:01:30
We're noticing a large number of email to gmail accounts are being bounced back by gmail servers this afternoon. We are investigating the cause of this.
9 Jun 19:53:32
We have put in a temporary fix (sending via IPv4 instead of IPv6) and messages are new being delivered to gmail addresses without being bounced.
Started 9 Jun 16:00:00
Closed 9 Jun 22:00:00

18 Aug 16:00:00
18 Aug 16:30:43

This is affecting customers who we provide email services to for their domains.

We have noticed a small number of emails on the 17th and 18th August were being marked as spam when they are not spam. The problem should be fixed as of 16:30 on 18th August, but if you are missing email please check your spam folder(s) as the email may be in there.

Started 17 Aug 13:00:00
Closed 18 Aug 16:00:00

5 Sep 11:25:32
18 Aug 15:03:02
We have received advice from Three that they will be applying firmware upgrades to resolve issues identified on their new infrastructure. The maintenance will be carried out of the following days; 23rd August 2016 - 23:10 to 24th August 2016 - 04:30 BST 31st August 2016 5th September 2015 We are still awaiting a full schedule from Three regarding the times for the 31st August and 5th September and will post a further announcement once the times have been confirmed. This affects only our Data SIMs.
VoIP and SIMs Users Affected 15%
Started 18 Aug 14:51:51 by Carrier
Closed 5 Sep 11:25:32
Previously expected 5 Sep 04:30:00 (Last Estimated Resolution Time from Carrier)

10 Aug 18:00:00
29 Jul 17:19:16
There may be disruption to some outbound SMS between 07:00 and 09:00 BST on Tuesday the 9th of August 2016. Although the window is wide, the disruption is only expected to last 30 seconds. This is due to upstream carrier upgrades.
Started 9 Aug 07:00:00 by Carrier
Closed 10 Aug 18:00:00
Previously expected 9 Aug 09:00:00

8 Aug 10:37:44
[DNS, Email and Web Hosting] - SSL Certificates Updated - Info
8 Aug 10:37:44
We're updating SSL certificates for our email servers today. The old serial number is 11C4A8. The new serial number is 124247. Users who don't have the CAcert root certificate installed may see errors. This does not affect webmail. Details on http://aa.net.uk/cacert.html

6 Sep 20:50:57
5 Aug 15:31:35

We are making some changes to our Ip transit connectivity over the next few days. This should be pretty seamless as just mean traffic taking new routes. However, there is always a risk of issues.

Today we are doing a bit or preparation which means some minor routing changes for about 5% of our traffic, and which we also expect to have no impact.

1 Sep 11:01:04
Some of the preparation is being done this morning (1st Sep)
Resolution New transit installed
Started 5 Aug 15:30:00
Closed 6 Sep 20:50:57

3 Aug 16:18:26
[VoIP and SIMs] - SMS billing - Info
3 Aug 15:51:00

Customers may have noticed that some of the SMS itemisation has been wrong on the bills in terms of the time of the message.

Basically, some of the time sent logs were recorded in UTC even in the summer, but the time received was in local time, and so were some of the logs such as SIP2SIM usage.

We have corrected this today, so from now you should see all itemisation on local time.

Customers are reminded that if using our SIP2SIM service there is an airtime charge for sending a text (e.g. 2p) and if the text is to an external number such as someone else's mobile then there is an SMS charge (e.g. 6p), and these are separately itemised on the bill. At least after this change they should be next to each other and not an hour apart.

Sorry for any confusion.

Started 3 Aug 15:49:20

3 Aug 09:39:11
[Broadband and Ethernet] - Price reduction - Info
2 Aug 19:50:54

We are removing the extra £5/month charge for Home::1 customers on BT back haul with 80/20 capped FTTC. This means instead of being £40/month the 80/20 FTTC will be £35/month from your next bill.

This is consistent with our Talk Talk backhaul Home::1 terabyte services where the speed of FTTC does not impact the pricing.

Customers on BT backhaul Home::1 with 40/10 caps that have forecast speeds that could exceed 40Mb/s down or 10Mb/s can order a regrade on our broadband order form for £15. This does not restart the minimum term and usually happens next working day. This is no guarantee that your line will sync over 40/10, it simply changes the caps applied to the line sync.

Started 2 Aug 19:45:00

3 Aug 11:31:20
25 Jul 11:36:34
SIP2SIM may be affected for up to an hour between the hours of 20:00 on the 2nd of August and 06:00 on the 3rd of August. This is due to upstream GGSN upgrade work and the resulting testing of failover.
Started 2 Aug 20:00:00 by Carrier
Closed 3 Aug 11:31:20
Previously expected 3 Aug 06:00:00

25 Aug 14:42:10
27 Jul 15:08:24

We use two data centres in London, Equinix and (Telecity). Equinix have planned engineering works on their power feeds on the morning of 2nd August as follows:

Investigations into the power incident that occurred on Wednesday 20th July 2016 have concluded that the outage was caused by the malfunction of the protection module on the primary circuit breaker for the A power riser in Building 8/9 of Equinix's LD8 (Harbour Exchange) data centre. An identical protection module is installed on the B riser power supply. It is now essential that both circuit breakers are replaced. In order to carry out this work safely it is necessary to power down the affected circuits in accordance with the following schedule in the early hours of 2nd August 2016:

A Power riser:
Power down: 00:00
Reenergise: 01:00 or sooner once work is complete

B Power riser:
Power down: 02:00
Reenergise: 03:00 or sooner once work is complete

All our equipment is dual power - so we do not expect there to be any problems during this period.
Started 2 Aug
Closed 25 Aug 14:42:10
Previously expected 2 Aug 03:00:00