Order posts by limited to posts

Yesterday 17:24:00
[VoIP and SIMs] - DTMF handling - Open
Yesterday 17:23:47
We are planning some upgrades over the weekend that should improve some DTMF handling on our VoIP platform. Please do let us know if you have any issues.
Started Yesterday 18:00:00
Expected close Monday

20 Sep 11:26:07
20 Sep 11:31:39
Our upstream provider have advised us that they will be carrying out firmware upgrades on core 3G infrastructure on 23rd September 2016 between 00:10 and 04:30 BST.
During this period data SIMs may briefly disconnect as sessions are migrated to other nodes in the upstream provider's network to facilitate the upgrades.
VoIP and SIMs Users Affected 25%
Started 20 Sep 11:26:07 by Carrier
Update was expected Yesterday 11:30:00
Previously expected Yesterday 04:30:00 (Last Estimated Resolution Time from Carrier)

1 Sep 17:18:26
1 Sep 17:26:51
Our SMS gateway has always supported HTTPS but it still allows using HTTP only. As using HTTP only is probably a mistake these days we are going to change our gateway to redirect all HTTP requests to HTTPS next week. We don't expect anything to break as curl, etc., should "just work" but we're posting a PEW just in case anyone has a legacy script they think might need adjusting!
9 Sep 10:06:35
Unfortunately we've had to back out of making this change this week as it caused some unforeseen problems! We'll resolve those problems and make the change again soon.
Started 1 Sep 17:18:26
Previously expected 7 Sep 17:18:26

20 Jul 09:58:43
[Maidenhead Colocation] - Web and email outage - Open
04 Apr 2013 15:47:25

This is ongoing. We're investigating.

04 Apr 2013 16:07:41

This should now be fixed. Please let support know if you see any problems, or have any questions.

05 Apr 2013 09:15:04

This was resolved yesterday afternoon.

20 Jul 09:58:43
This may be related to a wider issue on the internet with a power outage at a major london data centre. Thos routing problems are still ongoing.
Started 04 Apr 2013 15:46:11

12 Jun 2015 11:06:49
12 Jun 2015 11:00:23
Our office connectivity blipped. We have internet again, but our phones are down. We're investigating. This is not customer affecting.
12 Jun 2015 11:06:49
Phones are working again.
Previously expected 12 Jun 2015 14:57:07

13 Apr 2015
09 Apr 2015 13:43:39

We have been advised by Three of works on their network which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some customers may experience dropped connections. Any device reconnecting will automatically be routed via a new path.

This only affects our data only SIMs.

Started 13 Apr 2015

12 Mar 2015 09:48:01
12 Mar 2015 09:48:01
Our wiki at http://wiki.aa.org.uk/ will be down for a while today due to an internal PEW. Sorry for any inconvenience.

08 Jan 2015 12:39:06
08 Jan 2015 12:49:24
We're going to remove the legacy fb6000.cgi page that was originally used to display CQM graphs on the control pages. This does not affect people who use the control pages as normal, but we've noticed that fb6000.cgi URLs are still being accessed occasionally. This is probably because the old page is being used to embed graphs into people's intranet sites, for example, but accessing graph URLs via fb6000.cgi has been deprecated for a long time. The supported method for obtaining graphs via automated means is via the "info" command on our API: http://aa.net.uk/support-chaos.html This is likely to affect only a handful of customers but, if you believe you're affected and require help with accessing the API, please contact support. We will remove the old page after a week (on 2015-01-15).
09 Jan 2015 08:52:28
We'll be coming up with some working examples of using our CHAOS API to get graphs, we'll post an update here today or monday.
12 Jan 2015 16:19:58
We have an example here: https://wiki.aa.net.uk/CHAOS
Started 08 Jan 2015 12:39:06 by AA Staff
Previously expected 15 Jan 2015 17:00:00

03 Jun 2014 17:00:00
03 Jun 2014 18:20:39
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption.
03 Jun 2014 18:47:21
First batch of updates done.
Started 03 Jun 2014 17:00:00
Previously expected 07 Jun 2014

14 Apr 2014
13 Apr 2014 17:29:53
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously.
21 Apr 2014 16:27:23

Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one.

We'll be updating documentation at the same time.

The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding.

The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun.

22 Apr 2014 08:51:09
We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly.
22 Apr 2014 09:50:44
MO texts from SIP2SIM are now using the new system - please let support know of any issues.
22 Apr 2014 12:32:07
Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages.
23 Apr 2014 09:23:20
We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP.
25 Apr 2014 10:29:50
We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues.
26 Apr 2014 13:27:37
We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know.
Started 14 Apr 2014
Previously expected 01 May 2014

11 Apr 2014 15:50:28
11 Apr 2014 15:53:42
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice.
Started 11 Apr 2014 15:50:28

07 Apr 2014 13:45:09
07 Apr 2014 13:52:31
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions.
10 Apr 2014 23:19:59
Completed earlier this evening.
Started 07 Apr 2014 13:45:09
Previously expected 10 Apr 2014 22:45:00

25 Sep 2013
18 Sep 2013 16:32:41
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout.
Started 25 Sep 2013

06 Jul 2015 12:49:42
We have been advised by Three of works on their network (22:30 8th July to 04:50 9th July 2015) Which will involve rerouting traffic from one of their nodes via alternate paths in their core. Although connections should automatically reroute, there will be brief amounts of packet loss. As a result, some partners may experience dropped connections. Any device reconnecting will automatically be routed via a new path. We apologise for any inconvenience this may cause and the short notice of this advisory.

12 Feb 2015 15:57:26
We have received the below PEW notification from one of our carriers that we take voice SIMS from. We have been advised by one of our layer 2 suppliers of emergency works to upgrade firmware on their routers to ensure ongoing stability. This will cause short drops in routing during the following periods: 00:00 to 06:00 Fri 13th Feb 00:00 to 04:00 Mon 16th Feb Although traffic should automatically reroute within our core as part of our MPLS infrastructure, some partners may experience disruption to SIM connectivity due to the short heartbeats used on SIM sessions.

Yesterday 17:24:51
[VoIP and SIMs] - New O2/EU SIMs - Info
31 Aug 16:21:27

We're sorry to say that we won't don't have any more of the O2/EU SIMs which roam to other UK networks when no O2 coverage. This is sadly not for technical reasons.

From now on we have O2 only SIMs (work in UK on O2), EU only SIMs (work in UK, EU, and world, but higher price), and O2/EU SIMs which switch, but only do O2 when in the UK and so do not roam to other UK networks.

We hope soon to have some new SIMs that have sensible UK pricing and roam to other UK networks, but we do not have a date for this. It is being worked on.

The web site has been updated, but if you normally order Uk roaming O2/EU SIMs, please take not that they will not be the same as you had previously.


Resolution The new O2/EU SIMs (which are always O2 in the UK) do have a nano-SIM cut out. We'll announce once we get some UK roaming SIMs again, and this is expected soon.
Started 1 Sep
Closed Yesterday 17:24:51
Previously expected 1 Sep

21 Sep 19:00:00
21 Sep 20:57:09
Some customers using us for their incoming email have had some delayed or duplicated messages this afternoon. This was due to a system problem which was resolved this evening. We do apologise for this and would like to reassure customers that messages haven't been lost.
Started 21 Sep 14:00:00
Closed 21 Sep 19:00:00

21 Sep 17:58:39
6 Sep 20:37:10

We've been doing a lot of expansion of our network this year. Earlier in the year we were working on increasing the capacity of our core London network by installing a 40G ring between our London sites and upgrading our core switch infrastructure.

Our aim is, and always has been, to provide low latency, uncongested Internet access for all our customers.

The bulk of our Internet traffic is carried over 'peering networks' in London such as LoNAP and LINX, where we get to peer directly with the major content providers. The rest of the traffic is carried over various 'transit' links which connect us to the rest of the Internet.

Today we made live a new transit link at our Telehouse site! This is using Level 3 and we have a second link going live at our Telecity/Equinix site in the next couple of weeks. This increases the capacity between us and the rest of the internet, and, more importantly, has reduced the number of hops to many places. In practice this means slightly lower latency to various places on the Internet - Good news for our customers.

21 Sep 17:59:06
The second link is now live!
Started 6 Sep 20:00:00
Closed 21 Sep 17:58:39

15 Sep 15:08:26
13 Sep 21:44:01
Apple released iOS 10 this evening, and looking at our traffic stats customers started downloading it at 6pm on the dot! Our network is coping with this increased load but it does look like it'll be the busiest evening we've ever had.

Looking back over the course of the evening BT connected lines look mostly unaffected, however some TalkTalk connected lines suffered with about an extra 30ms of latency and a decrease in download speeds. This was caused by one of our links to TalkTalk running at full capacity.

Started 13 Sep 18:00:00
Closed 15 Sep 15:08:26

13 Sep 16:50:14
[Broadband and Ethernet] - LNS roll-over upgrades - Info
9 Sep 15:41:27
We do upgrades on our L2TP Network Servers over night one at a time as a rolling upgrade from time to time. It has been around 6 months since the last upgrades, and new versions are now available. We are moving lines overnight as normal, using the preferred time as set on the control pages as a basis. This will be ongoing over the next couple of weeks, impacting a small proportion of customers each night with a PPP drop and reconnect. Depending on your equipment this could be anything from a fraction of a second outage to a few minutes. Thank you for your patience.
9 Sep 15:46:48
Note that careless, used for L2TP service, will be reloaded this evening.
13 Sep 16:26:24
Wholesale customers of those with L2TP relay may start to notice changes in LCP echo handling. Any issues please do let us know. We are planning some config options for this but the default for now is to provide us with loss/latency graphs on relayed connections which will be visible on the control pages. Other customers are not affected, so if this makes no sense, don't worry.
15 Sep 13:00:52
This is taking longer than expected and customers on the "G" LNS will be moved back to the "F" LNS this evening. We believe we have found an issue that was affecting half a dozen lines briefly this morning and should have that resolved to continue the upgrade over the weekend.
18 Sep 08:36:11
LNS roll out now progressing again as planned - we are aiming at one LNS every two days (over night, as usual).
Started 10 Sep
Expected close 30 Sep

12 Sep 18:12:54
12 Sep 18:12:30

The card printing system has had a few tweaks to help.

A couple of recently introduced bugs have been fixed.

The fields for VCARD can now have a default, which is useful for making VCARD entries for all the same company for example.

The fixed text field type can now have a multiple line text value - this makes it a lot easier to include standard text on a card template without having to make multiple entries, one for each line.

We'll work on any new improvements we can think of... Feel free to make suggestions.

Started 12 Sep 18:09:36
Previously expected 12 Sep 22:09:36

7 Sep 17:56:47
7 Sep 16:44:22
We are investigating a problem with some customers unable to send or receive SMSs through our service.
7 Sep 16:45:48
We have multiple SMS gateways and one of them looks to have a problem. We're taking that one out of service for the time being.
7 Sep 17:39:49
The fault has been narrowed down to a problem for SIP2SIM customers who use scripts to process outgoing SMSs from the SIM. Staff are still investigating.
7 Sep 18:42:31
This has been fixed.
Started 7 Sep 16:30:00
Closed 7 Sep 17:56:47

6 Feb 20:00:00
5 Feb 16:32:23
Some call recordings from today have been lost. We've identified the problem and have worked around it until we can fix it. We should be able to fix it with no customer impact. If you are expecting a call recording that you have not received, support can check whether the call was affected by this problem. The call recordings lost are in the minority and most calls were recorded as expected. This problem is unrelated to yesterday's problems with lack of ring tone!
Started 5 Feb 16:25:24
Closed 6 Feb 20:00:00

6 Sep 10:39:16
[Broadband and Ethernet] - New peering for gamers - Info
12 Aug 18:58:16

I am pleased to confirm we have peering with Blizzard, and are working on peering with Riot Games as well.

The Blizzard peering includes IPv6, as they have just fixed their EU realms IPv6.

Peering means we have a more direct route for traffic to and from our customers. We always have various fallbacks, but peering means the best connectivity for lowest latency in your games.

We always take latency seriously on broadband lines, and this just adds an extra edge.

6 Sep 10:39:16
Just a quick update, we are now peering with Riot Games over our LoNAP and LINX ports.
Started 12 Aug 18:55:57

30 Aug 18:15:02
30 Aug 18:15:02

Our plastic card printing service now allows multiple copies of the same card.

The original design was very much around ID cards where each card would have a different set of name, photo, barcode, other fields, and be printed once.

However, many people have used for things like business cards, wifi password cards, and all sorts where multiple prints of the same card are needed. So we have now allowed a quantity against each card.

There are also a few more standard templates now.

Started 30 Aug 18:13:24
Previously expected 30 Aug 22:13:24

12 Aug 18:21:20
[Broadband and Ethernet] - World of Warcraft IPv6 issues - Closed
24 Jul 19:42:01

It seems that once again Blizzard have some IPv6 issues that are impacting World of Warcraft.

This should be simple for them to fix. They have connectivity to their network, which seems to be very good (via LINX) with low pings, but the individual servers their end do not seem to be working on the IPv6 addresses the client is trying.

This could simply be a change of IPv6 SLAAC address following hardware changes, or could be something else, we don't know. They are not interested in investigating.

Normally a failure in one IP protocol is not a huge issue as clients fall back to the other version, e.g. trying IPv6 first then legacy IPv4. However the WoW client seems not to be doing this, simply hanging and eventually giving an error.

Obviously this has been reported to Blizzard, but I am sorry to say that they are taking the backwards step of taking no notice of IPv6 issues, effectively withdrawing support for the current version of Internet Protocol, IPv6

As such, customers may have to change their config on their game client not to try IPv6. This is obviously not something we would normally recommend as it removes the usual redundancy of having both protocols. It also gives the mistaken impression that IPv6 is somehow unreliable, rather than the more accurate impression that Blizzard are unreliable.

Hopefully they will fix it eventually. We have yet to see any official statement as to withdrawal of support for IPv6, a move that is very unusual for any company to take.

25 Jul 06:57:26
Blizzard have now said they have engineers working on this. No ETA.
26 Jul 20:19:57
An update from Blizzard suggests they have made changes to address this, but tests suggest it is not yet resolved, so still chasing.
27 Jul 10:38:54
Blizzard have made a number of changes to fix this issue. It is not quite fixed for everyone according to my tests, so they are working on it still.
7 Aug 08:54:09

Blizzard have said: "As mentioned previously, the game client is designed to take advantage of the IPV6 options simply by ticking the IPV6 option ingame and restarting the game. As IPV6 comes available, you will be automatically join it."

This is odd, as we have had IPv6 since 2002 and Blizzard have had it since 2011, so by that statement we should "join it".

At this point it is still not working in my test, still a routing loop within Blizzard's network.

As Blizzard mistakenly marked my ticket resolved, I have opened a new one. Sorry we don't have any more update on this.

7 Aug 14:33:25
There is a Blizzard EU forum post on this matter. http://eu.battle.net/forums/en/wow/topic/17612333772?page=1#post-8
Resolution Blizzard have finally resolved the issue.
Started 22 Jul
Closed 12 Aug 18:21:20

5 Aug 15:00:16
[Broadband and Ethernet] - L2TP services disrupted - Closed
5 Aug 14:43:08

L2TP services were disrupted and reconnected following a replacement of one of our LNSs that handled L2TP. The new box causes services to immediately switch from backup to live service automatically which caught us by surprise slightly.

There will be another brief disruption shortly then all should be well again.

Sorry for the inconvenience and very short notice.

Resolution The main L2TP server is fully back in service now.
Started 5 Aug 14:35:00
Closed 5 Aug 15:00:16
Previously expected 5 Aug 14:50:00

12 Aug 18:55:27
[Broadband and Ethernet] - Router issue - Closed
12 Aug 18:04:47

One of our edge routers appears to be playing up, we are investigating.

As normal traffic re-routed. We should have this sorted shortly.

12 Aug 22:51:15
Router was taken out of service until we could resolve the issue.
Resolution Router is back in service, thanks to staff going in to data centre this evening.
Started 12 Aug 18:00:00
Closed 12 Aug 18:55:27

9 Jun 22:00:00
[DNS, Email and Web Hosting] - Email problems to gmail addresses - Closed
9 Jun 17:01:30
We're noticing a large number of email to gmail accounts are being bounced back by gmail servers this afternoon. We are investigating the cause of this.
9 Jun 19:53:32
We have put in a temporary fix (sending via IPv4 instead of IPv6) and messages are new being delivered to gmail addresses without being bounced.
Started 9 Jun 16:00:00
Closed 9 Jun 22:00:00

18 Aug 16:00:00
18 Aug 16:30:43

This is affecting customers who we provide email services to for their domains.

We have noticed a small number of emails on the 17th and 18th August were being marked as spam when they are not spam. The problem should be fixed as of 16:30 on 18th August, but if you are missing email please check your spam folder(s) as the email may be in there.

Started 17 Aug 13:00:00
Closed 18 Aug 16:00:00

5 Sep 11:25:32
18 Aug 15:03:02
We have received advice from Three that they will be applying firmware upgrades to resolve issues identified on their new infrastructure. The maintenance will be carried out of the following days; 23rd August 2016 - 23:10 to 24th August 2016 - 04:30 BST 31st August 2016 5th September 2015 We are still awaiting a full schedule from Three regarding the times for the 31st August and 5th September and will post a further announcement once the times have been confirmed. This affects only our Data SIMs.
VoIP and SIMs Users Affected 15%
Started 18 Aug 14:51:51 by Carrier
Closed 5 Sep 11:25:32
Previously expected 5 Sep 04:30:00 (Last Estimated Resolution Time from Carrier)

10 Aug 18:00:00
29 Jul 17:19:16
There may be disruption to some outbound SMS between 07:00 and 09:00 BST on Tuesday the 9th of August 2016. Although the window is wide, the disruption is only expected to last 30 seconds. This is due to upstream carrier upgrades.
Started 9 Aug 07:00:00 by Carrier
Closed 10 Aug 18:00:00
Previously expected 9 Aug 09:00:00

8 Aug 10:37:44
[DNS, Email and Web Hosting] - SSL Certificates Updated - Info
8 Aug 10:37:44
We're updating SSL certificates for our email servers today. The old serial number is 11C4A8. The new serial number is 124247. Users who don't have the CAcert root certificate installed may see errors. This does not affect webmail. Details on http://aa.net.uk/cacert.html

6 Sep 20:50:57
5 Aug 15:31:35

We are making some changes to our Ip transit connectivity over the next few days. This should be pretty seamless as just mean traffic taking new routes. However, there is always a risk of issues.

Today we are doing a bit or preparation which means some minor routing changes for about 5% of our traffic, and which we also expect to have no impact.

1 Sep 11:01:04
Some of the preparation is being done this morning (1st Sep)
Resolution New transit installed
Started 5 Aug 15:30:00
Closed 6 Sep 20:50:57

3 Aug 16:18:26
[VoIP and SIMs] - SMS billing - Info
3 Aug 15:51:00

Customers may have noticed that some of the SMS itemisation has been wrong on the bills in terms of the time of the message.

Basically, some of the time sent logs were recorded in UTC even in the summer, but the time received was in local time, and so were some of the logs such as SIP2SIM usage.

We have corrected this today, so from now you should see all itemisation on local time.

Customers are reminded that if using our SIP2SIM service there is an airtime charge for sending a text (e.g. 2p) and if the text is to an external number such as someone else's mobile then there is an SMS charge (e.g. 6p), and these are separately itemised on the bill. At least after this change they should be next to each other and not an hour apart.

Sorry for any confusion.

Started 3 Aug 15:49:20

3 Aug 09:39:11
[Broadband and Ethernet] - Price reduction - Info
2 Aug 19:50:54

We are removing the extra £5/month charge for Home::1 customers on BT back haul with 80/20 capped FTTC. This means instead of being £40/month the 80/20 FTTC will be £35/month from your next bill.

This is consistent with our Talk Talk backhaul Home::1 terabyte services where the speed of FTTC does not impact the pricing.

Customers on BT backhaul Home::1 with 40/10 caps that have forecast speeds that could exceed 40Mb/s down or 10Mb/s can order a regrade on our broadband order form for £15. This does not restart the minimum term and usually happens next working day. This is no guarantee that your line will sync over 40/10, it simply changes the caps applied to the line sync.

Started 2 Aug 19:45:00

3 Aug 11:31:20
25 Jul 11:36:34
SIP2SIM may be affected for up to an hour between the hours of 20:00 on the 2nd of August and 06:00 on the 3rd of August. This is due to upstream GGSN upgrade work and the resulting testing of failover.
Started 2 Aug 20:00:00 by Carrier
Closed 3 Aug 11:31:20
Previously expected 3 Aug 06:00:00

25 Aug 14:42:10
27 Jul 15:08:24

We use two data centres in London, Equinix and (Telecity). Equinix have planned engineering works on their power feeds on the morning of 2nd August as follows:

Investigations into the power incident that occurred on Wednesday 20th July 2016 have concluded that the outage was caused by the malfunction of the protection module on the primary circuit breaker for the A power riser in Building 8/9 of Equinix's LD8 (Harbour Exchange) data centre. An identical protection module is installed on the B riser power supply. It is now essential that both circuit breakers are replaced. In order to carry out this work safely it is necessary to power down the affected circuits in accordance with the following schedule in the early hours of 2nd August 2016:

A Power riser:
Power down: 00:00
Reenergise: 01:00 or sooner once work is complete

B Power riser:
Power down: 02:00
Reenergise: 03:00 or sooner once work is complete

All our equipment is dual power - so we do not expect there to be any problems during this period.
Started 2 Aug
Closed 25 Aug 14:42:10
Previously expected 2 Aug 03:00:00

28 Jul 16:47:17
20 Jul 16:56:12
On Tuesday the 26th and Wednesday the 27th of July 2016, between 22:00 and 23:00, upstream router maintenance is expected to cause minor disruption to data SIM connectivity. The disruption should not be for more than a few minutes a time.
Started 26 Jul 22:00:00
Closed 28 Jul 16:47:17
Previously expected 27 Jul 23:00:00

20 Jul 12:41:10
20 Jul 08:18:13
We can see a major routing issue, and we are investigating. Not entirely clear the extent of the problem at this stage.
20 Jul 08:21:55
At the moment we are seeing what appears to be an issue with the links between London and Maidenhead. This can impact some Ethernet customers, VoIP, hosting, etc. However, a number of systems have a fallback for this, so many are still working, e.g our VoIP services are OK.
20 Jul 08:22:23
The link is partially back, and we are investigating the cause.
20 Jul 10:19:15
This is looking like a major fault within BT's core network. This is affecting many ISPs in the UK. We'll post more details when we can.
20 Jul 10:19:48
also see http://www.bbc.co.uk/news/technology-36844712 and http://www.ukinternetreport.co.uk/?p=d&id=215 Which are reporting the problem within BT's network.
20 Jul 10:30:51
This fault is affecting our communications with BT's systems. This means that our sales and support staff have very limited access to BT's ordering, diagnostic and fault reporting problems. so please to bear with us as staff will be struggling to do their work at the moment. (TalkTalk lines are ok though)
20 Jul 10:35:40
This is looking related to a power issue at a major london datacentre
20 Jul 10:48:05
We've had an update from BT regarding this issue, they say: There is an intermittent incident affecting the Openreach B2B and XML Gateways, as a result of this issue you may be experiencing a delay in the receipt of KCI's for both Fulfilment and Assurance requests as well as failures in Dialogue Services.
20 Jul 12:40:00
It looks like everything affected by this outage has returned to normal now.
20 Jul 12:40:06
It looks like everything affected by this outage has returned to normal now.
20 Jul 12:40:23
It looks like everything affected by this outage has returned to normal now.
Started 20 Jul 08:04:00
Closed 20 Jul 12:41:10

12 Jul 21:21:00
12 Jul 21:24:48
We've had some lines dropping and reconnecting this evening, between 20:50 and 21:20. Things seem back to normal now, but we're investigating the cause.
12 Jul 21:26:13
During the times above customers would have experienced routing problems, perhaps packetloss and VoIP problems. (although my Netflix stream was unaffected smiley )
12 Jul 21:34:11
Sorry about that everyone, early indicators is that a few DLS lines were being DOSd, and although our automated systems did drop the PPP and blackhole their IPs across the most of the internet, it did cause problems for some customers. We'll continue to monitor the network this evening and will do some more investigations in the morning as to the cause and how to help prevent this sort of thing in the future.
Started 12 Jul 20:50:00
Closed 12 Jul 21:21:00

8 Jul 04:00:00
[Broadband and Ethernet] - TalkTalk Outage - Closed
8 Jul 04:17:51
We lost TalkTalk lines shortly after 2am, and they were down for around 100 minutes. Our links to TalkTalk were either down or not passing traffic for this time. Things are returning to normal now, and we're monitoring the links and chasing TalkTalk for an explanation.
8 Jul 08:00:48
Service was restored at around 4am. We are investigating the cause. It looks like there was planned work within TalkTalk at their Telehouse location, but there was some problems with lines re-connecting via our Telecity connection. More information to follow as we get it.
Started 8 Jul 02:10:35 by Customer report
Closed 8 Jul 04:00:00

4 Jul 12:48:25
30 Jun 15:10:48
We are aware of problems affecting our L2TP and data SIM service, this is being investigated by our engineers now.
30 Jun 16:13:47
Apologise this has not yet been resolved, we are seeing further problems and it's still being investigated.
Started 30 Jun 15:09:09
Closed 4 Jul 12:48:25
Previously expected 30 Jun 19:09:09

19 Jun 13:39:55
[Broadband and Ethernet] - L2TP direct service - Info
3 Apr 16:36:34

Now that we have the new network upgrade we are hoping to properly launch the new L2TP specific service. Previously this was a somewhat informal service with somewhat complicated billing.

This new service is an L2TP over IP service for people using other broadband lines to access us and use our IPs and services including fixed IPv4 and IPv6 blocks. The plan is £10/month for a service capped at 100Mb/s with 1TB monthly allowance, clamping to 3Mb/s from next month's allowance on reaching allowance. Very similar to Home::1. No date yet but we are hoping to make this our standard L2TP service some time in April.

17 Apr 19:34:56
I'm sorry this is taking so long to sort - it is basically a tariff change, and order form, but we have been very keen to ensure the infrastructure is in place for this. I still hope to do this by end of month.
12 May 11:09:15
Information: http://aa.net.uk/broadband-l2tp.html
Order form: https://order.aa.net.uk/chaos2l2tp.html
Started 1 Apr
Previously expected 1 May

17 Jun 18:38:00
17 Jun 16:38:45
Talktalk have informed us that they are experiencing issues with one of their network routers in London. This has impacted all Talktalk lines in the surrounding area, covering ADSL, VDSL(FTTC) and EoFTTC lines since approximately 15:21. Affected areas are Canonbury, Lower Holloway, Kentish Town, Upper Holloway, Kingland Green and Clerkenwell.
17 Jun 19:56:24
All affected customers appear to be back online now, and TalkTalk have advised us that a fibre break has been repaired.
Resolution Openreach have repaired a break in TalkTalk's fibres.
Started 17 Jun 15:21:00 by TalkTalk
Closed 17 Jun 18:38:00
Cause TT

17 Jun 04:00:00
16 Jun 21:23:10
Talktalk have informed us that they are experiencing issues with their network core in Swansea. This has impacted all Talktalk lines in the surrounding area, covering ADSL, VDSL(FTTC) and EoFTTC lines since approximately 19:53. Talktalk have engineers on their way to the equipment to ensure it is correctly restarted, and currently believe the outage is due to a power failure in the BT exchange which hosts the equipment. BT Openreach have been informed of the power failure, and are also investigating. The following area codes and exchanges are impacted by this outage: 01792, 01554, 01269, 01639, Llanboidy, Amman Valley, Pontardawe, Port Talbot, Llangennech and Crosshands
16 Jun 21:48:54
We have received reports that the outage is also effecting the 01656 area code, and lines around Bridgend. We will post further updates when we have information from Talktalk.
Resolution The cause of this incident has not been determined. BT confirmed they had a fibre related issue in Merthyr Tydfil and restored our service by moving the Talk Talk circuits to a spare fibre cable. As no further issues have been reported, we can confirm closure to this incident with further investigations taking place under the Problem Management process.
Started 16 Jun 19:53:00 by TalkTalk
Closed 17 Jun 04:00:00
Previously expected 16 Jun 22:30:00

17 Jun 10:39:29
6 Jun 14:00:34
Our carrier for Data only SIMS are performing a software update on their LNS, this will mean a short outage for data SIMS for a minute or two between 10PM and 11PM on Thursday 9th June.
10 Jun 10:10:01
Aql will be performing another upgrade on the remaining infrastructure on Thursday 16th June 2016 at 22:00. This part of the migration to the new LTE infrastructure. They have advised that the patch has been tested in lab conditions, and no issues are expected, however in the event of a fault they revert the patch and investigate. As with last night some of our sessions may disconnect and reroute while the patch is applied and the LNS restarted, however no more than five minutes disruption is anticipated.
17 Jun 10:40:45
Looks like planned works completed successfully last night. We will post further information on our carriers LTE upgrades as we receive them.
Started 16 Jun 22:00:00 by Carrier
Closed 17 Jun 10:39:29

4 Jun 12:33:49
3 Jun 09:33:49

We now have new options for VDSL (Fibre to the cabinet capped at 80/20) with terabyte allowance and free install. This no longer forces you to include the phone line copper pair.

We also now have the option if you are taking over the phone line and it has a BT number you want to keep where we can port the number to VoIP. This is a cost for phone line install or take over.

This gives a lot more choice on the Terabyte services. It also allows ordering where the line is on a satellite exchange and could not get TT MPF (phone line) before. Do try the order form to see if it is available to you now.


4 Jun 12:33:49
We have worked out that around half of the customers currently with ADSL services could upgrade to VDSL. We are working on showing the estimated speed for FTTC upgrade on the control pages and bills.
Started 3 Jun

27 May 20:15:12
27 May 18:59:26

A number of customers were emailed that their 1st of month DD was cancelled.

This process was stopped as soon as we realised what was happening so not many people will be affected. You will receive normal DD collection notices in due course for your monthly bills or have collection within 3 working days as per exiting notice (so may be 3rd instead of 1st) as per DD rules.

Our DD management system is coded to be as cautious and customer focused as possible and a minor error meant it was not happy with the DDs for regular collections being in place before the invoice was issued. This has been corrected now.

Sorry for any confusion caused.

Started 27 May 18:00:00
Previously expected 27 May 22:56:44

27 May 04:30:00
27 May 02:24:53
At around 1:51 today TT dropped offline. We are investigating.
27 May 02:56:38
We've seen reports of other ISPs lines via TT also having the same problem.
27 May 03:16:06
This looks very much like a major outage within TalkTalk. Our BGP session to them that carry our traffic is down at one datacentre and up and our second datacentre but with no prefixes being advertised to us from TalkTalks side.
27 May 03:58:05
We are seeing some lines starting to reconnect.
27 May 03:58:26
We are seeing some lines starting to reconnect.
27 May 04:01:19
No word from TT as to the cause of this, but we'll update this post in due course.
27 May 04:14:03
Majority of lines are back on now - waiting for the last ones to reconnect.
Resolution It looks like this was caused by some TT planned work of which we were unaware, and clearly did not go to plan. We are trying to get a full explanation from TT. In the mean time the problem is resolved. If you are off line, try restarting your router to force it to reconnect.
Started 27 May 01:51:00
Closed 27 May 04:30:00