| Details Posted: Today 13:02:21 |
We are working on solving a problem that we're currently seeing with IMAP indexing on our mail servers. The symptoms customers are likely to see are small oddities such as emails appearing not to move between folders, or appearing twice. This problem is only index related and so doesn't actually affect the emails themselves. This problem is not causing email to be lost. |
| Update
Today 16:47:26 | We are still investigating a proper fix for this problem, but in the mean time we are making changes that should work around it. There is a small risk that if you are using Sieve filtering that it may stop working. If that is the case, please contact support for assistance |
| Details Posted: 14 Aug 11:48:45 |
A couple of our incoming mail servers have gone down due to a power issue in the datacentre. Our other mail servers have picked up the load however there would have been a delay in receiving mail during the fail over. Incoming mail should be fine now and we are investigating what caused the issue. |
| Started | 14 Aug 11:10:38 by AA Staff |
| Details Posted: 11 Aug 18:44:38 |
We're needing to restart the 'e.gormless' LNS - this will cause PPP to drop for customers. Update to follow. |
| Update
11 Aug 18:46:19 | Customer on this LNS should be logging back in - (if not already) |
| Update
11 Aug 19:00:27 | There are still some lines left to log back in, but most are back now |
| Update
11 Aug 19:10:47 | Most customers are back now. |
| Update
13 Aug 12:12:47 | This happened again on Sunday morning, and again a restart was needed. The underlying problem is being investigated. |
| Resolution | We have now identified the cause of the issue that impacted both "careless" and "e.gormless". There is a temporary fix in place now, which we expect to hold, and the permanent fix will be deployed on the next rolling update of LNSs. |
| Started | 11 Aug 18:30:00 |
| Closed | 14 Aug 09:14:59 |
| Details Posted: 9 Aug 10:21:04 |
At 02:00 on Friday we will be performing planned maintenance on one of our cross-London fibres. We do not anticipate any service disruption, however any work on the core network should be considered at risk. |
| Update
11 Aug 02:01:43 | The planned work window has now started. |
| Update
11 Aug 02:27:04 | Planned works completed without any issues. |
| Started | 11 Aug 02:00:00 by AA Staff |
| Closed | 11 Aug 02:26:48 |
| Previously expected | 11 Aug 03:00:00 (Last Estimated Resolution Time from AAISP) |
| Details Posted: 17 Jun 15:24:16 |
We've seen very slight packet loss on a number of TalkTalk connected lines this week in the evenings. This looks to be congestion, it's may show up on our CQM graphs as a few pixels of red at the top of the graph between 7pm and midnight. We have an incident open with TalkTalk. We moved traffic to our Telehouse interconnect on Friday afternoon and Friday evening looked to be better. This may mean that th econgestion is related to TalkTalk in Harbour Exchange, but it's a little too early to tell at the moment. We are monitoring this and will update again after the weekend. |
| Update
19 Jun 16:49:34 | TalkTalk did some work on the Telehouse side of our interconnect on Friday as follows: "The device AA connect into is a chassis with multiple cards and interfaces creating a virtual switch. The physical interface AA plugged into was changed to another physical interface. We suspect this interface to be faulty as when swapped to another it looks to have resolved the packet loss." We will be testing both of our interconnects individually over the next couple of days. |
| Update
20 Jun 10:29:05 | TalkTalk are doing some work on our Harbour Exchange side today. Much like the work they did on the Telehouse side, they are moving our port. This will not affect customers though. |
| Update
28 Jun 20:46:34 | Sadly, we are still seeing very low levels of packetloss on some TalkTalk connected circuits in the evenings. We have raised this with TalkTalk today, they have investigated this afternoon and say: "Our Network team have been running packet captures at Telehouse North and replicated the packet loss. We have raised this into our vendor as a priority and are due an update tomorrow." We'll keep this post updated. |
| Update
29 Jun 22:12:17 | Update from TalkTalk regarding their investigations today:- Our engineering team have been working through this all day with the Vendor. I have nothing substantial for you just yet, I have been told I will receive a summary of today's events this evening but I expect the update to be largely "still under investigation". Either way I will review and fire an update over as soon as I receive it. Our Vendor are committing to a more meaningful update by midday tomorrow as they continue to work this overnight. |
| Update
1 Jul 09:39:48 | Update from TT: Continued investigation with Juniper, additional PFE checks performed. Currently seeing the drops on both VC stacks at THN and Hex. JTAC have requested additional time to investigate the issue. They suspect they have an idea what the problem is, however they need to go through the data captures from today to confirm that it is a complete match. Actions Juniper - Review logs captured today, check with engineering. Some research time required, Juniper hope to have an update by CoB Monday. Discussions with engineering will be taking place during this time. |
| Update
2 Jul 21:19:57 | Here is an example - the loss is quite small on individual lines, but as we are seeing this sort of loss on many circuits and the same time (evenings) it make this more severe. It's only due to to our constant monitoring that this gets picked up.  |
| Update
3 Jul 21:47:31 | Today's update from Talktalk: "JTAC [TT's vendor's support] have isolated the issue to one FPC [(Flexible PIC Concentrator] and now need Juniper Engineering to investigate further... unfortunately Engineering are US-based and have a public holiday which will potentially delay progress... Actions: Juniper - Review information by [TalkTalk] engineering – Review PRs - if this is a match to a known issue or it's new. Some research time required, Juniper hope to have an update by Thursday" |
| Update
7 Jul 08:41:26 | Update from TalkTalk yesterday evening: "Investigations have identified a limitation when running a mix mode VC (EX4200’s and EX4550's), the VC cable runs at 16gbps rather than 32gbps (16gbps each way). This is why we are seeing slower than expected speeds between VC’s. Our engineering team are working with the vendor exploring a number of solutions." |
| Update
17 Jul 14:29:29 | Saturday 15th and Sunday 16th evenings were a fair bit worse than previous evenings. On Saturday and Sunday evening we saw higher levels of packet loss (between 1% and 3% on many lines) and we also saw slow single TCP thread speeds much like we saw in April. We did contact TalkTalk over the weekend and this has been blamed on a faulty card that TalkTalk had on Thursday that was replaced but has caused traffic imbalance on this part of the network. We expect things to improve but we will be closely monitoring this on Monday evening (17th) and will report back on Tuesday. |
| Update
22 Jul 20:23:24 | TalkTalk are planning network hardware changes relating to this in the early hours of 1st August. Details here: https://aastatus.net/2414 |
| Update
1 Aug 10:42:58 | TalkTalk called us shortly after 9am to confirm that they had completed the work in Telehouse successfully. We will move traffic over to Telehouse later today and will be reporting back the outcome on this status post over the following days. |
| Update
3 Aug 11:23:55 | TalkTalk confirmed that they have completed the work in Harbour Exchange successfully. Time will tell if these sets of major work have helped with the problems we've been seeing on the TalkTalk network; we will be reporting back the outcome on this status post early next week. |
| Update
10 Aug 16:39:30 | The packetloss issue has been looking better since TalkTalk completed their work. We are still wanting to monitor this for another week or so before closing this incident. |
| Started | 14 Jun 15:00:00 |
| Details Posted: 10 Aug 16:17:39 |
We have had a problem with our call recording and voicemail systems. This problem started on Wednesday afternoon and was fixed by 13:20 today. This has meant that some call recordings have been lost and there would have been times when callers would have heard silence when they reached voicemail. |
| Started | 9 Aug 16:00:00 |
| Closed | 10 Aug 13:20:00 |
| Details Posted: 7 Aug 15:12:51 |
We've had two incidents of one of our L2TP LNSs locking up over the weekend and causing disruption to some L2TP connected customers. Therefore will be swapping over the hardware in the morning of Tuesday 8th August at around 6AM. At this time L2TP sessions will be dropped and then re-establish shortly after on the new hardware. |
| Resolution | Cancelled! Following discussions with FireBrick developers we've decided not to swap the hardware in this case. The fault is likely to be software related and instead we've changed the LNSs configuration slightly and are working on adding extra debugging in to the software which will be loaded once that coding work has been completed, which should be in a couple of days time. |
| Started | 8 Aug 06:00:00 |
| Closed | 7 Aug 22:06:11 |
| Details Posted: 27 Jul 14:28:32 |
We are moving our Web IRC client (https://webirc.aa.net.uk/) off our network to increase availability in the unlikely event of an MSO. This work will be carried out on Tuesday, and will be carried out during support hours so that staff are available to explain to anyone who is unable to connect to it. |
| Started | 1 Aug 11:00:00 |
| Closed | 1 Aug 17:00:00 |
| Previously expected | 1 Aug 12:00:00 |
| Details Posted: 2 Aug 16:44:03 |
Between 2am and 3am we will be making changes to the configuration of our core switches, this will be to aid in our diagnostics in regards to the MSO that occurred in July. We expect there to be a few short disruptions to routing and there may be a PPP drop or two for some customers during this window. |
| Update
4 Aug 02:01:23 | This work is about to commence. |
| Update
4 Aug 02:40:56 | We have four small jobs to do, the first has been completed without any disruption. We're moving the estimated completion time to 4AM though, so as to give us a bit more time. |
| Update
4 Aug 02:58:57 | The second job has been completed without any disruption. |
| Update
4 Aug 03:21:34 | The third job has been completed, it did cause some routing issues for around 10 minutes. |
| Resolution | This work has been completed. |
| Started | 4 Aug 02:00:00 |
| Closed | 4 Aug 03:42:00 |
| Previously expected | 4 Aug 03:00:00 |
| Details Posted: 24 Jul 13:32:48 |
The following is from TalkTalk and relates to this incident post: https://aastatus.net/2401 and also the other similar planned work for 1st August: https://aastatus.net/2414 This is happening between 00:00 and 06:00 Wednesday 3rd August. "The wholesale LTS platform has been suffering from packet loss and congestion over the last few months. Juniper have now advised that this is due to a limitation with the hardware we have and we need to carry out some essential vendor recommended maintenance on our DSL Interconnects platform/switch at Harbour Exchange in order to fix customer slow throughput.This RFC is to rebuild ldn-vc1.thn from a mixed EX4200/EX4550 estate to a entirely EX4550 virtual switch. We have a limitation with our EX4550's - running in a mostly EX4200 VC (mixed mode), whereby the VC cables/modules will not support any more bandwidth past 16gbps (each way). This limitation is because we are running a mixed mode VC. We therefore need to upgrade this VC to run all EX4550s which will mean the VC cables/modules will support 32gbps and the hope is that low speed issues will be resolved."During this work AAISP will move traffic over to our Telehouse interconnect so as to minimise the impact on our customers, however there still may be drops between midnight and 4am. |
| Update
3 Aug 02:00:12 | This work is under way, we do have a number of lines that dropped and have not come back yet. |
| Update
3 Aug 02:30:23 | There are still a few hundred circuits offline, they are reconnecting slowly. We suspect this work by talk talk has put strain on other parts of their network which is causing delays in logging back in. |
| Update
3 Aug 05:09:09 | The work has now finished, any customers still off line should try restarting their router so as to force it to try logging in. |
| Resolution | TalkTalk confirm that their work completed successfully. From our side, we did see some line drop PPP and reconnect, however some lines did take a long time (up to 2 hours) to reconnect. We have raised this with TalkTalk. Further updates to the evening packetloss issue which this work should fix will be posted against this incident: https://aastatus.net/2401 |
| Closed | 3 Aug 05:07:34 |
| Previously expected | 3 Aug 00:04:00 |
| Details Posted: 22 Jul 20:21:08 |
The following is from TalkTalk and relates to this incident post: https://aastatus.net/2401 and also similar work on 3rd August https://aastatus.net/2415 "The wholesale LTS platform has been suffering from packet loss and congestion over the last few months. Juniper have now advised that this is due to a limitation with the hardware we have and we need to carry out some essential vendor recommended maintenance on our DSL Interconnects platform/switch at Telehouse North in order to fix customer slow throughput.This RFC is to rebuild ldn-vc1.thn from a mixed EX4200/EX4550 estate to a entirely EX4550 virtual switch. We have a limitation with our EX4550's - running in a mostly EX4200 VC (mixed mode), whereby the VC cables/modules will not support any more bandwidth past 16gbps (each way). This limitation is because we are running a mixed mode VC. We therefore need to upgrade this VC to run all EX4550s which will mean the VC cables/modules will support 32gbps and the hope is that low speed issues will be resolved."During this work AAISP will move traffic over to our Harbour Exchange interconnect so as to minimise the impact on our customers, however there still may be drops between midnight and 4am. |
| Update
31 Jul 16:25:24 | TalkTalk traffic has been moved over to Harbour Exchange, away from where this overnight work is happening. |
| Update
1 Aug 03:25:09 | A number of TT lines dropped their PPP session at 00:38, byt 01:30 most had reconnected. The planned work window is still open and there may be a further drop. |
| Update
1 Aug 07:17:14 | It looks like TalkTalk completed their work at around 06:30. We've not yet received an update from TalkTalk as to how it went from their side, but we'll update this post when we do. There may be a very small number of customers who are still off line, these may need a router reboot so as to force it to log back in. |
| Update
1 Aug 10:42:11 | TalkTalk called us shortly after 9am to confirm that they had completed the work in Telehouse successfully. We will move traffic over to Telehouse later today and will be reporting back the outcome in status post: https://aastatus.net/2401 |
| Update
1 Aug 10:45:19 |  |
| Closed | 1 Aug 06:30:00 |
| Previously expected | 1 Aug 04:00:00 |
| Details Posted: 10 Jul 18:12:06 |
We're updating SSL certificates for our email servers today. The old serial number is 12AD7B. The new serial number is 130CAB. Users who don't have the CAcert root certificate installed may see errors. This does not affect webmail or outgoing SMTP. Details on http://aa.net.uk/cacert.html We have a new email proxy that should fix these problems; those affected by this can try setting their incoming mail server to mail.aa.net.uk (TLS only, no STARTTLS). Please note that this is not yet "launched" and is therefore not yet officially supported. More info here: https://aastatus.net/2407 |
| Started | 10 Jul 15:00:00 |
| Details Posted: 13 Jul 16:37:48 |
Next week we will be making a change to our incoming email servers which is aimed at reducing the amount of spam email from reaching customer's mailboxes. Historically, we've been purposefully cautious of rejecting email outright and prefer the method of marking messages as spam based on a 'spam score'. Customers have options as to what score to mark and to reject messages. However, due to the high number of spam messages, the cost of scanning each message and the extremely low risk of false positives we're going to introduce rejecting messages from IP addresses that are known spammers. Specifically, the change is to reject messages from email servers that are listed in the "Spamhaus" block lists. These lists contain IP addresses that are known to be spam senders or are compromised machines in some way. Spamhaus have "a long-held reputation as having a false positive rate so low as to be unmeasurable and insignificant". Many mail servers around the world use these same block lists, but if you are in anyway concerned about this then please to get in touch with us. |
| Update
19 Jul 16:15:43 | We are making these changes at the moment. (Wednesday afternoon). As described above, we're not expecting this to impact customers in a negative way. |
| Started | 13 Jul 16:30:00 |
| Details Posted: 13 Jul 11:21:37 |
We are investigating an issue with some TalkTalk lines that disconnected at 10:51 this morning, most have come back but there are about 20 that are still off line. We are chasing TalkTalk business. |
| Update
13 Jul 11:23:50 | Latest update from TT..... We have just had further reports from other reseller are also experiencing mass amount of circuit drops at the similar time. This is currently being investigated by our NOC team and updates to follow after investigation. |
| Started | 13 Jul 10:51:49 by AAISP Pro Active Monitoring Systems |
| Closed | 13 Jul 18:00:00 |
| Previously expected | 13 Jul 15:19:49 |
| Details Posted: 27 Jul 23:22:55 |
We are doing routine maintenance of one of our email servers this evening and at the moment one of the servers is sulking and not allowing logins. Some customers may be seeing login error messages when trying to receive email this evening. We're taking the affected server out of the 'pool' and expect to solve this shortly. |
| Update
27 Jul 23:39:18 | Fixed! Sorry for the disruption. |
| Started | 27 Jul 22:30:00 |
| Closed | 27 Jul 23:38:13 |