| Details 17 Jul 10:08:44 |
Our email services can learn spam/non-spam messages. This feature is currently down for maintenance as we work on the back-end systems. This means that if you move email in to the various 'learn' folders they will stay there and will not be processed at the moment. For the moment, we advise customers not to use this feature. Will will post updates in the next week or so as we may well be changing how this feature works. This should not affect any spam scores etc, but do contact support if needed. |
| Started | 17 Jul 10:00:00 |
| Update was expected | Today 11:00:00 |
| Details 15 Jul 10:41:58 |
We are reworking the SMS/twitter/email line up/down notifications and hope to have the new system launched later this week. There may be slightly different wording of the messages. |
| Update
15 Jul 18:06:51 | We're looking to do this in stages. i.e. switch over emails then texts then tweets or something like that. So please bear with us. Ideally the changes should not lose any messages. |
| Update
17 Jul 09:21:13 | We have switched over to the new system - the most noticable change is that SMS and Tweets are now independant. You can have either or both if you require - settings are on the control pages. SMS still has a back off if you have lots of line flaps, but tweets and emails do not delay. Do let us have any feedback on the new system. |
| Started | 16 Jul |
| Previously expected | 20 Jul |
| Details 7 Jul 15:34:10 |
There is a problem activating some new data SIMs which being is caused by a problem at Three. This has been escalated within Three and we expect an update by the end of the day. Please note that this only affects the activation of new Three data SIMs. Existing data SIMs are not affected, nor is the data on our O2 based voice SIMs. |
| Update
8 Jul 09:15:12 | Three are still working on this problem as a Priority 1 case. We should have updates every 4 hours and will post them here. |
| Update
8 Jul 16:19:30 | An update: "Three have advised that the interface card that they believe was causing the fault has been replaced as of 01:00 this morning. We are continuing to see API failures, however these have lessened since 11:00. We have asked Three to investigate further and feed back." |
| Update
9 Jul 13:12:12 | This seems to be fixed for new SIMs being activated now. Customers with SIMs which attempted to activate SIMs over the past few days may be stuck in a broken partially-activated state. We are getting these reset by the carrier so we can activate them again. |
| Update
14 Jul 16:11:18 | Activating seems broken again. We've reported it and are awaiting an update. |
| Update
15 Jul 17:18:11 | We do have a small number of SIMs that are in a 'stuck' state, and we're waiting for the carrier to clear them. |
| Update
17 Jul 15:48:25 | Three are still having further problems with activating SIMS. This has been happening for the last couple of days, and is on going. This has been raised with Three. |
| Update
21 Jul 15:33:17 | This is still an ongoing problem, the latest from our supplier is:
"The current understanding is that some of Three's load balanced platforms are returning an inconsistent response when the requests are submitted. This is in turn causing our platform to lock the SIMs from further activation attempts so that they can be investigated." |
| Update
Today 11:27:28 | This is still ongoing, but is more of an intermittent problem. We have been able to activate most SIMs. |
| Started | 6 Jul 12:00:00 |
| Details 3 Jun 18:20:39 |
The router upgrades went well, and now there is a new factory release we'll be doing some rolling upgrades over the next few days. Should be minimal disruption. |
| Update
3 Jun 18:47:21 | First batch of updates done. |
| Started | 3 Jun 17:00:00 |
| Previously expected | 7 Jun |
| Details 23 May 12:07:01 |
Our legacy 'A' VoIP platform will be removed from service on November 10th 2014. This platform is our original Asterisk based system; it is now old, tired and we have a better VoIP platform: our FireBrick based 'Voiceless' platform. We have created a wiki page with details for customers needing to move platforms: http://wiki.aa.org.uk/VoIP_-_Moving_Platform We will be contacting customers individually by email in the coming weeks. Our Asterisk platform has historically been used by customers using IAX or who have phones behind NAT. Our current 'Voiceless' platform should work just as well for phones behind NAT. This does mean that we will no longer be providing a IAX service. The wiki pages have details on configuring asterisk to use SIP instead. The feature set on Voiceless is the same (if not better) than on 'A' (apart from the IAX support). Please see the wiki page for more information: http://wiki.aa.org.uk/VoIP_-_Moving_Platform |
| Started | 23 May 12:00:00 by AAISP Staff |
| Update expected | 10 Nov 11:00:00 |
| Expected close | 10 Nov 10:00:00 |
| Details 13 Apr 17:29:53 |
We handle SMS, both outgoing from customers, and incoming via various carriers, and we are now linking in once again to SMS with mobile voice SIM cards. The original code for this is getting a tad worn out, so we are working on a new system. It will have ingress gateways for the various ways SMS can arrive at us, core SMS routing, and then output gateways for the ways we can send on SMS. The plan is to convert all SMS to/from standard GSM 03.40 TPDUs. This is a tad technical I know, but it will mean that we have a common format internally. This will not be easy as there are a lot of character set conversion issues, and multiple TPDUs where concatenation of texts is used. The upshot for us is a more consistent and maintainable platform. The benefit for customers is more ways to submit and receive text messages, including using 17094009 to make an ETSI in-band modem text call from suitable equipment (we think gigasets do this). It also means customers will be able to send/receive texts in a raw GSM 03.40 TPDU format, which will be of use to some customers. It also makes it easier for us to add other formats later. There will be some changes to the existing interfaces over time, but we want to keep these to a minimum, obviously. |
| Update
21 Apr 16:27:23 | Work is going well on this, and we hope to switch Mobile Originated texting (i.e. texts from the SIP2SIM) over to the new system this week. If that goes to plan we can move some of the other ingress texting over to the new system one by one. We'll be updating documentation at the same time. The new system should be a lot more maintainable. We have a number of open tickets with the mobile carrier and other operators to try and improve the functionality of texting to/from us. These cover things like correct handling of multi-part texts, and correct character set coding. The plan is ultimately to have full UTF-8 unicode support on all texts, but that could take a while. It seems telcos like to mess with things rather than giving us a clean GSM TPDU for texts. All good fun. |
| Update
22 Apr 08:51:09 | We have updated the web site documentation on this to the new system, but this is not fully in use yet. Hopefully this week we have it all switched over. Right now we have removed some features from documenation (such as delivery reports), but we plan to have these re-instated soon once we have the new system handling them sensibly. |
| Update
22 Apr 09:50:44 | MO texts from SIP2SIM are now using the new system - please let support know of any issues. |
| Update
22 Apr 12:32:07 | Texts from Three are now working to ALL of our 01, 02, and 03 numbers. These are delivered by email, http, or direct to SIP2SIM depending on the configuration on our control pages. |
| Update
23 Apr 09:23:20 | We have switched over one of our incoming SMS gateways to the new system now. So most messages coming from outside will use this. Any issues, please let support know ASAP. |
| Update
25 Apr 10:29:50 | We are currently running all SMS via the new platform - we expect there to be more work still to be done, but it should be operating as per the current documentation now. Please let support know of any issues. |
| Update
26 Apr 13:27:37 | We have switched the DNS to point SMS to the new servers running the new system. Any issues, please let support know. |
| Started | 14 Apr |
| Previously expected | 1 May |
| Details 11 Apr 15:53:42 |
There is a problem with the C server and it needs to be restarted again after the maintenance yesterday evening. We are going to do this at 17:00 as we need it to be done as soon as possible. Sorry for the short notice. |
| Started | 11 Apr 15:50:28 |
| Details 7 Apr 13:52:31 |
We will be carrying out some maintenance on our 'C' SIP server outside office hours. It will cause disruption to calls, but is likely only to last a couple of minutes and will only affect calls on the A and C servers. It will not affect calls on our "voiceless" SIP platform or SIP2SIM. We will do this on Thursday evening at around 22:30. Please contact support if you have any questions. |
| Update
10 Apr 23:19:59 | Completed earlier this evening. |
| Started | 7 Apr 13:45:09 |
| Previously expected | 10 Apr 22:45:00 |
| Details 18 Sep 2013 16:32:41 |
We have received notification that Three's network team will be carrying out maintenance on one of the nodes that routes our data SIM traffic between 00:00 and 06:00 on Weds 25th September. Some customers may notice a momentary drop in connections during this time as any SIMs using that route will disconnect when the link is shut down. Any affected SIMs will automatically take an alternate route when they try and reconnect. Unfortunately, we have no control over the timing of this as it is dependent on the retry strategy of your devices. During the window, the affected node will be offline therefore SIM connectivity should be considered at risk throughout. |
| Started | 25 Sep 2013 |
| Details 8 Jul 13:19:27 |
We have the SIMs and should be able to ship today with any luck. We still have some checks to do. However, I have enabled ordering. See http://aa.net.uk/telecoms-sip2sim.html
We have an umber of people wanting to upgrade the UK only O2 SIMs to roaming. For this month these are being supplied at zero buy cost with postage added at cost. After this month we will be back to £5+VAT. |
| Update
8 Jul 16:33:21 | We have not been able to finish testing yet - just waiting on a minor change in the mobile network. We have the SIMs and hope to be able to ship tomorrow. |
| Update
8 Jul 17:48:31 | The good news is that we now have the keys for the SIMs for provisioning now, and are all ready to go on that front. I see several orders waiting to ship.
However we have just been advised that there is a snag in the mobile operator tariffing systems which may take a couple of days to sort. Until that is sorted we cannot really send SIMs out as the call rates will be all screwed up. So please bear with us. |
| Update
9 Jul 12:55:43 | We are still waiting on an update from the mobile carrier. We have checked, and the tariffing is way out at present which means we still can't ship the SIMs. It could be a couple more days by the look of it. |
| Update
11 Jul 08:18:04 | I am sorry to say that we are being told it may be another week before the tariff is sorted. Please bear with us. |
| Update
18 Jul 07:04:21 | Still waiting on the mobile operator, sorry for delay. |
| Update
18 Jul 13:14:18 | We should be posting SIMs Monday - or possibly today - we are just going through testing with the mobile carrier now to confirm everything is set up right. Thank you all for your patience. |
| Update
18 Jul 17:07:44 | Still testing, I'm afraid - going well but not quite there yet, so we are looking at Monday for the SIMs to be shipped. |
| Update
21 Jul 15:30:37 | Still not sorted - not sure if they will ship today. I appreciatet hat this is very frustrating. |
| Update
21 Jul 17:47:58 | The good news is that testing has finally got close enough to ship the SIMs, in that the main part of the tariffing code is right. We have a couple of small snags to sort and texts from the mobile while roaming are not working yet but hopefully that can be sorted tomorrow. It is not a reason to avoid shipping them I feel. |
| Update
21 Jul 18:59:16 | OK SIMs will go out tomorrow. At present texts from the mobile while on EU profile not working, but we expect that to be sorted very soon. |
| Update
Tuesday 13:51:24 | SIMs are shipping today - at present outgoing texts are not working on the EU profile but we expect that to be fixed very soon. |
| Update
Tuesday 18:49:07 | All of the back orders were shipped today. We expect to put prices back to £5 to buy the SIM from next month. |
| Started | 8 Jul 13:00:00 |
| Details 21 Jul 15:49:07 |
We now have a new official URL for our Status Pages: https://aastatus.net The reason for the change is to make the status pages completely independent of any AAISP infrastructure. They were already hosted on a server in Amsterdam our side of our network but now the DNS is independent too. Anyone using status.aa.net.uk should update to use aastatus.net |
| Started | 21 Jul 15:45:00 |
| Details 19 Jul 12:21:16 |
There are some ordering issues which we expect to be resolved in a couple of hours, sorry for any inconvenience. Please try later. |
| Update
19 Jul 12:58:10 | If you are trying to do a top-up on Home::1, for now, we suggest changing to auto-topup. This will allow the line to work whilst waiting for top-up to be working again. Once the top up then happens you can change back. |
| Resolution | Work completed |
| Started | 19 Jul 11:20:00 |
| Closed | 19 Jul 13:05:20 |
| Previously expected | 19 Jul 14:00:00 |
| Details 17 Jul 16:23:15 |
We have a few reports from customers, and a vague Incident report from BT that looks like there may be PPP problem within the BT network which is affecting customers logging in to us. Customers may see their ADSL router in sync, but not able to log in (no PPP). |
| Update
17 Jul 16:40:31 | This looks to be affecting BT ADSL and FTTC circuits. A line which tries to log in may well fail. |
| Update
17 Jul 16:42:34 | Some lines are logging in successfully now. |
| Update
17 Jul 16:54:15 | Not all lines are back yet, but lines are still logging back in, so if you are still offline it may take a little more time. |
| Resolution | This was a BT incident, reference IMT26151/14. This was closed by BT at 17:45 without giving us further details about what the problem was or what they did to restore service. |
| Started | 17 Jul 16:00:00 |
| Closed | 17 Jul 17:45:00 |
| Details 16 Jul 10:13:15 |
Just to be clear on our policy here - when DRIPA comes in to force, and if A&A become subject to a retention notice for all customers, we aim to work on all practical legal means to minimise the amount of data retained under that legislation - making full use of the bad wording in the Schedule in the 2009 regulations where possible. We also aim to clearly publish what is retained under such a notice and what steps we have taken to minimise such data. Such steps may mean separate companies running email or other services, or even hosting some servers outside the UK, if those are practical steps we can take.
Why? Because blanket mass surveillance is illegal under EU law as it is against our basic human right to privacy as decided by a court, that's why! Feel free to comment on my blog. |
| Started | 15 Jul |
| Details 15 Jul 12:52:51 |
The usage reports sent on 15th of the month for customers that have requested it have apparently not all worked. Some were blank.
These are being resent now, so apologies if you get two of them. |
| Started | 15 Jul |
| Details 11 Jul 17:00:48 |
The "B" LNS restarted today, unexpectedly. All lines reconnected within minutes (however fast the model retries). We'll clear some traffic off the "D" server back to the "B" server later this evening. |
| Resolution | We're investigating the cause of this. |
| Broadband Users Affected | 33% |
| Started | 11 Jul 11:03:52 |
| Closed | 11 Jul 11:03:55 |
| Details 10 Jul 19:18:35 |
We are seeing a problem with BT 21CN ADSL and FTTC circuits being unable to log in since approximately 18:00 today. Existing sessions are working fine but are failing to reconnect when they drop. 20CN ADSL and TalkTalk backhaul circuits are working fine.
BT have raised incident IMT25152/14 which looks to be related, but just says they are investigating a problem. |
| Update
10 Jul 22:16:28 | BT have reported that service should have been restored as of 20:10 this evening.
Customers who are still having problems should attempt to re-connect as they may be stuck on a BT holding session. Anyone still having problems after doing that should contact tech support. |
| Started | 10 Jul 17:15:00 |
| Closed | 10 Jul 20:10:00 |
| Cause | BT |
| Details 9 Jul 15:27:03 |
We are experiencing problem with VoIP platform, this is affecting calls for customers. We are investigating. |
| Update
9 Jul 15:38:20 | Outgoing and 'internal' calls are OK - the problem is with inbound calls not working. |
| Update
9 Jul 15:42:38 | Actually - outbound and internal calls over IPv6 were working, but over IP4 failing. |
| Update
9 Jul 15:43:10 | The problem has been found. Calls should start working again shortly. |
| Resolution | VoIP has been a bit off for a couple of days with two unrelated incidents. Not good. It looks like in both cases many incoming calls were failing. Yesterday was an issue with the code that steers calls - we made a change done in an emergency for a customer (who was trying to abuse SIP and not have to pay for lots of licence fees on his VoIP switch). The change worked for him, but broke many other customers (not all, and not our office). The testing done before this was deployed did not pick up the issue as the test calls were the type that worked (like calls to our office). It was spotted and fixed very quickly. We're trying to work out how we can create more comprehensive tests in future - we want to be agile and responsive to customer needs, but we need any changes to be robust and not cause problems. Today was very different. We noticed an issue on one called server impacting some SIP2SIM customers, so took it out of service to investigate. We have multiple call servers, and switching the active servers is a routine task that can be done in cases just like this to ensure service continues. Switching a call server out of service is done using a function on our control pages which has been tested many times in the past. Unfortuntely it pushes out the zone files for one of the domains as part of the process as it adjusts SRV records for the call servers. This is, again, normally quite safe. What has caught us out is that somehow the zone database was broken, missing some key records, so when it was pushed out the call servers vanished on IPv4. This caused yet more confusion as all of our test calls worked as they come from IPv6 equipment. This took several minutes to pin down and fix. We're now checking archives to try and find when and how the DNS records vanished. We're still investigating why the one call server is playing up, and hope to put it back in service later this evening. |
| Started | 9 Jul 15:20:00 |
| Closed | 9 Jul 15:50:00 |
| Details 4 Jul 11:00:06 |
Just to update - we have the physical SIM cards now, and we have pricing agreed. They are not yet provisioned on the network and that will hopefully be start of next week at which point we'll be able to start selling them. Thank you all for your patience. |
| Started | 4 Jul |
| Previously expected | 8 Jul |
| Details 1 Jul 16:31:19 |
SIP2SIM currently appears not to be working. We are investigating and will post an update as soon as possible. |
| Update
1 Jul 18:26:56 | Working again now. We'll post an explanation as soon as possible. |
| Update
1 Jul 18:44:53 | It was a fault in an upstream carrier's network. We don't have any further information at this point. |
| Started | 1 Jul 16:30:27 |
| Closed | 2 Jul 09:03:10 |
| Details 26 Jun 10:08:26 |
An after-the-fact status post, since the status pages were unreachable during the outage.
The authoritative DNS servers which, amongst other things, serve aa.net.uk and reverse DNS zones were unable to serve records due to a database problem from about 09:10 to 10:00 this morning. This affected reverse DNS records and services under aa.net.uk, including email, voip and the clueless control panel. Customer hosted zones were not affected by this outage. A fuller post-mortem will be posted here in due time. |
| Update
26 Jun 10:20:20 | The current status as of 10:15 is that we have moved the delegation from Nominet for aa.net.uk from the affected DNS servers to alternative servers as of 09:40. Nominet's servers are serving these new records but resolvers will have cached the old records and will take some time to update.
Doing this has reduced the load on the affected servers, and has allowed them to begin working again, so all DNS services should now be working again. There may be ongoing issues with customer facing email access which uses hostnames of the format yourdomain.com.mail.aa.net.uk to reach our mail servers. This is being worked on. |
| Update
26 Jun 10:54:52 | We believe we have fixed the problem with serving records for yourdomain.com.mail.aa.net.uk style zones, and have fixed the delegation for mail.aa.net.uk to go back to the previous servers which understand how to serve these zones.
This again will take a short while for DNS updates to propagate, but should mean that all services are back to normal now. |
| Started | 26 Jun 09:09:00 |
| Closed | 26 Jun 10:48:19 |
| Details 1 Jul 20:50:32 |
We have identified some TalkTalk back haul lines with congestion starting around 16:20 and now 100ms with 2% loss. This affects around 3% of our TT lines.
We have techies in TalkTalk on the case and hope to have it resolved soon. |
| Update
1 Jul 20:56:19 | "On call engineers are being scrambled now - we have an issue in the wider Oxford area and you should see an incident coming through shortly." |
| Resolution | Engineers fixed the issue last night. |
| Started | 1 Jul 16:20:00 |
| Closed | 1 Jul 23:25:00 |
| Previously expected | 2 Jul |
| Details 1 Jul 17:44:07 |
We have been looking again in to CDRs. We have some new logic that should show correct I/C logs for calls to your number. The exception is where that is done as part of an also ring from another number that was called. It should also show calls to another A&A number (as free) with the exception of where this is an also ring from another number. I.e. the intermediate legs are not longer showing (they were not showing properly before). We believe the charging is correct.
As a separate issue, customers billed for use of BT lines we provide (which is only a few customers that have outgoing calls) have seen a lack of call bills for a few weeks. This should catch up (hopefully today) with any previously unbilled calls being billed. |
| Started | 1 Jul 17:30:00 |
| Details 30 Jun 17:14:37 |
The CDR timestamps for SIP2SIM were in UTC. As we do normal call CDRs in local time, this was somewhat confusing.
We have now changed CDRs for SIP2SIM in local UK time. This will affect new call records for calls/text/data on SIP2SIM from now. |
| Started | 30 Jun 17:10:00 |
| Details 28 Jun 09:25:33 |
We hope to have our first roaming SIMs late next week. These are the same price to get as the existing UK O2 only SIMs (£5 to buy, £2/month).
They have a dual personality in that they can be O2 to work in the UK on O2 at the same prices as the O2 only SIMs (2p/min, 2p/text, 2p/MB), or can switch to an EU profile which has the same price in UK and throughout the EU at 10p/min, 5p/text, 10p/MB. The EU profile allows use on other UK networks so allowing you much more coverage than a normal UK only SIM. These SIMs do allow roaming outside EU at a higher price, which we'll confirm in due course. For a limited period, we are planning to offer these free (pay for postage only) to customers that need roaming and have got one of our O2 only SIMs. Once we have the SIMs in stock, we'll work out the details. |
| Started | 28 Jun |
| Details 27 Jun 10:08:52 |
It looks like we have not been recording logs of incoming calls that have been answered in the CDRs on our VoIP service. These have no call cost implications obviously. This has now been fixed and the logs should now show such calls. Sorry for any concern this may have caused. |
| Started | 23 Jun 15:00:00 |
| Closed | 27 Jun 10:07:44 |
| Details 25 Jun 05:39:05 |
The minor change last night is still having some issues with some servers being slow, and this is currently impacting VoIP. This is still being working on. |
| Update
25 Jun 06:07:24 | VoIP services starting to work properly again, still not 100% right. |
| Update
25 Jun 06:15:04 | VoIP looking a lot healthier now. Still working on tracking down the poor performance issues. |
| Update
25 Jun 08:35:02 | For more details on what happened, please see https://www.facebook.com/AAISP/posts/673421826086051 |
| Resolution | We'll be monitoring this closely during the day. |
| Started | 25 Jun 05:38:24 |
| Closed | 25 Jun 06:28:18 |
| Details 24 Jun 18:22:57 |
We are making a very minor change to database formats, but this has resulted in some unexepected impact on some servers using lots of CPU to effect the change. The upshot is some DNS lookups are stalling causing some disruption especially to access to our servers.
The tests we did on the backup/test system did not show this. However the work is almost over and will be back to normal shortly. |
| Update
24 Jun 19:22:28 | The database update really did take a surprisingly long time to complete, but seems to have finished on the clueless now. The slave servers are completing the updates and we took one of the DNS servers that was being sluggish out of replication until this is finished. Sorry for any inconvenience. |
| Update
24 Jun 20:11:33 | One of the commands in the update has replicated and is taking over an hour on some of the servers. This is not normally an issue, but it looks like the older "C" call server may have problems with registered SIP connections as this has now exceeded the normal SIP registration time. The new call server is not reliant on the central database in the same way so should be fine.
We don't expect the command to take much longer to complete now, but if you need a work around, register on the new call server (voiceless) and that should be fine. |
| Resolution | The servers have caught up. |
| Started | 24 Jun 18:00:00 |
| Closed | 24 Jun 20:37:53 |
| Previously expected | 24 Jun 19:00:00 |
| Details 23 Jun 10:29:01 |
We have added some quite sensitive checks to try and detect VoIP attacks, i.e. where someone has compromised your call server or obtained your SIP details and is using them.
These types of attacks can cause rather large unexpected bills. They are quite rare for us as we have a default low call rate allowed on international calls, but some customers with these rates set higher have had some attacks lately. The new measures should not cause any problems, but we are monitoring the situation closely. If your account is locked down you will get a recorded message when trying to make calls, and an email. You can unlock the account from our control pages without the need to contact us first. Obviously, if you do get locked down and a warning, please do check the call records to confirm this is not an attack. We are able to adjust the thresholds we have set if we find it is too sensitive. As these new features are only on the new call server (voiceless) we recommend people move to the new server. Once you have moved to the new server you will not be able to make calls via the old call servers so that the new security measures cannot be bypassed. |
| Started | 23 Jun 10:00:00 |
| Details 11 Mar 10:11:55 |
We are seeing multiple exchanges with packet loss over BT wholesale. We are chasing BT on this and will update as and when we have updates. GOODMAYES CANONBURY HAINAULT SOUTHWARK LOUGHTON HARLOW NINE ELMS UPPER HOLLOWAY ABERDEEN DENBURN HAMPTON INGREBOURNE COVENTRY 21CN-BRAS-RED6-SF |
| Update
14 Mar 12:49:28 | This has now been escalated to the next level for further investigation. |
| Update
17 Mar 15:42:38 | BT are now raising faults on each Individual exchange. |
| Update
21 Mar 10:19:24 | Below are the exchanges/RAS which has been fixed by capacity upgrades. We are hoping for the remanding four exchanges to be fixed in the next few days.
HAINAULT SOUTHWARK LOUGHTON HARLOW ABERDEEN DENBURN HAMPTON INGREBOURNE GOODMAYERS RAS 21CN-BRAS-RED6-SF |
| Update
21 Mar 15:52:45 | COVENTRY should be resolved later this evening when a new link is installed between Nottingham and Derby. CANONBURY is waiting for CVLAN moves that begin 19/03/2014 and will be competed 01/04/2014. |
| Update
25 Mar 10:09:23 | CANONBURY - Planned Engineering works have taken place on 19.3.14, and there are three more planned 25.3.14 , 26.3.14 and 1.4.14.
COVENTRY - Is now fixed NINE ELMS and UPPER HOLLOWAY- Still suffering from packet loss and BT are investigating further. |
| Update
2 Apr 15:27:11 | BT are still investigating congestion on Canonbury, Nine Elms and Upper Holloway. |
| Update
23 Apr 11:45:44 | CANONBURY - further PEW's on 7th and 8th May
NINE ELMS - A total of 384 EU’s have been migrated. A further 614 are planned to be migrated in the early hours of the 25/04/14. UPPER HOLLOWAY - Planned Engineering Work on 28th April BEULAH HILL and TEWKESBURY - Seeing congestion peak times and Chasing BT on this also. |
| Update
30 Apr 12:51:24 | NINE ELMS - T11003 - Still ongoing investigations for nine elms.
UPPER HOLLOWAY - T11004 - BT are working on this and a resolution should be available soon. TEWKESBURY - T11200 - This is on the Backhaul list and will be dealt with shortly. Work request closed as no investigation required. BT are working on this and a resolution should be available soon. MONMOUTH - T11182 - ALS583669 - This was balanced. I have advised BT that this is still not up to standards. They will continue to investigate. This is on the Backhaul Spreadhsheet also. So this is being investigate by capacity. BEULAH HILL - Being investigated. |
| Update
2 May 12:45:16 | CANONBURY - 580 EU's being migrated on 7th May and 359 EU's on 8th May
NINE ELMS - Emergency Pew PW238650 that will take place in the early hours on the 02/05/14. This is to move 500 circuits off 4 ISPV's onto other IPSV's. UPPER HOLLOWAY - Currently BT TSO have 12 projects scheduled for upper Holloway. TEWKESBURY - This is with BT TSO / Backhaul upgrades. MONMOUTH - This is with BT TSO / Backhaul upgrades. BEULAH HILL - Possibly fixed last night. Will monitor to see if any better this evening BAYSWATER - Packet loss identified and reported to BT |
| Update
6 May 11:44:59 | TEWKESBURY - Fixed
CANONBURY - EU's being migrated on 7th May and 359 EU's on 8th May Still seeing some lines with issues after the upgrade. Passed back to BT. NINE ELMS MONMOUTH UPPER HOLLOWAY BEULAH HILL READING EARLEY |
| Update
9 May 16:16:33 | CANONBURY - NINE ELMS- Now fixed
UPPER HOLLOWAY - Have asked the team dealing for the latest update. Email sent today 9/05/2014 MONMOUTH - BT TSO are still chasing this. BEULAH HILL - BT Tso chasing for a date on a PEW for work to be carried out. BAYSWATER - BT TSO are still chasing this READING EARLEY - Unbalanced LAG identified. Rebalancing will be completed out of hours. No ETA on this sorry. |
| Update
15 May 10:47:22 | UPPER HOLLOWAY - Now fixed
MONMOUTH - We have been advised that the target date for the capacity increase is the 22nd May. BEULAH HILL - Escalated this to a Duty Manager asking if he can gain an update. EARLEY - TSO advised Capacity team have replied and hope to get the new 10gig links into service this month. No further updates, so escalated to Duty Manager to try and ascertain a specific date in May 2014 when this will take place. |
| Update
21 May 09:32:00 | Reading Early / Monmouth - Now fixed
Bayswater - We have received a reply from the capacity management team, advising that to alleviate capacity issues, moves are taking place on May 23rd and May 28th. Beulah Hill - Due to issues with cabling this has been delayed , we are currently awaiting a date that the cables can be run so that the integration team can bring this into service |
| Update
2 Jun 15:15:55 | Bayswater - Now fixed
Belauh Hill - To alleviate capacity issues, moves are taking place between June 2nd and June 6th. |
| Update
10 Jun 12:16:52 | Belauh Hill - Now fixed
AYR - Seeing congestion on many lines, which has been reported. |
| Update
19 Jun 14:33:06 | AYR - Is now fixed |
| Broadband Users Affected | 1% |
| Started | 9 Mar 10:08:25 by AAISP Pro Active Monitoring Systems |
| Closed | 19 Jun 14:33:59 |
| Details 17 Jun 11:15:33 |
Our authoritative DNS servers are failing at the moment, this is being investigated. [Update 17:30 - a new problem has occurred, we are investigating. |
| Update
17 Jun 11:25:48 | This is affecting various services that we host. Our engineers are working on this and expect it to be fixed shortly. |
| Update
17 Jun 11:30:11 | The primary DNS server for our zones (aa.net.uk) is now back online. Services should be starting to work again. |
| Update
17 Jun 17:58:58 | The DNS problem has returned - similar to this morning. We are investigating at the moment. |
| Resolution | DNS servers are resolving correctly now. We do apologies for this problem. Our engineering team are investigating the cause and how we can prevent this from happening again. |
| Started | 17 Jun 11:10:00 |
| Closed | 17 Jun 17:58:58 |
| Details 11 Jun 15:12:53 |
It looks like one of our LNSs restarted. This will have affected a third of our broadband customers. Lines all reconnected straight away and customers should not see any further problems. The usage graphs from midnight until the restart will have been lost. |
| Broadband Users Affected | 33% |
| Started | 11 Jun 15:05:00 |
| Closed | 11 Jun 15:08:59 |
| Details 10 Jun 20:01:12 |
For customers on the current platform (voiceless) we are now sending emails when we see a new device (user agent) or IP address make use of your VoIP number credentials. This should help mitigate and detect fraud. You can change the setting to just check user agent (e.g. if using mobile or frequenctly changing IP) or disable it (though that is not recommended).
If you have an IP lock down, the warning is sent for any new IP/device that does not meet the IP lock down subnet but with a correct password, regardless of the warning settings. Any questions please ask. |
| Started | 10 Jun 20:00:00 |
| Details 8 Jun 18:53:59 |
Customers with UK domains registered through us will see a new option on the control pages to "Check .uk". This checks if you are entitled to the corresponding domain directly under .uk (i.e. a shorter domain). From Tuesday customers who are entitled to register the .uk domains can then click a button to copy their existing domain and register the new domain. These new domains and corresponding services are the same price as other normal UK domains.
In most cases the new domain is available if you have a domain ending .co.uk, but in a few cases customers with other second level domains are entitled if there was no .co.uk version. |
| Started | 8 Jun 18:00:00 |
| Previously expected | 10 Jun 08:00:00 |
| Details 6 Jun 18:33:22 |
Junk callers are getting smart and calling from "real" numbers. We don't have a very big list of these yet but we have added a new option on the call control pages to allow screening against known junk callers. The suppliments the existing anonymous call reject options that we offer. The plan is that the recording of a monologue with the junk caller will be emailed to the Information Commissioners Office automatically after each call (as per section 32 of the PECR). We are enabling this basic screening on all new numbers, so do log in and disable under the ACR section of you do not want this. |
| Started | 6 Jun 18:31:04 |
| Details 29 May 16:39:41 |
We've added a button to the Control Pages that enable Home::1 and Office::1 customers to purchase usage top up before the monthly quota has run out. More details on the wiki page: http://wiki.aa.org.uk/Home::1 |
| Started | 29 May 16:30:00 by AAISP Staff |
| Details 28 May 17:48:23 |
We're doing some routine router upgrades, and LNS upgrades over the next few days. This should have minimal impact on services. |
| Started | 28 May 17:00:00 |
| Closed | 30 Jun 13:49:06 |
| Previously expected | 2 Jun |
| Details 28 May 16:11:35 |
One of our interconnects in Maidenhead has planned work scheduled for Wednesday 4th June. The interconnect will be taken out of service for around 4 hours. During this time will will routing traffic over our backup link. Details: What is at risk: Connectivity to services hosted in Maidenhead. (Hosted servers, web, email, VoIP, control page access, Ethernet customers in Maidenhead). General internet connectivity for ADSL, FTTC and Ethernet customers in London will not be affected. What is the risk? The risk is that during this time access to some services will be affected. That said, we will have backup link in place which will minimise this risk. Time this is happening: 10pm Wednesday 4th June through to 6am Thursday 5th June, with downtime estimated of 4 hours. |
| Started | 28 May 15:51:03 by AAISP Staff |
| Closed | 4 Jun 11:00:00 |
| Previously expected | 4 Jun 22:00:00 (Last Estimated Resolution Time from Datacentre Staff) |
| Details 23 May 20:05:56 |
We are making a number of changes to the main page on clueless, and the search options for dealers/managers. This should be gradually applied over the weekend as work is done. The end result should be faster and more flexible. Any issues do ask RevK on irc. |
| Resolution | We have done the main work on this - changing over the search system completely. This has meant some small details having been removed which will be added back over the coming week depending on demand. |
| Started | 23 May 16:00:00 |
| Closed | 25 May 08:02:51 |
| Previously expected | 27 May |
| Details 17 May 12:02:10 |
We are running in to some problems calling some mobiles. This is be being worked on. |
| Update
17 May 15:00:27 | A work around was deployed on the current call server (voiceless) shortly after posting, and has now been deployed on the old call servers as well. |
| Started | 17 May 12:00:00 |
| Closed | 17 May 17:00:00 |
| Details 16 May 09:32:09 |
We will be upgrading our Roundcube webmail on Monday afternoon. This may mean that Roundcube is unavailable for short periods of time. We do have 2 webmail servers, and will endeavour to redirect customers to the working version as each are upgraded. |
| Update
16 May 09:40:23 | Further information about this upgrade: This update will upgrade Roundcube to the latest version (1.0.1). Customers will probably not notice many changes to how it works or looks though. New features include:
|
| Update
19 May 13:26:14 | Roundcube webmail has been updated. Do let staff know if you notice any problems. |
| Update
19 May 14:06:43 | We've add an extra new feature to Roundcube: "Global Address book". In the address book within Roundcube, you will see a 'Global Addresses' Group. This is a shared address book for everyone on your domain. All users in the domain can view, add and edit addresses to this group. |
| Started | 19 May 13:30:00 by AAISP Staff |
| Closed | 19 May 13:25:40 |
)
| Details 17 May 11:26:59 |
Our old call server (due to be deprecated soon) has had some sort of issue this morning, and is being rebooted to try and clear the problem. Some customers reported they were unable to get incoming calls. |
| Resolution | Restart completed quickly. |
| Closed | 17 May 12:01:25 |
| Details 16 May 11:21:56 |
The status page is once again posting incidents to Facebook and Twitter: https://www.facebook.com/AAISPStatus https://twitter.com/aaisp |
| Started | 16 May 11:18:55 |
| Details 15 May 12:31:00 |
For SIP2SIM we register to VoIP services. The contact we register is what is used for the server to make a call in to us to call the phone with the SIM.
The contact we used is of the format sip:[user]@[ip];line=[id] for example sip:fred@81.187.30.111;line=654478631245645613. This is a completely valid SIP URI and is the format used by many SIP phones. The line= part allows us to validate the incoming calls as from that registered carrier and is a very long random number. Sadly at least one carrier is incorrectly strippign the line= part of the URI. This is wrong, and means we cannot identify the call and so the call fails. We are making a work around for this. Obviously we should not have to, but we think we have come up with a way to do things that will work and be valid. We are changing the contacts to be of the form sip:[user]@[host] where the host part starts with a long number using a wildcard DNS record. We do not expect any problems with this, but if there are any, please let support know ASAP as we can easily change back. |
| Started | 15 May 12:00:00 |
| Closed | 15 May 22:16:30 |
| Previously expected | 15 May 18:00:00 |
| Details 14 May 12:56:02 |
Our SIP2SIM service registers with a VoIP servers, and makes calls via VoIP services.
This is done from our call servers, but as we have multiple call servers it is possible for the source IP (the call server being used) to change from time to time. It seems, however, that some VoIP providers are expecting to only get calls from the same IP address as the registration was made. This is odd as calls each way should be independent. However, because of this, we are working on changes over the next few days which would route calls from SIP2SIM to come from the same IP as the registration. The IP in use should remain fairy consistent, but may change from time to time. When this happens we will change calls to come from the same IP as registrations. |
| Resolution | Changes have been made - any issues, please contact support. |
| Started | 14 May |
| Closed | 14 May 18:46:05 |
| Previously expected | 18 May |
| Details 10 May 15:52:02 |
At 15:33 all 20CN lines on Kingston RASs dropped. We are chasing BT now. |
| Update
10 May 16:05:18 | BT have raised an incident. Apparently issue has been caused by power issues at London Kingston. |
| Update
12 May 08:55:29 | This was fixed after power was restored and a remote reset was performed. |
| Started | 10 May 15:50:27 by AAISP Staff |
| Closed | 12 May 08:55:06 |
| Cause | BT |
| Details 6 Apr 11:43:17 |
We are working on changes to the source IP addresses used for SIP messages we send where we have authentication details to provide. The reason is that asterisk boxes cannot easily be configured to tell "peers" from "users" in order to decide if a challenge is needed. This has caused problems for customers using VoIP unauthenticated from our IPs and SIP2SIM authenticated as a user. The first stage of the change has been done today, and involves the source IPv6 addresses. For all authenticated IPv6 messages (i.e. those for which we expect a challenge) we are sending from 2001:8b0:0:30::5060:a000/116 whilst any unauthenticated are from 2001:8b0:0:30::5060::/116 As this is within the 2001:8b0:0:30::5060::/112 block we have advised for VoIP use, this should not need any firewall changes or config changes, but we are interested in any feedback on any issues encountered. The change for IPv4 will take a bit longer - as we are unsure if to try and free up space in the 81.187.30.110-119 range previously advised, or to allocate a new range for this. This only affects calls originating from the new "voiceless" servers, which includes all SIP2SIM, but affects all authenticated messages whether SIP2SIM or otherwise. Please let us know of any issues. |
| Started | 6 Apr 11:00:00 |
| Details 14 Apr 20:10:20 |
Work on the billing system has resulted in interim data and call usage bills today. These will have normal payment terms, and obviously mean lower charges on your next bill. If any customers need extra time to pay because these are interim bills, please contact accounts who can extend the terms on this occasion. |
| Update
14 Apr 20:20:46 | We have extended the terms on these interim invoices by two weeks for those otherwise due this month. |
| Update
15 Apr 07:55:13 | For the technically minded - what we were attempting to do was sort an issue where a SIM gets suspended on the date of the last bill, and so the suspend date and the billed-to date are the same but some usage has not been billed for that day. We had customer complaints about this and the fact that up to a day's worth of calls and usage were not billed. The fix was a tad overzealous. We think we have this sorted now though. I do apologise for any concern this has caused. |
| Started | 14 Apr 20:09:03 |
| Previously expected | 15 Apr 00:09:03 |
| Details 10 May 13:18:53 |
A number of customers had asked us about recent news reports that ISPs will be sending educational letters to customers suspected of downloading media without appropriate permission from the copyright holder.
Please be assured that AAISP are under no obligation to send such letters, any more than the power companies that power and charge the devices used for such activities, or the device manufacturers. We have no intention of sending such letters. As always, if we receive an abuse report it will either go directly to the customer as the contact details on the whois for IP addresses, or come to us and we will simply pass it on (as well as pointing out the sender that we are neither the police or the civil courts). I hope this clears up any misunderstandings. |
| Started | 10 May |
| Details 9 May 10:13:56 |
The carrier are aware of an issue and working on it. |
| Resolution | Seems to be working now. |
| Started | 9 May 08:50:00 |
| Closed | 9 May 10:25:42 |