Some customers have been experiencing irregular phone call behaviour - such as when incoming calls are set to ring multiple phones, some phones carry on ringing after the call has been successfully answered elsewhere.
We have been investigating the problem, and have planned a software upgrade on some of our back-end servers for 6AM Thursday morning which will then enable some further configuration changes thereafter to improve the situation.
Our early morning work is sell under way, there will be a brief disruption to services at around 06:40-07:00 which will cause new calls to not connect for few short moments.
The disruption was short lived and didn't affect calls that were in progress.
We've had some further problems this afternoons with some call and registration failures. This is related to our work to improve our back-end systems. This is being worked on.
We are still seeing call setup or registration problems and are working to resolve this. More information to follow shortly.
Some background information: Our VoIP system has a cluster of back-end servers which manage SIP messaging which involves tracking registered phones, call routing information, and so on. We had needed to improve the performance of this system as we were seeing occasional timeouts. There were a few steps required to carry out this work. We needed to upgrade the OS of our servers (Completed on Thursday morning), and then change our code to run in a new, more efficient, way. We also need run our code in a 'migration' mode which means running both the old method and new method at the same time, so that the new system can have its databases populated and indexed. The plan was to run that for a period of time and we started this earlier this morning. Even though this procedure was tested earlier in the week, today's work hasn't quite gone to plan, and we're currently experiencing an increase of timeouts which is affecting the service. This is being investigated.
We are putting in place a short term change which should help reduce the load, we are also reviewing the cause of the timeouts as there may be a separate root cause of the timeouts.
Our efforts to reduce load have not completely helped, we have still seen intermittent call problems this afternoon. We are working on other avenues to tackle this.
We have code changes prepared that we'll apply early Friday morning.
New code is being installed now.
The service has been running well this morning.
Some call and registration failures have been the result of slow back-end systems. We have made various optimisations and code changes which have been working well throughout the day (Friday). We also have avenues for further improvements which we be implemented in the near future. We do apologise to those who had calls disrupted due to this.