RESOLUTION

VoIP has been a bit off for a couple of days with two unrelated incidents. Not good. It looks like in both cases many incoming calls were failing. Yesterday was an issue with the code that steers calls - we made a change done in an emergency for a customer (who was trying to abuse SIP and not have to pay for lots of licence fees on his VoIP switch). The change worked for him, but broke many other customers (not all, and not our office). The testing done before this was deployed did not pick up the issue as the test calls were the type that worked (like calls to our office). It was spotted and fixed very quickly. We're trying to work out how we can create more comprehensive tests in future - we want to be agile and responsive to customer needs, but we need any changes to be robust and not cause problems. Today was very different. We noticed an issue on one called server impacting some SIP2SIM customers, so took it out of service to investigate. We have multiple call servers, and switching the active servers is a routine task that can be done in cases just like this to ensure service continues. Switching a call server out of service is done using a function on our control pages which has been tested many times in the past. Unfortuntely it pushes out the zone files for one of the domains as part of the process as it adjusts SRV records for the call servers. This is, again, normally quite safe. What has caught us out is that somehow the zone database was broken, missing some key records, so when it was pushed out the call servers vanished on IPv4. This caused yet more confusion as all of our test calls worked as they come from IPv6 equipment. This took several minutes to pin down and fix. We're now checking archives to try and find when and how the DNS records vanished. We're still investigating why the one call server is playing up, and hope to put it back in service later this evening.