We are making a very minor change to database formats, but this has resulted in some unexepected impact on some servers using lots of CPU to effect the change. The upshot is some DNS lookups are stalling causing some disruption especially to access to our servers. The tests we did on the backup/test system did not show this. However the work is almost over and will be back to normal shortly.
The database update really did take a surprisingly long time to complete, but seems to have finished on the clueless now. The slave servers are completing the updates and we took one of the DNS servers that was being sluggish out of replication until this is finished. Sorry for any inconvenience.
One of the commands in the update has replicated and is taking over an hour on some of the servers. This is not normally an issue, but it looks like the older "C" call server may have problems with registered SIP connections as this has now exceeded the normal SIP registration time. The new call server is not reliant on the central database in the same way so should be fine. We don't expect the command to take much longer to complete now, but if you need a work around, register on the new call server (voiceless) and that should be fine.
The servers have caught up.