From the datacentre:
As part of Pulsants planned maintenance regime, we have been informed by the power company that they need to undertake a inspection of the infrastructure that delivers power on site, in alignment with these works, we have decided that we will align these works with our annual darken building test to ensure all infrastructure works correctly in the event of a full loss of mains power and to minimise further interruptions.
WHY: The Annual Darken Building Test and transformer inspections forms part of our stringent maintenance regime and this is to ensure all infrastructure works at its optimum but also to ensure it correctly operates in the event of a full loss of mains power to site. To do this we will be working with our 3rd party contractors and the electricity company to isolate the mains power to the site and perform essential testing of our HV and LV infrastructure and then to carry out the transformer inspection work. During this period, it will be necessary to run the site on generators. Pulsant will follow our Darken Building Test procedure, which will isolate the incoming utility supply to site The testing will be completed by Pulsant staff and a number of specialised 3rd party contractors.
The following works will be carried out; -Monitoring will continue for a further 40 minutes post restoration of utility power to ensure there are no outstanding issues.
All diesel supplies to the site have been checked and confirmed adequate supplies, reserve supplies are on standby. Once works have been completed, the fuel supplies will be replenished accordingly. We have provisioned a maintenance window of up to 10 hours to allow us an adequate time window to complete all the necessary inspections, tests and servicing. All personnel will remain on-site during this maintenance window and for a period of up to 1-hour post the maintenance being completed window. Works are expected to be non-service affecting as customers will remain on UPS protected supplies throughout until all works have been completed. This maintenance does involve changes to power source to the site, hence an At-Risk notification is being sent out.
Pulsant has been preparing extensively for this scheduled Darken Building Test and worked with the associated external parties to align these works.
We will have a number of different contractors and Pulsant personnel on site including expert contractors to support our Generators, UPS devices, electrical infrastructure and most importantly Pulsant network and site support staff. We have worked with the electricity utility supplier to mitigate risk as far as possible, below is the planned approach with regards to completing all necessary works and undertaking this Annual Darken Building Test
SERVICE IMPACT: NONE - No interruption or disruption to service is expected during these essential works, however due to the level of switching that will be undertaken during executing our Darken Building Test procedure and isolation of power so that the mains incomer and transformer can undergo their inspection, we are issuing a courtesy At Risk notification to all our customers.
WHEN: Thursday 5th March 2020, commencing at 08.00 GMT and completed by Thursday 5th March 2020 by 17.00 GMT. To mitigate risk and impact, Pulsant have scheduled for these works to be carried out during the working day. The At Risk window will be kept to an absolute minimum and the site will have utility mains power restored and resuming full site protection at the earliest opportunity.
This was quite a difficult problem to diagnose and we apologise to those affected. The problem affected, intermittently, about a quarter of phones that register to our VoIP service. Customers who we send calls to your own server or we redirect to other numbers were not affected.
The problem was caused by a misconfigured or buggy phone which hit an edge-case bug in our code. This caused registration updates of that phone to fail. This only affected one of four back-end servers, but it did mean that further registration attempts from other phones to this back-end server also failed. A short term fix was applied and registration details were then being updated successfully.
We do have already lots of monitoring and logging from our VoIP platform but this fault has shown that we can do better and we will be working to improve this so as to help diagnose faults quicker in future.