Summary
It has been observed by the TTB fulfilment team that since the deployment of a recent change (CRQ22103) on 26/6/19 new notifications are being produced for IPVPN and EA memberships that are not required. The result from the increase of notifications is causing latency issues when trying to access and navigate NSOM
Â
N/A
Latest Update Bug 447597 has been created by the Pega ops team and a fix developed by our OSS delivery team. Delivery will now need to test then deploy the fix out of hours on a new CRQ, we will provide details of this change as soon as the information is available Â
Latest Update OSS Delivery have advised the testing of the fix needs to take place in production meaning it will need to commence out of hours. Once testing has completed successfully a CRQ will then be raised and assigned to our change team for approval then our release team for deployment. We are currently liaising with delivery to determine when the testing will commence Â
Latest Update OSS Delivery have advised the testing of the fix needs to take place in production meaning it will need to commence out of hours. Once testing has completed successfully a CRQ will then be raised and assigned to our change team for approval then our release team for deployment. We are currently liaising with delivery to determine when the testing will commence Â
Latest Update It has been confirmed by OSS delivery that CRQ000000027963 has now been raised and is currently having the IA (impact assessment) completed before passed to the change team. We will provide a further update as soon as more information is available Â
Latest Update Technical teams have revisited the solution and have confirmed that amendments to the fix have been made and has been tested successfully. A GNG took place at 16:00 between relevant teams where approval for the new solution to go ahead was agreed by all. Confirmation has been received that the change has been implemented and sanity checks have been carried out to ensure the solution has been completed successfully. In parallel, TTB have been engaged to carry out post checks who have confirmed they have conducted a number of tests and there no latency issues have currently be seen. A further check point call is set up tomorrow morning to check if any issues have been seen BAU when a number of users are logged in, we will also need to confirm that following the deployment we arenât receiving notifications for EAD & EFM orders that are not expected. It has been advised that the back log will be cleared by CGI and could will take approx. 10-15 days. TTB have advised they will also provide any support if required to assist with this. We have also requested the notification queue stats daily so we can continue to monitor the progress. Â
Technical / Suspected Root Cause It had been observed by the TTB fulfilment team that since the deployment of a recent change (CRQ22103) on 26/6/19 new notifications were being produced for IPVPN and EA memberships that were not required. The result from the increase of notifications was causing huge latency issues when trying to access and navigate the NSOM application making it inaccessible and service testing had started to fail on EoFTTC. Following a period of monitoring and observations it has been confirmed that following the implementation of the solution, NSOM continues to function BAU and no major issues have been reported which were logged as part of the incident. The backlog of notifications continues to be worked on by the CGI PEGA team who have confirmed as part of BAU process a clean-up activity will be taking place tomorrow at 07:00am to clear down a large number of notifications. TTB or any relevant teams will provide any support if required to assist with this. CGI PEGA have confirmed the current notifications will not cause any further latency issues to NSOM. We have requested that CGI PEGA continue to provide daily notification queue stats so we can continue to monitor the progress. The latency issue for which this incident was raised for is now resolved. Any further BAU actions to be tracked will now be followed via the Problem Management process. Â