RESOLUTION

Technical / Suspected Root Cause It had been reported that users were experiencing application issues with the following systems - Portal, Consumer MyAccount, Sales Portal. When attempting to access Portal, a runtime error message is being received - `Description: An exception occurred while processing your request. Additionally, another exception occurred while executing the custom error page for the first exception. The request has been terminated.` This impacted TT agents and Partners ability to conduct Fulfilment and Assurance activities and also partners ability to raise Service Requests into our support teams. A bridge call was held with relevant teams, where it was confirmed that API calls were also seeing authentication failures. After investigations it had been identified that there was an issue with the session state server SESSSTATE1IR2. CGI initially carried out an IIS restart however the issue was still occurring. CGI Wintel team had then been engaged to investigate further as the server was not reachable. In parallel, we were advised that there was an issue with the REDIS server which was causing issues to MyAccount and Sales Portal, this was not related to the Portal and API calls issue on server SESSSTATE1IR2. The holding pages for both were placed and ISP teams restarted the REDIS server to restore service to both MyAccount and Sales Account. Sanity checks carried out from technical teams and business had confirmed that they could both be accessed as normal. CGI Wintel team had confirmed from investigations that the server SESSSTATE1IR2 was reporting memory resource issues and required a full reboot, a reboot of server SESSSTATE1IR2 was then carried out to restore service. Sanity checks were carried out by technical teams which then confirmed Portal was available and API checks were not failing. Confirmation has also been received from the business that all impacted applications were available and accessible. Root cause investigations with server SESSSTATE1IR2 will be carried out via the Problem Management process, in parallel we have been advised that new replacement REDIS servers are to be added as an upgrade. A CRQ for this will be raised and also tracked via the Problem Management process.