Summary We have received confirmation from CGI, that following patching activity, Wintel have advised they are not able to RDP or iLO on to the NECTMG01 server. This will mean that external websites for TTB such as billing.mytalktalkbusiness.co.uk and www.talktalkbusiness.co.uk are unavailable for business customers. Â N/A
Latest Update A technical bridge is ongoing with TT MIM, Online Support, CGI MIM & CGI Wintel. A ticket has been raised with SCC (7540687) to get Intelligent Hands support to reboot the server. SCC are advising that the server serial number is not on their system. CGI ITSD are currently trying to approve the potential charges for this. It has been requested that Network support be engaged to try and see if NECTMG02 the secondary serer can be used as the switchover did not work. MIM are currently arranging this. We are escalating to Digital Ops management as we are currently unable to get in contact with the on call engineer. Â
Latest Update SCC (7540687) have advised that they will be able to get to site (NEC) to complete an investigation/reboot of the server by 23:30hrs. Network support have joined the call and are working with CGI to get the secondary server (NECTMG02) to force it over to restore service while SCC get to site to investigate the server currently down NECTMG01. Â
Latest Update It was identified on the call that following patching activity (CRQ000000036641) the server at NEC (NECTMG01) was restarted however experienced a hard drive failure and did not come back up. Service did not automatically switch over to the secondary server NECTMG02. With the assistance of our Network Support team and CGI Technical Service Delivery team we have been able to bring up the secondary server to restore service since approximately 22:50hrs. SCC (Ticket ref 7540687) have advised that they will be able to get to site (NEC) at approximately 23:30hrs to complete an investigation/replacement of the failed HDD and bring the primary server back up. CGI MIM and Wintel teams will manage and monitor this work. Once completed and back up the services will be shared between the 2 servers as they are load balanced. TT MIM will continue to liaise with CGI and confirm once both servers are back up. Â
Technical / Suspected Root Cause This incident was raised due to being identified that External websites for TTB such as billing.mytalktalkbusiness.co.uk and www.talktalkbusiness.co.uk were unavailable for business customers (28/09). It was identified on the bridge call that following patching activity (CRQ000000036641) the server at NEC (NECTMG01) was restarted however experienced a hard drive failure and did not come back up. Service did not automatically switch over to the secondary server NECTMG02. With the assistance of our Network Support team and CGI Technical Service Delivery team we were able to bring up the secondary server to restore service since approximately 22:50hrs. SCC (Ticket ref 7540687) advised that they will be able to get to site (NEC) at approximately 23:30hrs to replace the HD and bring the primary server back up. CGI Wintel confirmed that SCC has successfully restored service to the TMG01 server & Wintel confirmed that heath checks on the server were successful. As no further issues have been identified we can ow confirm closure to this incident with further RCA being completed by our Problem Management team. Â