ZyXEL have published a security advisory which covers some of the ZyXEL routers we have provided our customers, the DX3301 and VMG3927.
New firmware has been released. Customers can either load this manually, or use the 'Upgrade Firmware' button found on the router information page on our Control Pages.
We have received further clarification that the vulnerabilities affect the router's web interface when exposed to the internet. Routers we configure for customers will have the web interface only allowable from our office IP range, and so this vulnerability is deemed low risk. We still recommend that customers upgrade. The upgrade process will take a few minutes but it does not reset the configuration.
We will make direct contact with customers with these routers, and support staff will mention this if and when they are talking to customers and notice the firmware is out of date.
Over the past week we have seen a huge number of 'bots' trying to guess customer email credentials in order to try to send email through our outbound email servers: smtp.aa.net.uk.
The attempts were being blocked due to wrong passwords being used, but this caused significant load on our severs due to all the database lookups involved. To address this, we are blocking IP addresses that are listed on the Spamhaus 'Exploits Blocklist (XBL)' and the Spamhaus 'Combined Spam Sources (CSS)' lists. -These are typically IP address known to have hijacked in some way or known spam senders.
This has reduced the load on the email servers significantly, however we are are still blocking around 1.5 million unique IP addresses each day.
We have had a small number of legitimate customers affected by this as their IP address is on these blocklists. (IPs can be looked up on https://check.spamhaus.org). In these cases, please do contact support and we can discuss workarounds.
Thank you for your patience with this - the problem would have affected SIP messages some of the time, and caused either a few seconds of delay in connecting a call or a call timing out and failing.
The problem was to do with round trip times between our datacentres. Late last year one of our layer 2 links between Maidenhead and London was cut off without warning - we believe due the supplier having financial problems and the datacentre unplugged them, completely. We have 2 links between London and Maidenhead so this wasn't disruptive but it lowered our resilience. We connected up a new Layer 2 link to replace the one that was cut off.
The replacement link was slightly higher latency. Our VoIP system uses backend SQL and Redis servers, and as part of a SIP registrations or SIP calls a number of SQL and Redis queries are made to gather the data needed. We had got to a tipping point where the quantity of lookups required to process SIP messages and the increase in the latency between sites built up to enough to cause SIP timeouts and retries (which generally happen at 500ms).
The fix we did today, was simply to make the master Redis server the one local to our VoIP servers, and therefore not needing to traverse between datacentres.
This has taken us a little while to figure out - our investigations initially were showing SQL and Redis queries themselves being processed quickly - it wasn't until deeper investigation in to the which nodes were passing the traffic between themselves did the inter-site latency we realised that along with the quantity of queries that are required that this was the cause of the latency. Our processing time for SIP messages is now back to a healthy < 50ms.