Server and System Status

NC020: Web server restarted

10 November 2014 13:24:48 +0000

The web server on NC020 was down between approximately 11:21 and 12:33 UTC today. We are investigating and will post a report here later today.

Tags: down time, web server
Categories: Incidents, NC020
Comments Off

POODLE maintenance: Complete

23 October 2014 22:54:04 +0000

We have completed the maintenance to mitigate against exploitation of the POODLE vulnerability.

Because the effects of this maintenance could potentially be wide-ranging, please contact support if you have any issues with connecting over a secure connection to any of our servers. We don’t believe there will be any issues with web (HTTPS) connections, but there may be with some email (SMTP, IMAP and POP) connections by older email programs. That said, an exploit against this vulnerability has only been demonstrated using JavaScript and a web browser, so email connections (even if using SSLv3) are not generally considered to be at risk.

Thanks for your patience during this maintenance.

UPDATE, 2014-10-24: Please read some details on this maintenance, and learn more about SSL and TLS and email security, on our corporate blog.

Tags: ssl, web server
Categories: Maintenance, NC018, NC020, NC023, NC026, NC027
1 Comment »

POODLE maintenance: Update 1

19 October 2014 00:33:28 +0000

The maintenance described in our last post is under way, but we are running into some issues that are delaying completion. While there may be sudden and unexpected interruptions in some services for some clients, we are aware of them and are working on them. If necessary, we will revert to old configurations while we re-evaluate the new configurations recommended to defend clients against the POODLE vulnerability.

Thank-you for your patience.

Tags: ssl, web server
Categories: Maintenance, NC018, NC020, NC023, NC026, NC027
Comments Off

SSL version 3 “POODLE” vulnerability, system-wide maintenance

17 October 2014 05:25:59 +0000

We will be conducting maintenance this weekend to address the above noted vulnerability. Please read more details about this on our corporate blog.

Tags: ssl, web server
Categories: Maintenance, NC018, NC020, NC023, NC026, NC027
Comments Off

NC027 POP email issue

12 October 2014 20:02:25 +0000

We have determined that the issue with downloading email from NC027 last week only affected POP email accounts, and was likely related to the limit on POP3 connections configured on the server. We have raised that limit and don’t expect this to be an issue in the future.

This did not affect those who were connecting via IMAP. Our apologies to those who were affected.

Tags: connection issues
Categories: Email, Incidents, NC027
Comments Off

NC026 emergency maintenance complete

2 October 2014 12:14:12 +0000

The emergency maintenance on server NC026 described earlier has been completed successfully. Thank-you for your patience.

Tags: down time, hardware, reboot, web server
Categories: Maintenance, NC026
Comments Off

NC027 email issue update

2 October 2014 11:32:55 +0000

We have rebooted server NC027 and this seems to be at least a temporary solution to the problem. We’re still looking into the cause so that we can actually fix the problem.

Tags: connection issues, reboot
Categories: Email, Incidents, NC027
Comments Off

Possible email issues on NC027

2 October 2014 11:00:17 +0000

We are getting reports of issues with downloading — but not sending — email from server NC027. We’re having trouble nailing down what exactly the problem is, so you can help by sending us the exact text of any error messages your email program is giving you if you use this server.

From what we’ve seen so far, it seems that this is related to using POP and the problem in intermittent. So far we haven’t heard from IMAP users that they’re having problems.

Please contact support if you’re experiencing this problem, and please be sure to include the actual text of whatever error messages you see in your email program. The answers to the following questions would also be helpful:

What email program are you using?
Are you using POP or IMAP?
Are you using a secure (TLS/SSL) or an insecure connection to the mail server?
What is the exact text of any error messages you are seeing?

Thanks very much for your time, and we apologise if this is causing a problem for you.

Tags: connection issues
Categories: Email, Incidents, NC027
Comments Off

NC026 emergency maintenance

1 October 2014 04:40:48 +0000

To deal with an as-yet unpublished security vulnerability, server NC026 will undergo emergency maintenance today (1 October) starting shortly after 15:00 UTC. Down time should be brief, but will be noticeable for websites hosted on this server during the maintenance. Most of the websites hosted on this server are newer ones using more recent versions of WordPress. If you are not sure whether or not your website is hosted on this server, you can contact support to find out.

No email is hosted on NC026, so this maintenance will not affect email flow at all.

Tags: down time, hardware, reboot, web server
Categories: Maintenance, NC026
Comments Off

NC020 outage report

25 August 2014 00:05:53 +0000

The problem with server NC020 actually started before the outage on Friday, about midday UTC. A client’s website was compromised, and the cracker downloaded scripts to cause the sending of spam. This took place over the course of a day or so, and eventually the spammer consumed all of the resources of the server. Although the server was still up, it was unresponsive.

Most websites are authorised to use the mail server on a hosting server, so we can’t block that ability completely. However, we will make a concerted effort to improve the monitoring of abuse (intended by the client or not) of this function.

In this case, we cleaned up all of the generated spam, removed the offending scripts, and will work with the client on the issues with their website security.

For the record, there are two levels of compromises: at the root level or the user level. At the root level is very bad, and requires a brand new server to be provisioned. This was a user-level compromise which, while bad, is easier to fix and recover from.

We again apologise for the downtime. We will be contacting and crediting affected accounts. We are always working to prevent security issues like this, but it’s an ongoing task. Thank-you for your patience and continued business. If you have any questions, please feel free to contact support.

Tags: down time, web server
Categories: Incidents, NC020
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?