Server and System Status

NC027 email issue update

2 October 2014 11:32:55 +0000

We have rebooted server NC027 and this seems to be at least a temporary solution to the problem. We’re still looking into the cause so that we can actually fix the problem.

Tags: connection issues, reboot
Categories: Email, Incidents, NC027
Comments Off

Possible email issues on NC027

2 October 2014 11:00:17 +0000

We are getting reports of issues with downloading — but not sending — email from server NC027. We’re having trouble nailing down what exactly the problem is, so you can help by sending us the exact text of any error messages your email program is giving you if you use this server.

From what we’ve seen so far, it seems that this is related to using POP and the problem in intermittent. So far we haven’t heard from IMAP users that they’re having problems.

Please contact support if you’re experiencing this problem, and please be sure to include the actual text of whatever error messages you see in your email program. The answers to the following questions would also be helpful:

What email program are you using?
Are you using POP or IMAP?
Are you using a secure (TLS/SSL) or an insecure connection to the mail server?
What is the exact text of any error messages you are seeing?

Thanks very much for your time, and we apologise if this is causing a problem for you.

Tags: connection issues
Categories: Email, Incidents, NC027
Comments Off

NC020 outage report

25 August 2014 00:05:53 +0000

The problem with server NC020 actually started before the outage on Friday, about midday UTC. A client’s website was compromised, and the cracker downloaded scripts to cause the sending of spam. This took place over the course of a day or so, and eventually the spammer consumed all of the resources of the server. Although the server was still up, it was unresponsive.

Most websites are authorised to use the mail server on a hosting server, so we can’t block that ability completely. However, we will make a concerted effort to improve the monitoring of abuse (intended by the client or not) of this function.

In this case, we cleaned up all of the generated spam, removed the offending scripts, and will work with the client on the issues with their website security.

For the record, there are two levels of compromises: at the root level or the user level. At the root level is very bad, and requires a brand new server to be provisioned. This was a user-level compromise which, while bad, is easier to fix and recover from.

We again apologise for the downtime. We will be contacting and crediting affected accounts. We are always working to prevent security issues like this, but it’s an ongoing task. Thank-you for your patience and continued business. If you have any questions, please feel free to contact support.

Tags: down time, web server
Categories: Incidents, NC020
Comments Off

NC020 back online

23 August 2014 10:22:32 +0000

Server NC020 experienced a significant outage over Friday and Saturday. The problem has been resolved and the server and all websites on it are back online.

We sincerely apologise for the problems caused by this outage to some website owners. We will provide a full report on what happened, when, and why by 01:00 UTC on Monday, 25 August.

Tags: down time
Categories: Incidents, NC020
Comments Off

NC018 web server up

14 August 2014 18:41:09 +0000

An automated alert was issued at 14:04 UTC today that the web server on NC018 was down. Unfortunately no immediate action was taken by the data centre where the server is located, and the server was not restarted until 17:03 UTC.

We apologise for this significant delay. We have taken this issue up with the data centre to determine the cause of the delay and to ensure that it doesn’t happen again.

Other services on the server — email, nameservers, FTP and databases — were not affected by the outage.

Tags: down time, web server
Categories: Incidents, NC018
Comments Off

NC018: Web server restarted

19 June 2014 07:08:10 +0000

We have restarted the web server and are keeping an eye on it, specifically the sources of traffic to it.

Tags: dos attack, down time, performance, web server
Categories: Incidents, NC018
Comments Off

NC018: Web server down

19 June 2014 06:57:04 +0000

We have taken down the web server while we investigate an overload condition. We will have it back up as soon as possible, and will post further updates here.

Email and other services on server NC018 are not affected.

Tags: dos attack, down time, performance, web server
Categories: Incidents, NC018
Comments Off

NC018: Issue resolved

5 May 2014 08:15:06 +0000

Web server performance on server NC018 has returned to normal. Thank-you for your patience.

Tags: performance, web server
Categories: Incidents, NC018
Comments Off

NC018: Web server slow

5 May 2014 07:45:40 +0000

Some websites on server NC018 are loading quite slowly. This does not seem to be affecting other services such as email.

We are looking into the cause and will report back here.

Tags: performance, web server
Categories: Incidents, NC018
Comments Off

NC027: SSL case closed

24 March 2014 09:10:02 +0000

We are closing this issue, although we have not found a cause. As far as we can tell, everything related to authentication on the server is working. Research on similar occurrences also shows the problem apparently resolving itself after a short time, and also points to a number of possible causes, including faulty operating system and software updates on client machines. However, in this case we saw the problem on different operating systems (Windows, Mac and Android) and different email programs. We also suspected anti-virus software, but again we heard of problems on computers running different anti-virus software.

Oddly, we could not reproduce the problem on any machines in our office, and so we had no primary references on which we could do research.

About the only commonality we could find — other than the server itself, of course — is that we believe all of the reporting clients are on VSATs. However, even that doesn’t fully explain the problem as if, for some reason, VSATs were the issue we should theoretically have seen the same problem on other servers. That said, the problem reported to us was only with downloading email, and the other mail server on which the bulk of our clients are hosted uses slightly different SSL technology; additionally our default instructions for that server do not mention using SSL. We saw a similar issue with VSAT connections several years ago, so if this is related it’s not unprecedented.

Another possibility that can’t be ruled out is network issues that may have interfered with encryption on the connection. Given the number of reports, such an issue (if it existed) would likely have been closer to the server than the client. If this issue reoccurs we will contact our upstream providers to see what they have to say.

While it is unsatisfactory to close an issue without having a smoking gun to identify it, we are going to do so at this time. If you do see this issue again, please contact support and we will again look at the issue.

Thank-you for your patience while we looked at this issue.

Tags: ssl
Categories: Email, Incidents, NC027
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?