Server and System Status

Server NC018 web server restarted

24 August 2013 08:21:57 +0000

We have identified the particular websites that are being attacked and have made adjustments to try and deflect the attack. In the meantime we have restarted the web server and will continue to monitor the attack.

Additionally, the mail queue has now cleared.

Tags: connection issues, dos attack, email delivery, performance
Categories: Incidents, NC018
Comments Off

NC018 web server stopped

24 August 2013 07:24:12 +0000

We have temporarily stopped the NC018 web server in an attempt to fight this attack.

Tags: connection issues, dos attack, down time, performance
Categories: Incidents, NC018
Comments Off

Server NC018 under attack again

24 August 2013 07:08:51 +0000

As the title says, it appears that server NC018 is again the target of an attack. We are working to address the issue as quickly as possible.

Tags: connection issues, dos attack, down time, email delivery, performance
Categories: Incidents, NC018
Comments Off

Server NC018 denial of service attack

23 August 2013 20:55:10 +0000

Server NC018 suffered a denial of service attack beginning at approximately 07:40 on 23 August, UTC. This continued until the server succumbed under the load at about 12:40. This affected all functions of the server: websites, sending and receiving emails, databases, etc.

Unfortunately we did not respond to this situation in a timely manner. This was partially because automated monitoring of server functions had been turned off due to an increasing number of false alarms from the system.

Once the server was rebooted at 13:47 and the attack mitigated, the server then still had a backlog of several thousand emails to process. This took several hours, with the queue being cleared by about 18:35. While emails were “missing” for several hours, no emails should have been lost; they were just delivered several hours late.

We sincerely apologise for the disruption caused by this event. We will reinstate monitoring and the associated actions that are taken when monitoring alarms go off.

We will shortly be responding to all emails and phone messages regarding this incident. If you have any further questions about this event, please contact us.

Tags: connection issues, dos attack, down time, email delivery, performance, reboot
Categories: Incidents, NC018
Comments Off

Server NC018 issues

23 August 2013 15:54:34 +0000

Server NC018 appears to be suffering the effects of a denial of service attack. We are working to address the issue.

Tags: connection issues, dos attack, performance
Categories: Incidents, NC018
Comments Off

Server NC018 replacement: Update 4

20 April 2013 22:55:35 +0000

On Thursday the 18th we set up a new server in a new data centre. We’re excited about the innovations in security that this new configuration will bring to your hosting account.

We are working diligently to configure and test this server, and test the migration process to minimise the time it will take. Due to issues on the old server, our priority is to get the migration done as soon as possible. While we had hoped to do this over this weekend, we may need to do it during the week. Either way we will ensure that any disruption will be kept to a minimum.

We will post further updates here over the course of the next few days.

Tags: email delivery, hardware, performance
Categories: Email, Incidents, Maintenance, NC018, NC026
Comments Off

Server NC018 replacement: Update 3

21 March 2013 18:59:03 +0000

Contrary to previous information, the replacement server will not be ready until mid to late next week.

The good mews is that we have taken further steps to mitigate the performance issues, and we have seen no evidence of performance problems in the last 48 hours.

We’ll post another update here when we have firmer time lines to tell you about.

Tags: email delivery, hardware, performance
Categories: Email, Incidents, Maintenance, NC018
Comments Off

Server NC018 replacement: Update 2

20 March 2013 09:50:48 +0000

Server load today has been within tolerance, and email has not been backed up for more than about a couple of minutes at any point.

We’ll post an update later today on the acquisition of new hardware.

Tags: email delivery, hardware, performance
Categories: Email, Incidents, Maintenance, NC018
Comments Off

Server NC018 replacement: Update 1

19 March 2013 23:32:01 +0000

A contributory cause of the performance issues on server NC018 is one of the anti-spam systems on that server. Unfortunately, we have had to deactivate part of the anti-spam system, and we’re already seeing the results in the form of spam that would normally be blocked reaching mail boxes. We apologise for this temporary issue. On the plus side, the anti-spam systems on the new server will be an improvement over the one on the old server.

Tags: email delivery, spam
Categories: Email, Incidents, Maintenance, NC018
Comments Off

Update on server NC018

19 March 2013 12:58:28 +0000

As we informed you on 6 March, we had intended to increase the memory (RAM) on server NC018 on 9 March. (Server NC018 hosts the majority of websites, and all email except specialised email services.) This maintenance would have solved performance issues that were happening with more regularity. However, the maintenance failed due to hardware compatibility issues.

When the maintenance failed, we immediately began preparing to acquire new hardware to replace this server. During the week of 11-15 March we obtained a number of proposals and advice on a plan we had begun developing some months ago.

However, on 18 March the mail server was overwhelmed. The most obvious result of this was that email delivery was delayed during peak periods by as many as several hours. While it was not clear exactly why the mail server was overwhelmed, there was an obvious way to deal with the problem. We implemented this, and the mail queue was cleared in a short amount of time. We believed that this approach should have worked to prevent the mail server from being overwhelmed if the mail server again experienced heavy volume, but it did not when the same thing happened on 19 March.

We are therefore urgently preparing to set up new hardware. I don’t have an exact time line for you at the moment, but it is our goal to have the hardware in place and online by the weekend, and we will endeavour to transfer everything as quickly as we can over the weekend.

We will update you on major events by email, but please check our status page once or twice a day at status.niner.net or status.ninernet.net for progress reports.

We apologise for the inconvenience that this transition is causing, and ask for your patience as we go through this upgrade.

Tags: down time, email delivery, hardware, performance, web server
Categories: Email, Incidents, Maintenance, NC018
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?