Server and System Status

NC027: Server back online

21 November 2014 04:23:48 +0000

We have completed adding additional storage to server NC027. The server was taken down at 03:13 and was back online 28 minutes later at 03:41.

However, we have decided to postpone the other upgrades until this weekend, during our weekly maintenance window starting at 19:00 UTC on Saturday, 22 November.

Thanks again for your patience and understanding.

Tags: down time, email delivery, hardware, reboot
Categories: Maintenance, NC027, Upgrades
Comments Off

We will be performing emergency maintenance on server NC027 today during our daily maintenance window starting at 03:00 UTC. We are allowing a 90-minute window for this maintenance to add more RAM, bandwidth and storage, but we expect that it will take less time. We will post an update here as soon as the maintenance is complete.

If you have any questions or concerns about this, please contact NinerNet support. Thank-you.

Tags: down time, email delivery, hardware, reboot
Categories: Maintenance, NC027, Upgrades
Comments Off

NC027: Maintenance complete

15 March 2014 19:53:18 +0000

Server NC027 was brought back online at 19:43 UTC. This maintenance has been completed successfully.

If you have any questions or concerns, please contact NinerNet support. Thanks very much for your patience while we upgraded the resources on this server.

Tags: down time, email delivery
Categories: Maintenance, NC027
Comments Off

NC027: Maintenance update

15 March 2014 19:33:23 +0000

The server was taken down for maintenance at 19:23 UTC. The maintenance is underway at this time.

We will post an update here as soon as the server is back online.

Tags: down time, email delivery
Categories: Maintenance, NC027
Comments Off

NC027: System maintenance

13 March 2014 10:38:17 +0000

We will be upgrading server NC027 over the weekend to increase its capacity. This maintenance will take about 40 minutes, during which the server will be completely inaccessible. However, incoming mail will be spooled on sending systems, and will be delivered when the server comes back up. No email will be lost or bounced.

Here are the details:

Start of maintenance window: 19:00 15 March 2014 UTC
Length of maintenance window: 1 hour

While the length of the maintenance window is an hour, the actual length of the maintenance should only be about 40 minutes sometime during that hour. If either the maintenance window or the length of the maintenance should change, we will post updates here.

To convert the above time and date to your time zone, please visit the World Time Server website.

If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience while we work to improve the service that we offer to you.

Tags: down time, email delivery
Categories: Maintenance, NC027
Comments Off

NC027: Investigation findings

26 January 2014 16:00:09 +0000

Server NC027 (the new mail-only server) experienced a great and very brief increase in activity at about 04:00 UTC on 23 January, and we suspect that this lead to the server getting behind in scanning and processing email. The hard drives simply couldn’t keep up.

In order to allow the server to catch up, we disabled virus and spam scanning. Over the course of the coming week we’ll be re-enabling each of these (inbound and outbound) one at a time per day. This will allow us to determine which activity is the bottleneck and make finer adjustments accordingly. We also have a back-up plan to move the accounts on this server to one with faster (solid-state) drives, if necessary.

We apologise again for the inconvenience of significantly delayed email on Thursday. If you have any questions about this incident, please contact NinerNet support. Thank-you.

Tags: email delivery, performance
Categories: Email, Incidents, NC027
Comments Off

NC023: Spam cleaned up

26 January 2014 15:45:01 +0000

The spam on server NC023 (the relay server) has been cleaned up and the offending IP address blocked. Most of the legitimate email to destination servers that had been deferred as a result of the spam has also now been delivered; however, some mail servers are still not accepting email from this server, so that email will continue to be delayed a while longer until their systems are automatically updated to realise that this server is no longer spamming. This is currently only affecting about twenty messages.

We apologise for this interruption. If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience.

Tags: email delivery, mail relay, spam
Categories: Email, Incidents, NC023
Comments Off

NC023: Relay server stopped to clean up spam

26 January 2014 13:28:29 +0000

The relay server has been stopped while we clean up a spam outbreak, and identify and block the source. We will post updates here. We expect to have the server back online within the next 30-40 minutes.

Tags: email delivery, mail relay, spam
Categories: Email, Incidents, NC023
Comments Off

NC027: Mail queue almost clear

23 January 2014 14:48:04 +0000

The mail queue will be cleared within the next 20-30 minutes.

Once the queue is clear we’ll be looking for a cause for this incident. There was a great spike in activity on this server around 04:00 UTC, and it’s quite possible that the initial back-up started around that time and didn’t quite manage to catch up.

Tags: email delivery, performance
Categories: Email, Incidents, NC027
Comments Off

NC027: New mail delivered immediately

23 January 2014 13:25:21 +0000

Since reconfiguring the mail server we’ve noticed that new email is being delivered right away. It is now only the backlog that is still slowly being processed.

Tags: email delivery, performance
Categories: Email, Incidents, NC027
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?