NinerNet Communications™
System Status

Server and System Status

NC036: Maintenance port-mortem

16 October 2021 10:41:00 +0000

As stated previously, the maintenance on server NC036 is complete, and has been successful.

That said, it had to be carried out earlier than scheduled because disk space on the server was the issue that was causing a problem. To be clear, the disk space used by email accounts was not and is not the issue (that’s on a separate disk), it was the disk space used by the operating system and the logs in particular. When this problem was noticed earlier this week we projected that addressing the issue during our regular weekend maintenance window on Saturday evening UTC would be sufficient. Well, we were wrong. As soon as it became clear that our temporary mitigation efforts were failing to keep up with the rate at which disk space was being consumed (mostly by the aforementioned logs), we immediately implemented the maintenance that we had planned and already practised on a test server. This practice paid off, as instead of allowing an hour for the maintenance to complete, we were done in four minutes, including a server reboot.

That said, the lack of disk space on the server did cause disruption in the rate at which mail was processed, that led to some of you getting errors when sending email, and some incoming email being delayed. For this we apologise. The backed-up mail queue was fully processed by 09:18 UTC.

Had our prediction held up and the maintenance been carried out this evening as scheduled, we wouldn’t even have needed to post this detail. However, in order to clear up any concern, we have.

Early in 2022 server NC036 is due to be considered for replacement. The experience of this issue and other issues we have run into in recent months will inform our decisions about where to locate the replacement server (hint: it won’t be in the current data centre) and various configuration improvements we can make. We’ll also, of course, be using a more current version of the control panel.

If you have any questions at all about today’s maintenance, please let us know by contacting NinerNet support. Thank-you for your patience, and continued patronage.

NC036: Maintenance complete

16 October 2021 09:05:46 +0000

The maintenance on server NC036 (the primary mail server) is complete and the server is back online. It was unavailable between 08:28 and 08:32 UTC. According to some reports there have been issues with delayed emails and “lookup failure” errors, but these have now cleared.

We’ll be posting some further information shortly with more details.

NC036: Weekend maintenance moved up, starting in the next few minutes

16 October 2021 08:23:31 +0000

The situation on server NC036 needs to be addressed immediately. We will be doing this maintenance in the next few minutes. We will post updates here.

NC036: Maintenance this weekend

13 October 2021 11:30:19 +0000

Server NC036 (the primary mail server) will be down briefly this weekend during the weekly maintenance window while we again increase the disk space available on the server. The downtime should be less than an hour during the least busy part of the weekend on Saturday evening UTC.

During the downtime you will not be able to send or check email, and incoming email will be held on the sending servers until server NC036 is back online. Once the server is back online mail will be delivered in line with standard retry rates. This will mean that some incoming mail may be delayed briefly, but no mail will be lost.

We will post updates here shortly before the maintenance begins, and again once it has concluded.

If you have any questions or concerns, please contact NinerNet support. Thank-you.

NC041: Server not rebooted

8 October 2021 06:36:57 +0000

Contrary to our earlier post a few minutes ago, we are not rebooting server NC041. The reboot is not needed. Apologies for the false alarm.

NC041: Emergency server reboot

8 October 2021 06:30:16 +0000

We will be rebooting server NC041 (the primary web server) very shortly following some updates. It will be down for fewer than 60 seconds, and we’ll post here to confirm it is up again. This does not affect email at all.

NC041: Server still down

18 June 2021 22:57:55 +0000

We apologise, but it seems that a configuration file has been corrupted, and will have to be rebuilt, so it’s going to take longer than “a few minutes”. We will post updates here as they become available.

NC041: Server down

18 June 2021 22:24:26 +0000

Server NC041 is currently down. We are working on the issue right now and hope to have it back online within the next few minutes.

NC041: Server back online

18 June 2021 22:13:39 +0000

NC041 is back online having been rebooted. It was down between 15:10 and 15:11. We are continuing with the maintenance.

NC041: Server reboot

18 June 2021 22:08:30 +0000

We need to reboot server NC041 (the primary web server) to finish an upgrade. There server will be down for about 60 seconds in a few minutes.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: