NinerNet Communications™
System Status

Server and System Status

NC036: Maintenance port-mortem

16 October 2021 10:41:00 +0000

As stated previously, the maintenance on server NC036 is complete, and has been successful.

That said, it had to be carried out earlier than scheduled because disk space on the server was the issue that was causing a problem. To be clear, the disk space used by email accounts was not and is not the issue (that’s on a separate disk), it was the disk space used by the operating system and the logs in particular. When this problem was noticed earlier this week we projected that addressing the issue during our regular weekend maintenance window on Saturday evening UTC would be sufficient. Well, we were wrong. As soon as it became clear that our temporary mitigation efforts were failing to keep up with the rate at which disk space was being consumed (mostly by the aforementioned logs), we immediately implemented the maintenance that we had planned and already practised on a test server. This practice paid off, as instead of allowing an hour for the maintenance to complete, we were done in four minutes, including a server reboot.

That said, the lack of disk space on the server did cause disruption in the rate at which mail was processed, that led to some of you getting errors when sending email, and some incoming email being delayed. For this we apologise. The backed-up mail queue was fully processed by 09:18 UTC.

Had our prediction held up and the maintenance been carried out this evening as scheduled, we wouldn’t even have needed to post this detail. However, in order to clear up any concern, we have.

Early in 2022 server NC036 is due to be considered for replacement. The experience of this issue and other issues we have run into in recent months will inform our decisions about where to locate the replacement server (hint: it won’t be in the current data centre) and various configuration improvements we can make. We’ll also, of course, be using a more current version of the control panel.

If you have any questions at all about today’s maintenance, please let us know by contacting NinerNet support. Thank-you for your patience, and continued patronage.

NC036: Maintenance complete

16 October 2021 09:05:46 +0000

The maintenance on server NC036 (the primary mail server) is complete and the server is back online. It was unavailable between 08:28 and 08:32 UTC. According to some reports there have been issues with delayed emails and “lookup failure” errors, but these have now cleared.

We’ll be posting some further information shortly with more details.

NC036: Weekend maintenance moved up, starting in the next few minutes

16 October 2021 08:23:31 +0000

The situation on server NC036 needs to be addressed immediately. We will be doing this maintenance in the next few minutes. We will post updates here.

NC036: Mail server disk space issue

15 October 2021 11:34:08 +0000

Server NC036 (the primary mail server) briefly ran out of disk space on Friday 15 October at 11:02 UTC. This was immediately rectified, and the server was running normally again at 11:10.

This is the reason for this weekend’s scheduled maintenance. This problem won’t happen again after the maintenance. We apologise for this temporary problem.

If you have any questions, please contact NinerNet support. Thank-you.

NC036: Maintenance this weekend

13 October 2021 11:30:19 +0000

Server NC036 (the primary mail server) will be down briefly this weekend during the weekly maintenance window while we again increase the disk space available on the server. The downtime should be less than an hour during the least busy part of the weekend on Saturday evening UTC.

During the downtime you will not be able to send or check email, and incoming email will be held on the sending servers until server NC036 is back online. Once the server is back online mail will be delivered in line with standard retry rates. This will mean that some incoming mail may be delayed briefly, but no mail will be lost.

We will post updates here shortly before the maintenance begins, and again once it has concluded.

If you have any questions or concerns, please contact NinerNet support. Thank-you.

NC036: Maintenance complete

16 June 2021 02:17:12 +0000

Server NC036 has been rebooted and is back online. All functions have been checked, and all services are running normally.

If you have any questions, please contact NinerNet support. Thank-you.

NC036: Server restart

16 June 2021 02:07:49 +0000

Server NC036 will be rebooted shortly, following recent maintenance. It will be inaccessible for about one minute. We will post again when it is back online.

NC036: Maintenance complete

30 May 2021 23:03:44 +0000

The scheduled maintenance on server NC036 is complete. The ability to send and receive email was not available between 22:47 and 22:53 on 30 May UTC. The new disk space has been tested and all is running normally.

Thank-you for your patience. If you have any questions or concerns, please contact NinerNet support. Thank-you.

NC036: Scheduled mail server maintenance

30 May 2021 22:44:06 +0000

During this weekend’s maintenance window we will be adding hard drive storage to server NC036 to continue to provide more storage space for a growing number of growing email accounts. This maintenance is scheduled to start in a few minutes, it will conclude in less than half an hour.

During the maintenance the ability to send and receive email will not be available, both via standalone email programs (e.g., Outlook, Thunderbird, etc.) and the webmail. Incoming email will be queued on the sending servers until our server is back online again, after which it will then be delivered to our server and your email account. This may result in a delay longer than the planned length of the maintenance though.

Please monitor this status page to be notified of the end of the maintenance. If you have any questions or concerns, please contact NinerNet support.

Thank-you for your patience as we continue to work to improve our services to you.

Virus update 3

12 August 2020 05:41:35 +0000

As we approach a week with this issue, we are forced to make a decision. We continue to evaluate a number of options, and we’ll implement more than one, but the bottom line for us is a usable email system for our clients, or at least the vast majority of you.

In order to allow the vast majority of our clients to carry on business as usual, at 04:15 UTC today, 12 August 2020, we removed the attachment restriction we put in place late last week. According to reports, this will result in difficulty for one, or possibly two clients who have been overwhelmed recently by spam email that contained these attachments. On the other hand, we have heard those clients that have told us that their businesses have been stalled by these restrictions.

However, that isn’t the last or only action we have taken or will take on this matter. These are others:

  • The lack of speed with which our anti-virus vendor (ClamAV) has picked up this virus means we will be looking at options for either internal or external secondary virus scanning of our incoming and outgoing email streams. If this was something we already had in place, this disruption would have been hardly noticed by more than a few clients. This is the priority with which we are currently seized.
  • We continue to look into ways to apply certain restrictions to some domains and not others in a virtual-hosting environment.
  • Virus samples have been submitted to ClamAV and we anticipate that they will add the recently received viruses to their virus-detection database in due course.
  • We will make an offer to the client most affected by this virus outbreak to move their hosting to a virtual private server of their own.
  • We have blocked several thousand IP addresses that have been the sources of the problem attachments over the last week, and we will block more.
  • Server NC036 is approaching its scheduled replacement time frame. Had our plans been further along when this happened we’d have executed them immediately. However, there are always potential issues when implementing half-baked plans too far ahead of schedule, so we didn’t.

There are potential issues that you need to be aware of:

  • Some anti-virus scanners, not the least of which is ClamAV, are not detecting some of the recent viruses we have seen. You must ensure that you have anti-virus software installed on all of your machines and devices, and you must ensure that this software is automatically updated at least daily. Please remember that the responsibility for the safety of your computer(s) and your data is ultimately yours.
  • You and your employees need to be aware of the risks of opening attachments, and need to be aware of how to evaluate that risk. The risks are to both your machines and devices, and to your organisation and employees.

What have we learned from this experience? Other than our surprise at the degree to which some businesses rely significantly on Microsoft Word documents flying around the Internet, we have learned that the anti-virus vendor on which we have relied without issue for about a decade is not, in fact, infallible. We need, and will obtain, redundancy in this area.

Something else we have decided to act on is SPF (Sender Policy Framework) records. All domains we host have SPF records that tell all mail servers on the Internet that they should accept email only from our servers. The records have all ended with “~all”, but this weekend we will update all records to read “-all”. The difference is that the old records with the tilde (~) allowed receiving servers to act with some leeway, a so-called softfail; the hyphen (-) will enforce that all mail received from domains we host must come from our servers, or any others that are designated in the domain’s SPF record. What will this accomplish? One of the things we have seen is the domain of one of our clients being extensively and aggressively “spoofed”. This is when emails are sent purporting to come from a domain other than their real origin. SPF is designed to prevent this, but the directive with the tilde allows leeway that, it seems, can be too easily abused.

If you believe this may be an issue for you, or if you have any questions at all, please contact NinerNet support and we will assist.

Thank-you very much for your patience.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), OPERATIONAL.NC023OperationalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings configuration connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: