NinerNet Communications™
System Status

Server and System Status

NC036: Maintenance port-mortem

16 October 2021 10:41:00 +0000

As stated previously, the maintenance on server NC036 is complete, and has been successful.

That said, it had to be carried out earlier than scheduled because disk space on the server was the issue that was causing a problem. To be clear, the disk space used by email accounts was not and is not the issue (that’s on a separate disk), it was the disk space used by the operating system and the logs in particular. When this problem was noticed earlier this week we projected that addressing the issue during our regular weekend maintenance window on Saturday evening UTC would be sufficient. Well, we were wrong. As soon as it became clear that our temporary mitigation efforts were failing to keep up with the rate at which disk space was being consumed (mostly by the aforementioned logs), we immediately implemented the maintenance that we had planned and already practised on a test server. This practice paid off, as instead of allowing an hour for the maintenance to complete, we were done in four minutes, including a server reboot.

That said, the lack of disk space on the server did cause disruption in the rate at which mail was processed, that led to some of you getting errors when sending email, and some incoming email being delayed. For this we apologise. The backed-up mail queue was fully processed by 09:18 UTC.

Had our prediction held up and the maintenance been carried out this evening as scheduled, we wouldn’t even have needed to post this detail. However, in order to clear up any concern, we have.

Early in 2022 server NC036 is due to be considered for replacement. The experience of this issue and other issues we have run into in recent months will inform our decisions about where to locate the replacement server (hint: it won’t be in the current data centre) and various configuration improvements we can make. We’ll also, of course, be using a more current version of the control panel.

If you have any questions at all about today’s maintenance, please let us know by contacting NinerNet support. Thank-you for your patience, and continued patronage.

NC036: Mail server disk space issue

15 October 2021 11:34:08 +0000

Server NC036 (the primary mail server) briefly ran out of disk space on Friday 15 October at 11:02 UTC. This was immediately rectified, and the server was running normally again at 11:10.

This is the reason for this weekend’s scheduled maintenance. This problem won’t happen again after the maintenance. We apologise for this temporary problem.

If you have any questions, please contact NinerNet support. Thank-you.

NC041: Connectivity issue with the primary web server this afternoon

28 July 2021 23:54:28 +0000

Server NC041, the primary web server, experienced connectivity issues between 16:04 and 16:13 earlier today, 28 July, for those 9 minutes. The server was up, but the data centre and our external monitoring service reported a lack of connectivity during that time.

The data centre now reports that it has resolved the issue, and no more issues have been experienced since.

We apologise for this gap in connectivity. If you have any questions or concerns, please contact NinerNet support. Thank-you.

NC036: Problems with shaw.ca again

13 July 2021 19:21:41 +0000

Shaw Communications Inc. and their mail filtering sub-contractor (Cloudfilter/Cloudmark) are again delaying, and now even bouncing, email sent to shaw.ca addresses. However, we have no way of communicating with Shaw mail server admins, so we cannot talk to them about how their filtering practices are non-standard and unpredictable.

Our mail server is still/again not in any of the 250 major blacklists that we monitor, so it is unclear why Shaw/Cloudfilter are causing this problem. If you are a Shaw customer, we strongly recommend that you contact Shaw and complain. We also recommend that you do not forward your email to any addresses on their domain.


Update, 2021-07-13 15:45 (PDT): Bounce messages will contain this line, or something very similar to it:

host shw-central.mx.a.cloudfilter.net[3.96.81.40] refused to talk to me: 421 shw-ibgw-4001a.ext.cloudfilter.net cmsmtp too many sessions from 178.62.195.26 AUP#CNCT

Normally bounce messages can be easily deciphered, but even we don’t know what this means or how it makes any sense.

NC041: Primary web server back online

19 June 2021 01:55:39 +0000

The primary web server (NC041) is back online with a back-up copy of the corrupted configuration file. We will be analysing the cause of the problem so that we can complete this maintenance later. The server and all websites on it were down between 22:11 and 01:45 UTC. We apologise for this, and will be making adjustments for all web-hosting accounts on your next invoice.

We will not reintroduce a new configuration file until it has been tested first. A bug report has been filed with the control panel vendor.

We apologise sincerely for the down time. If you have any questions or concerns, please contact NinerNet support.

NC041: Progress update 1

19 June 2021 00:58:24 +0000

We are in the process of restoring a back-up. Once that is done we will assess what needs to be done to manually re-create the corrupted configuration file.

NC041: Server still down

18 June 2021 22:57:55 +0000

We apologise, but it seems that a configuration file has been corrupted, and will have to be rebuilt, so it’s going to take longer than “a few minutes”. We will post updates here as they become available.

NC041: Server down

18 June 2021 22:24:26 +0000

Server NC041 is currently down. We are working on the issue right now and hope to have it back online within the next few minutes.

NC036: Email to shaw.ca delayed

7 June 2021 04:15:13 +0000

As of Sunday 6 June 2021 (Pacific Time), emails to shaw.ca are being delayed. This problem may also be affecting other domains owned by Shaw Communications Inc., but at this point we’re only aware of the issue affecting email to shaw.ca

The problem appears to be that Shaw has changed their MX record to point to a new, external mail filtering company. (The change is new; the mail filtering company is not new.) All emails to shaw.ca are being temporarily deferred, rather than hard-bounced. It’s unclear why this is, as our mail server’s IP address is not in any major blacklists of the approximately 250 that we regularly monitor. There is also no notice on Shaw’s “Service Updates & Outages” page, and no notification was received by Shaw customers of a planned outage.

We will continue to monitor this situation and, if we can, take any action to work around Shaw’s action and the unavailability of their mail system at the moment. If you have any questions in the meantime, please contact NinerNet support. If you yourself are a Shaw customer, we strongly recommend you contact them and voice your displeasure at your email being delayed and/or blocked.


Update, 2021-06-07 15:00 (PDT): We submitted two support requests to the company to which Shaw outsources their email, but they have been no help, suggesting that we have to contact Shaw directly. However, since they’re blocking our email and they don’t seem to have any kind of a corporate website, that’s rather difficult.

After our last reply to a ticket at 14:23 we checked the mail server to confirm that the copies to Shaw were blocked, and we found that almost all of the queued emails to shaw.ca addresses had been cleared; a few moments later all had been cleared and delivered.

So it seems that this issue has been resolved. However, we have had no contact from Shaw, so without an explanation for why their email stopped working we can’t be sure this won’t happen again.

NC036: Post-mortem

21 September 2020 08:42:47 +0000

As noted in the previous two posts, there was a virus outbreak on server NC036 (the primary mail server) this morning. Apparently the machines associated with five email accounts on three domains were compromised, allowing criminals to use those accounts to send thousands of viruses. These were intercepted by our anti-virus scanner, but due to the volume of activity on the server we had to shut down the SMTP side of the mail server while we determined which email accounts were compromised, suspended them and removed their messages from the mail queue.

Please note that what happens in almost all cases when email accounts are compromised is that the computer (or one of the machines or devices on which those accounts are configured) is what is actually compromised; it is not the server. The account owner’s machine is usually infected with a virus or other malware, and the account’s password is then transmitted to the criminals behind the virus. They then launch an attack via the legitimate and correct password. It’s as if your car was stolen and the thief used it to commit a crime; the car behaved as it was told by the guy with the key, but is not responsible for the crime. On the other hand, the owner of the car may have left the key in their car and the door unlocked, contributing to the compromise. This is why it is vitally important that you have anti-virus software installed on your computer, and kept up-to-date.

If you have any questions about this, please feel free to contact NinerNet support, and we’ll be happy to answer your questions or concerns. Our apologies for the interruption.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), OPERATIONAL.NC023OperationalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings configuration connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: