NinerNet Communications™
System Status

Server and System Status

NC041: Server down

18 June 2021 22:24:26 +0000

Server NC041 is currently down. We are working on the issue right now and hope to have it back online within the next few minutes.

NC036: Email to shaw.ca delayed

7 June 2021 04:15:13 +0000

As of Sunday 6 June 2021 (Pacific Time), emails to shaw.ca are being delayed. This problem may also be affecting other domains owned by Shaw Communications Inc., but at this point we’re only aware of the issue affecting email to shaw.ca

The problem appears to be that Shaw has changed their MX record to point to a new, external mail filtering company. (The change is new; the mail filtering company is not new.) All emails to shaw.ca are being temporarily deferred, rather than hard-bounced. It’s unclear why this is, as our mail server’s IP address is not in any major blacklists of the approximately 250 that we regularly monitor. There is also no notice on Shaw’s “Service Updates & Outages” page, and no notification was received by Shaw customers of a planned outage.

We will continue to monitor this situation and, if we can, take any action to work around Shaw’s action and the unavailability of their mail system at the moment. If you have any questions in the meantime, please contact NinerNet support. If you yourself are a Shaw customer, we strongly recommend you contact them and voice your displeasure at your email being delayed and/or blocked.


Update, 2021-06-07 15:00 (PDT): We submitted two support requests to the company to which Shaw outsources their email, but they have been no help, suggesting that we have to contact Shaw directly. However, since they’re blocking our email and they don’t seem to have any kind of a corporate website, that’s rather difficult.

After our last reply to a ticket at 14:23 we checked the mail server to confirm that the copies to Shaw were blocked, and we found that almost all of the queued emails to shaw.ca addresses had been cleared; a few moments later all had been cleared and delivered.

So it seems that this issue has been resolved. However, we have had no contact from Shaw, so without an explanation for why their email stopped working we can’t be sure this won’t happen again.

NC036: Post-mortem

21 September 2020 08:42:47 +0000

As noted in the previous two posts, there was a virus outbreak on server NC036 (the primary mail server) this morning. Apparently the machines associated with five email accounts on three domains were compromised, allowing criminals to use those accounts to send thousands of viruses. These were intercepted by our anti-virus scanner, but due to the volume of activity on the server we had to shut down the SMTP side of the mail server while we determined which email accounts were compromised, suspended them and removed their messages from the mail queue.

Please note that what happens in almost all cases when email accounts are compromised is that the computer (or one of the machines or devices on which those accounts are configured) is what is actually compromised; it is not the server. The account owner’s machine is usually infected with a virus or other malware, and the account’s password is then transmitted to the criminals behind the virus. They then launch an attack via the legitimate and correct password. It’s as if your car was stolen and the thief used it to commit a crime; the car behaved as it was told by the guy with the key, but is not responsible for the crime. On the other hand, the owner of the car may have left the key in their car and the door unlocked, contributing to the compromise. This is why it is vitally important that you have anti-virus software installed on your computer, and kept up-to-date.

If you have any questions about this, please feel free to contact NinerNet support, and we’ll be happy to answer your questions or concerns. Our apologies for the interruption.

NC036: Mail server is back online

21 September 2020 07:26:54 +0000

Our apologies. The sending side of the mail server (NC036) is back up. It was down for 21 minutes between 06:56 and 07:17 UTC. The ability to check your email account was not down.

We will post additional information and contact the affected clients shortly.

NC036: Mail server paused while we clean up a spam outbreak

21 September 2020 07:05:48 +0000

Server NC036 (the primary mail server) is temporarily paused while we clean up a huge virus outbreak. I will be back online as soon as possible.

Phone system down

4 June 2020 04:31:52 +0000

Clients should be aware that our North American phone system is down right now, and has been since shortly after 17:00 Pacific time today, 3 June (00:00 UTC on 4 June). Currently we have no estimate from our phone provider when this issue will be rectified.

However, of course, email is our primary method of communication and providing support, so please continue to use that. It’s working just fine.


Update: The phone system was down for about five hours due to a data centre issue during maintenance.

NC041: Automated certificate renewals should be working again

12 January 2020 13:01:11 +0000

The issue with automated renewals of Let’s Encrypt certificates on server NC041 should now be fixed. We have taken the action suggested by the control panel vendor, and we are waiting for the first automated renewal to occur to see if everything is once again working. If, as we expect, the issue is fixed there will be no further posts on this subject.

If you receive any error messages related to certificate renewals — or any error emails at all — please contact NinerNet support and we’ll investigate further. Thank-you.

NC041: Automated renewals of Let’s Encrypt certificates failing

10 January 2020 03:21:00 +0000

Since about 1 January automated renewals of Let’s Encrypt SSL certificates have been failing on server NC041, and as a result you may receive automated emails from server NC041 that look like this:

Subject: Let’s Encrypt certificate renewal failed!

An error occurred requesting a new certificate for example.com, www.example.com from Let’s Encrypt : Web-based validation failed : <pre>sh: certonly: command not found
</pre>

We currently have a support ticket open with the control panel vendor to find out why this has suddenly started.

Since the renewal of Let’s Encrypt certificates is automatically scheduled to happen one month before they expire, the existing certificates will continue to be used and remain valid and in force. The issue will be resolved before these certificates start expiring in February.

If you have any concerns or questions about this, please contact NinerNet support. Thank-you.

NC033: Incident with primary nameserver resolved

15 November 2019 03:10:35 +0000

Following an operating system update that was applied at 00:37 UTC on 15 November the primary nameserver failed to restart. This was noticed immediately upon completion of the update, and we started troubleshooting.

After extensive research on the likely cause and resolution of the problem we determined that the quickest way to get the nameserver back online was to roll back the update and restore from a back-up. There were some issues with this, however, that necessitated implementing a back-up plan for company email in order for us to gain access to our account at the data centre where the nameserver is hosted. Once this was accomplished the restore was initiated.

The nameserver was back online at 02:28 UTC.

During the time the server was down domain information would have been cached by DNS resolvers that already had a cached copy of the DNS information for your domain. However, it is possible that some incoming email may have been delayed or possibly bounced, or people trying to browse your website would not have been able to see it. For this we apologise.

If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience.

Mail server latency issue resolved

30 September 2019 23:25:22 +0000

This issue on server NC036 appears to have been resolved, although we are awaiting official confirmation from the data centre. Should there be no change of status this issue will be closed with no further update here.

As always though, if you have any issues or concerns please contact support. Thank-you for your patience.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: