NinerNet Communications™
System Status

Server and System Status

NC036: Post-mortem

21 September 2020 08:42:47 +0000

As noted in the previous two posts, there was a virus outbreak on server NC036 (the primary mail server) this morning. Apparently the machines associated with five email accounts on three domains were compromised, allowing criminals to use those accounts to send thousands of viruses. These were intercepted by our anti-virus scanner, but due to the volume of activity on the server we had to shut down the SMTP side of the mail server while we determined which email accounts were compromised, suspended them and removed their messages from the mail queue.

Please note that what happens in almost all cases when email accounts are compromised is that the computer (or one of the machines or devices on which those accounts are configured) is what is actually compromised; it is not the server. The account owner’s machine is usually infected with a virus or other malware, and the account’s password is then transmitted to the criminals behind the virus. They then launch an attack via the legitimate and correct password. It’s as if your car was stolen and the thief used it to commit a crime; the car behaved as it was told by the guy with the key, but is not responsible for the crime. On the other hand, the owner of the car may have left the key in their car and the door unlocked, contributing to the compromise. This is why it is vitally important that you have anti-virus software installed on your computer, and kept up-to-date.

If you have any questions about this, please feel free to contact NinerNet support, and we’ll be happy to answer your questions or concerns. Our apologies for the interruption.

NC036: Mail server is back online

21 September 2020 07:26:54 +0000

Our apologies. The sending side of the mail server (NC036) is back up. It was down for 21 minutes between 06:56 and 07:17 UTC. The ability to check your email account was not down.

We will post additional information and contact the affected clients shortly.

NC036: Mail server paused while we clean up a spam outbreak

21 September 2020 07:05:48 +0000

Server NC036 (the primary mail server) is temporarily paused while we clean up a huge virus outbreak. I will be back online as soon as possible.

Virus update 3

12 August 2020 05:41:35 +0000

As we approach a week with this issue, we are forced to make a decision. We continue to evaluate a number of options, and we’ll implement more than one, but the bottom line for us is a usable email system for our clients, or at least the vast majority of you.

In order to allow the vast majority of our clients to carry on business as usual, at 04:15 UTC today, 12 August 2020, we removed the attachment restriction we put in place late last week. According to reports, this will result in difficulty for one, or possibly two clients who have been overwhelmed recently by spam email that contained these attachments. On the other hand, we have heard those clients that have told us that their businesses have been stalled by these restrictions.

However, that isn’t the last or only action we have taken or will take on this matter. These are others:

  • The lack of speed with which our anti-virus vendor (ClamAV) has picked up this virus means we will be looking at options for either internal or external secondary virus scanning of our incoming and outgoing email streams. If this was something we already had in place, this disruption would have been hardly noticed by more than a few clients. This is the priority with which we are currently seized.
  • We continue to look into ways to apply certain restrictions to some domains and not others in a virtual-hosting environment.
  • Virus samples have been submitted to ClamAV and we anticipate that they will add the recently received viruses to their virus-detection database in due course.
  • We will make an offer to the client most affected by this virus outbreak to move their hosting to a virtual private server of their own.
  • We have blocked several thousand IP addresses that have been the sources of the problem attachments over the last week, and we will block more.
  • Server NC036 is approaching its scheduled replacement time frame. Had our plans been further along when this happened we’d have executed them immediately. However, there are always potential issues when implementing half-baked plans too far ahead of schedule, so we didn’t.

There are potential issues that you need to be aware of:

  • Some anti-virus scanners, not the least of which is ClamAV, are not detecting some of the recent viruses we have seen. You must ensure that you have anti-virus software installed on all of your machines and devices, and you must ensure that this software is automatically updated at least daily. Please remember that the responsibility for the safety of your computer(s) and your data is ultimately yours.
  • You and your employees need to be aware of the risks of opening attachments, and need to be aware of how to evaluate that risk. The risks are to both your machines and devices, and to your organisation and employees.

What have we learned from this experience? Other than our surprise at the degree to which some businesses rely significantly on Microsoft Word documents flying around the Internet, we have learned that the anti-virus vendor on which we have relied without issue for about a decade is not, in fact, infallible. We need, and will obtain, redundancy in this area.

Something else we have decided to act on is SPF (Sender Policy Framework) records. All domains we host have SPF records that tell all mail servers on the Internet that they should accept email only from our servers. The records have all ended with “~all”, but this weekend we will update all records to read “-all”. The difference is that the old records with the tilde (~) allowed receiving servers to act with some leeway, a so-called softfail; the hyphen (-) will enforce that all mail received from domains we host must come from our servers, or any others that are designated in the domain’s SPF record. What will this accomplish? One of the things we have seen is the domain of one of our clients being extensively and aggressively “spoofed”. This is when emails are sent purporting to come from a domain other than their real origin. SPF is designed to prevent this, but the directive with the tilde allows leeway that, it seems, can be too easily abused.

If you believe this may be an issue for you, or if you have any questions at all, please contact NinerNet support and we will assist.

Thank-you very much for your patience.

Virus update

10 August 2020 09:53:46 +0000

After three updates to the virus database on NC036 since Thursday we expected that the anti-virus scanner would detect and block the trojan that is currently overwhelming some of our clients. However, that is not the case.

For that reason we are once again blocking the attachments that are the cause of this problem. We sincerely apologise for this situation.

Again though, if you need to send a blocked attachment type, you can still do so if you compress the file into a .zip document. Your correspondents can also do the same.

If you are getting an error message when trying to send email, the first thing you need to do is check to see if you have attached a file to your message. If you have, and it is one of Microsoft’s typical Word documents, please remove it, compress it, attach the compressed .zip file, and then send that attachment.

If you have any questions or concerns, please contact NinerNet support. Thank-you.

Zero-day virus getting through the mail server

6 August 2020 13:39:04 +0000

Within the last four hours we have been made aware of a trojan that is getting through our anti-virus scanner undetected. Once we were able to determine the file types that were attached to the emails, we blocked those kinds of attachments from being delivered to the server. Doing so also resulted in our being able to compile a list of sources of the offending messages, and we have been busy blocking email from scores of IP addresses.

Although we do scan both inbound and outbound email in real time for viruses, we do very strongly recommend that you have an anti-virus program installed on your local computers so that if anything does get through, it will protect your machines. Please remember that unsolicited attachments from unknown senders are extremely risky for you to open; even attachments from known senders are risky; please contact the sender through some other method — e.g., a quick phone call or text message — to confirm that they sent it and that it is safe to open. Even then you should first ensure your anti-virus software has been updated, save the file to your hard drive, and then manually scan it for viruses. Only if you have carried out all of the above should you consider opening the file.

Please remember that the responsibility for the safety of your computer and your data is ultimately yours.

We expect that the anti-virus vendor will update their virus signatures in due course. Until then we will be blocking all attachments that look the same as this particular outbreak. If you have a correspondent who needs to get a blocked attachment through, please tell them to compress the file and send that attachment instead.

If you have any questions, please contact NinerNet support. Thank-you.

NC036: Maintenance complete

21 July 2020 01:03:48 +0000

The scheduled maintenance on server NC036 is complete. The ability to send and receive email was not available between 00:44 and 00:50 on 21 July UTC. The new disk space has been tested and all is running normally.

Thank-you for your patience. If you have any questions or concerns, please contact NinerNet support. Thank-you.

NC036: Scheduled mail server maintenance

21 July 2020 00:43:22 +0000

During tonight’s maintenance window we will be adding hard drive storage to server NC036 to continue to provide more storage space for a growing number of growing email accounts. This maintenance is scheduled to start between 00:45 and 01:00 UTC and we anticipate it will last less than one hour.

During the maintenance the ability to send and receive email will not be available, both via standalone email programs (e.g., Outlook, Thunderbird, etc.) and the webmail. Incoming email will be queued on the sending servers until our server is back online again, after which it will then be delivered to our server and your email account. This may result in a delay longer than the planned hour of the maintenance though.

Please monitor this status page to be notified of the start and end of the maintenance. If you have any questions or concerns, please contact NinerNet support.

Thank-you for your patience as we continue to work to improve our services to you.

Phone system down

4 June 2020 04:31:52 +0000

Clients should be aware that our North American phone system is down right now, and has been since shortly after 17:00 Pacific time today, 3 June (00:00 UTC on 4 June). Currently we have no estimate from our phone provider when this issue will be rectified.

However, of course, email is our primary method of communication and providing support, so please continue to use that. It’s working just fine.


Update: The phone system was down for about five hours due to a data centre issue during maintenance.

NC036: Emails to Hotmail/Outlook being redirected

22 May 2020 07:03:49 +0000

Some clients may have had emails they sent to Hotmail, Outlook and other Microsoft email service domains bounced in the last 24 hours. This is because the primary mail server’s IP address has been blacklisted by Microsoft, despite our being enrolled in their “Junk Mail Reporting Program”. Our being enrolled is supposed to result in our being informed of spam complaints (which are overwhelmingly false) before our IP address is blacklisted, but their system seems to have been broken for the last few years.

Until we can get in touch with Microsoft support to have our IP address removed, or until our listing expires, we are automatically directing all email to their domains through our relay server, which is not blacklisted.

If you had an email bounced, please resend it. Thank-you, and sorry for the hassle. If you have any questions please contact NinerNet support.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), OPERATIONAL.NC023OperationalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings configuration connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking outlook performance php phplist pop reboot smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: