NinerNet Communications™
System Status

Server and System Status

Virus update 3

12 August 2020 05:41:35 +0000

As we approach a week with this issue, we are forced to make a decision. We continue to evaluate a number of options, and we’ll implement more than one, but the bottom line for us is a usable email system for our clients, or at least the vast majority of you.

In order to allow the vast majority of our clients to carry on business as usual, at 04:15 UTC today, 12 August 2020, we removed the attachment restriction we put in place late last week. According to reports, this will result in difficulty for one, or possibly two clients who have been overwhelmed recently by spam email that contained these attachments. On the other hand, we have heard those clients that have told us that their businesses have been stalled by these restrictions.

However, that isn’t the last or only action we have taken or will take on this matter. These are others:

  • The lack of speed with which our anti-virus vendor (ClamAV) has picked up this virus means we will be looking at options for either internal or external secondary virus scanning of our incoming and outgoing email streams. If this was something we already had in place, this disruption would have been hardly noticed by more than a few clients. This is the priority with which we are currently seized.
  • We continue to look into ways to apply certain restrictions to some domains and not others in a virtual-hosting environment.
  • Virus samples have been submitted to ClamAV and we anticipate that they will add the recently received viruses to their virus-detection database in due course.
  • We will make an offer to the client most affected by this virus outbreak to move their hosting to a virtual private server of their own.
  • We have blocked several thousand IP addresses that have been the sources of the problem attachments over the last week, and we will block more.
  • Server NC036 is approaching its scheduled replacement time frame. Had our plans been further along when this happened we’d have executed them immediately. However, there are always potential issues when implementing half-baked plans too far ahead of schedule, so we didn’t.

There are potential issues that you need to be aware of:

  • Some anti-virus scanners, not the least of which is ClamAV, are not detecting some of the recent viruses we have seen. You must ensure that you have anti-virus software installed on all of your machines and devices, and you must ensure that this software is automatically updated at least daily. Please remember that the responsibility for the safety of your computer(s) and your data is ultimately yours.
  • You and your employees need to be aware of the risks of opening attachments, and need to be aware of how to evaluate that risk. The risks are to both your machines and devices, and to your organisation and employees.

What have we learned from this experience? Other than our surprise at the degree to which some businesses rely significantly on Microsoft Word documents flying around the Internet, we have learned that the anti-virus vendor on which we have relied without issue for about a decade is not, in fact, infallible. We need, and will obtain, redundancy in this area.

Something else we have decided to act on is SPF (Sender Policy Framework) records. All domains we host have SPF records that tell all mail servers on the Internet that they should accept email only from our servers. The records have all ended with “~all”, but this weekend we will update all records to read “-all”. The difference is that the old records with the tilde (~) allowed receiving servers to act with some leeway, a so-called softfail; the hyphen (-) will enforce that all mail received from domains we host must come from our servers, or any others that are designated in the domain’s SPF record. What will this accomplish? One of the things we have seen is the domain of one of our clients being extensively and aggressively “spoofed”. This is when emails are sent purporting to come from a domain other than their real origin. SPF is designed to prevent this, but the directive with the tilde allows leeway that, it seems, can be too easily abused.

If you believe this may be an issue for you, or if you have any questions at all, please contact NinerNet support and we will assist.

Thank-you very much for your patience.

Virus update

10 August 2020 09:53:46 +0000

After three updates to the virus database on NC036 since Thursday we expected that the anti-virus scanner would detect and block the trojan that is currently overwhelming some of our clients. However, that is not the case.

For that reason we are once again blocking the attachments that are the cause of this problem. We sincerely apologise for this situation.

Again though, if you need to send a blocked attachment type, you can still do so if you compress the file into a .zip document. Your correspondents can also do the same.

If you are getting an error message when trying to send email, the first thing you need to do is check to see if you have attached a file to your message. If you have, and it is one of Microsoft’s typical Word documents, please remove it, compress it, attach the compressed .zip file, and then send that attachment.

If you have any questions or concerns, please contact NinerNet support. Thank-you.

Zero-day virus getting through the mail server

6 August 2020 13:39:04 +0000

Within the last four hours we have been made aware of a trojan that is getting through our anti-virus scanner undetected. Once we were able to determine the file types that were attached to the emails, we blocked those kinds of attachments from being delivered to the server. Doing so also resulted in our being able to compile a list of sources of the offending messages, and we have been busy blocking email from scores of IP addresses.

Although we do scan both inbound and outbound email in real time for viruses, we do very strongly recommend that you have an anti-virus program installed on your local computers so that if anything does get through, it will protect your machines. Please remember that unsolicited attachments from unknown senders are extremely risky for you to open; even attachments from known senders are risky; please contact the sender through some other method — e.g., a quick phone call or text message — to confirm that they sent it and that it is safe to open. Even then you should first ensure your anti-virus software has been updated, save the file to your hard drive, and then manually scan it for viruses. Only if you have carried out all of the above should you consider opening the file.

Please remember that the responsibility for the safety of your computer and your data is ultimately yours.

We expect that the anti-virus vendor will update their virus signatures in due course. Until then we will be blocking all attachments that look the same as this particular outbreak. If you have a correspondent who needs to get a blocked attachment through, please tell them to compress the file and send that attachment instead.

If you have any questions, please contact NinerNet support. Thank-you.

NC036: Mail server blocked by Microsoft

14 December 2018 05:33:38 +0000

We are aware that the IP address of server NC036 (the primary mail server) has again been blocked by Microsoft’s various mail services, variously known as Outlook.com, MSN, Hotmail, Live.com, etc.

Although we are a member of their Smart Network Data Services programme and Junk Mail Reporting Program, which are supposed to allow us to proactively prevent these kinds of issues, we have been unable to use the service as advertised, or at least as we understand it’s supposed to work. We will continue to attempt to have this server’s IP address removed from their blacklist, and report here when we have success.

In the meantime, outgoing mail to their primary domains (hotmail.ca, hotmail.com, hotmail.co.uk, live.com, msn.com and outlook.com) is being routed through our relay server. If you receive a bounce message that reads similarly the one below to an email you’ve sent, it is probably for a private domain hosted by Microsoft of which we are not aware. Please contact us and we will add it to the list of domains for which email is routed through our relay server:

host 901e3cd0af6f44ab11b5a5e8a49da3.pamx1.hotmail.com[104.47.0.33] said: 550
5.7.1 Unfortunately, messages from [178.62.195.26] weren’t sent. Please
contact your Internet service provider since part of their network is on
our block list (S3140). You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors.
[HE1EUR01FT033.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
FROM command)

Please remember that all email you send through our mail server must be to recipients with whom you already have a business or personal relationship, and all mass email must be explicitly requested — i.e., Confirmed opt-in (COI) or Double opt-in (DOI) email.

Thanks for your cooperation, and our apologies for this inconvenience.

NC036: Migration update 25 — Final

18 June 2018 08:54:43 +0000

The migration of all email accounts from server NC027 to server NC036 is complete. In fact, it was successfully completed at 04:00 UTC on 4 June. What followed over the next few days was an unprecedented avalanche of misinformation and red herrings that resulted in our moving the new server to another data centre (a move that took ten times longer than the previous move from the data centre where NC027 was located) where the same “problems” experienced by only some of our clients magically reappeared.

We planned the migration to have absolutely no impact on existing email configurations. We did this by pointing legacy sub-domains of the niner.net domain that named server NC027 — e.g., smtp27.niner.net — to server NC036. At the conclusion of the migration these sub-domains were indeed pointing to the new server. In other words, on Monday morning (4 June) email programs would have thought they were still downloading mail from the same server, not realising (or needing to realise) that they were in fact downloading from a new server.

However, it turned out that a significant minority of email programs were somehow misconfigured with settings that worked on the old server, but stopped working when connecting to the new server. Those clients who were using the correct settings experienced no disruption at all, and when those clients with incorrect settings corrected them on the morning of Monday the 11th, the problems were fixed instantly.

Over the rest of that week (11-15 June) we helped a few clients with some issues unique to how they use email, especially where those practices clashed with current best practices for email transmission. We also dealt with some issues of senders whose mail servers were behaving improperly, causing their emails to be blocked because they looked like spammers. This notably affected email from the ZRA, but their emails are once again flowing unimpeded.

We’re monitoring the spam filtering on the new server. Any message that the server identifies as spam will have the subject of the message prefixed to add “[SPAM]“. You can use this to configure your email program or the webmail to deal with spam automatically, by filtering it into your “junk” folder or deleting it entirely. We recommend filtering to the junk folder so that you can catch the occasional legitimate message that is misclassified as spam.

Finally, in recognition of the fact that the emergency migration of the server to a new data centre on 6 June disrupted all clients’ email, and the fact that those clients with misconfigured email programs experienced a week of disruption before the issue was identified, we will be applying a one-week (quarter month) credit to the accounts of all clients hosted on server NC036. We apologise for the difficulties caused, and will apply what was learned this time to future migrations.

Thank-you, as always, for your custom and patience.

NC036: Migration update 23 — SMTP AUTH is required for users under this sender domain

11 June 2018 09:38:23 +0000

There are two reasons why you may be getting the above error in response to messages you’ve sent to addresses on domains hosted by NinerNet, likely your own domain:

  • It may be because you’re sending from an address on a domain that we host, but instead of sending your email through our SMTP server (smtp.niner.net) you’re sending through another SMTP server, possibly that of an ISP or another email service provider. In some cases this can happen because of a situation similar to that described in the sixth bullet point of our post “NC036: Migration update 20 — Solutions“, where you’ve sent the email through a third party, perhaps an ISP, or an email account you have with another provider.
  • If you’re using some cloud-hosted application that tries to send email to you as you (or another user on your own domain), then that email looks like spam to the mail server, because lots of spammers mistakenly try to get their email through by sending their spam from your email address to your email address, or from another address on your own domain to you.

The solutions are, respectively (and respectfully):

  • Configure your email program to use smtp.niner.net to send email from any domain that we host. If you’re following the configuration instructions we send you, then that is the case by default, and always has been.
  • Have the provider of the cloud service send those emails from an address — even a “no-reply” address — on their own domain, or use SMTP AUTH to send the email through smtp.niner.net from an address on your own domain, just as you or any other human with an address on your domain would.

NC036: Migration update 22 — A word about forwarding email

11 June 2018 08:36:59 +0000

Over the years we’ve noticed that a certain percentage of our clients are in the habit of forwarding all of their email to external free webmail services — e.g., Yahoo, Hotmail, Gmail, etc. Why do we even notice this? Well, because these free services often delay your email, and so it queues on our server for anywhere between minutes and days. There are complicated reasons for this, but once you realise that you’re not the only one forwarding your email, you can see how these free webmail services might decide to limit the number of messages that they accept from our servers. This is especially noticeable when (not if) a few spams get through and (ironically) the receivers — the very NinerNet clients who have configured their email accounts here to forward to their free webmail provider — complain to the free webmail provider about the spam by clicking the convenient “this is spam” button. The free provider then responds by blocking or limiting mail from our server, making the reporting of the spam by the NinerNet client self-defeating!

Among other reasons, what people who do this are running into here is introducing multiple points of failure. If a message arrives on the NinerNet mail server, it’s made it! It has arrived where it was intended by the sender to be delivered. But now you’ve told our server to forward it somewhere else. It’s like telling a runner at the finish line that he has to do the same race again. And the runner might not make it the second time, just as your email might not make it into your Gmail account.

Right now there are a few dozen emails queued on our server waiting to be accepted by these free email services. Given that some of them have been queued for several days, most of them will likely bounce back to the senders within the next few hours. There is nothing unusual about this; we see it all the time, and it has little (if anything) to do with the mail server migration.

If webmail is your preferred way of accessing your email, we do (obviously) provide webmail on your own domain. (And non-Gmail webmail these days is way better than it used to be!) If you prefer the webmail offered by your free provider of choice, that’s fine, as long as you’re aware of the inherent risks of delayed and bounced email if you choose to forward everything.

If you’d like to discuss alternatives to forwarding your email, let us know and we can provide options to you or address any concerns you may have.

NC036: Migration update 14 — Microsoft blocks

6 June 2018 15:43:33 +0000

It seems that Microsoft blocks every IP address on the Internet by default, except those for which mail server administrators like NinerNet have to beg repeatedly to have removed. Our requests keep being ignored, despite the fact that we are members of both their Smart Network Data Service (SNDS) and their Junk Mail Reporting Program (JMRP), but we will keep trying.

Currently this means that we route Microsoft’s main domains — hotmail.com, outlook.com, msn.com and live.com — through our relay server which is not blacklisted as it pre-dates their aggressive blocking practices. However, if you send email to a non-Microsoft domain hosted by Outlook/Office365, you will almost certainly receive a bounce message that looks like this (if the domain you sent to hosted by Microsoft is “exampledomain.com”):

Remote-MTA: dns; exampledomain-com.mail.protection.outlook.com
Diagnostic-Code: smtp; 550 5.7.606 Access denied, banned sending IP
    [178.62.195.26]. To request removal from this list please visit
    https://sender.office.com/ and follow the directions. For more information
    please go to  http://go.microsoft.com/fwlink/?LinkID=526655 (AS16012609)

NC036: Migration update 7

4 June 2018 08:56:05 +0000

My apologies for not getting an update out sooner, but there have been some inevitable issues.

  • mail27.niner.net: First, if you have somehow been using the mail27.niner.net sub-domain in your email configuration, this was not a sub-domain intended to be used that way, and so was not pointed to the new server. However, we have reconsidered, and it is now pointing to the new server to assist those who apparently have it configured in their email.
  • Control panels: We are moving away from the unhelpful “cp” (control panel) sub-domains. The new mail control panel is located at mail.niner.net.
  • Webmail: This is now located at webmail.niner.net. We’ll be updating your domains with a “webmail” sub-domain to redirect to this in due course.
  • Control panel user names: If you used to log into the control panel with a user name that was your client code @mail27.niner.net, these no longer exist. We’ll be sending new log-in information within the next 24 hours. In the meantime, if you need to do anything in the control panel, please advise and we will assist.
  • Microsoft domains (Hotmail, Outlook, etc.): Despite the fact that we advised Microsoft through the proper channels that our new IP address is in their blacklist and their informing us that it would be removed, they are still blocking us. We are routing emails to their primary domains through our relay server to avoid emails being bounced, but it’s still possible that emails to these domains my end up in your correspondents’ spam folders.
  • Delays/losses: While the mail servers were down for approximately 7.5 hours — a lot longer than the four hours we optimistically forecast — incoming email to you sent while they were down will have been held on the sending server. Some of that backlog may take several hours to clear, depending on the policies of the senders’ mail servers, but the servers were down during an off-peak period anyway. Additionally, while we strive never to lose a single email during maintenance and migrations, we have seen several of our own emails to clients bounce. We will investigate and fix this issue.
  • Spam/viruses: On a more positive note, we’re already seeing this new server doing a much improved job of stopping spam and viruses.

Finally, one last apology for promising this migration three weeks ago and not actually delivering until now, and for not having all of the issues ironed out well in advance of Monday morning.

We’ll likely have additional information to post here later, but in the meantime you are welcome to contact us if you are having any issues not covered by the above.

NC027: Blacklist update

27 April 2018 12:53:35 +0000

Good news! NC027‘s IP address has been removed from the blacklist it was in, so at 12:41 UTC we switched mail processing back to the primary server. NC027 is in no blacklists of which we are aware.

Please note that the events of the last couple of days have pushed the notices we issued of upcoming scheduled maintenance on three of our servers down the page. If you have missed them please see:

Thank-you for your patience over the last 24 hours. If you have any questions or concerns, please do let us know.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: