NinerNet Communications™
System Status

Server and System Status

NC036: Migration update 25 — Final

18 June 2018 08:54:43 +0000

The migration of all email accounts from server NC027 to server NC036 is complete. In fact, it was successfully completed at 04:00 UTC on 4 June. What followed over the next few days was an unprecedented avalanche of misinformation and red herrings that resulted in our moving the new server to another data centre (a move that took ten times longer than the previous move from the data centre where NC027 was located) where the same “problems” experienced by only some of our clients magically reappeared.

We planned the migration to have absolutely no impact on existing email configurations. We did this by pointing legacy sub-domains of the niner.net domain that named server NC027 — e.g., smtp27.niner.net — to server NC036. At the conclusion of the migration these sub-domains were indeed pointing to the new server. In other words, on Monday morning (4 June) email programs would have thought they were still downloading mail from the same server, not realising (or needing to realise) that they were in fact downloading from a new server.

However, it turned out that a significant minority of email programs were somehow misconfigured with settings that worked on the old server, but stopped working when connecting to the new server. Those clients who were using the correct settings experienced no disruption at all, and when those clients with incorrect settings corrected them on the morning of Monday the 11th, the problems were fixed instantly.

Over the rest of that week (11-15 June) we helped a few clients with some issues unique to how they use email, especially where those practices clashed with current best practices for email transmission. We also dealt with some issues of senders whose mail servers were behaving improperly, causing their emails to be blocked because they looked like spammers. This notably affected email from the ZRA, but their emails are once again flowing unimpeded.

We’re monitoring the spam filtering on the new server. Any message that the server identifies as spam will have the subject of the message prefixed to add “[SPAM]“. You can use this to configure your email program or the webmail to deal with spam automatically, by filtering it into your “junk” folder or deleting it entirely. We recommend filtering to the junk folder so that you can catch the occasional legitimate message that is misclassified as spam.

Finally, in recognition of the fact that the emergency migration of the server to a new data centre on 6 June disrupted all clients’ email, and the fact that those clients with misconfigured email programs experienced a week of disruption before the issue was identified, we will be applying a one-week (quarter month) credit to the accounts of all clients hosted on server NC036. We apologise for the difficulties caused, and will apply what was learned this time to future migrations.

Thank-you, as always, for your custom and patience.

NC036: Migration update 23 — SMTP AUTH is required for users under this sender domain

11 June 2018 09:38:23 +0000

There are two reasons why you may be getting the above error in response to messages you’ve sent to addresses on domains hosted by NinerNet, likely your own domain:

  • It may be because you’re sending from an address on a domain that we host, but instead of sending your email through our SMTP server (smtp.niner.net) you’re sending through another SMTP server, possibly that of an ISP or another email service provider. In some cases this can happen because of a situation similar to that described in the sixth bullet point of our post “NC036: Migration update 20 — Solutions“, where you’ve sent the email through a third party, perhaps an ISP, or an email account you have with another provider.
  • If you’re using some cloud-hosted application that tries to send email to you as you (or another user on your own domain), then that email looks like spam to the mail server, because lots of spammers mistakenly try to get their email through by sending their spam from your email address to your email address, or from another address on your own domain to you.

The solutions are, respectively (and respectfully):

  • Configure your email program to use smtp.niner.net to send email from any domain that we host. If you’re following the configuration instructions we send you, then that is the case by default, and always has been.
  • Have the provider of the cloud service send those emails from an address — even a “no-reply” address — on their own domain, or use SMTP AUTH to send the email through smtp.niner.net from an address on your own domain, just as you or any other human with an address on your domain would.

NC036: Migration update 22 — A word about forwarding email

11 June 2018 08:36:59 +0000

Over the years we’ve noticed that a certain percentage of our clients are in the habit of forwarding all of their email to external free webmail services — e.g., Yahoo, Hotmail, Gmail, etc. Why do we even notice this? Well, because these free services often delay your email, and so it queues on our server for anywhere between minutes and days. There are complicated reasons for this, but once you realise that you’re not the only one forwarding your email, you can see how these free webmail services might decide to limit the number of messages that they accept from our servers. This is especially noticeable when (not if) a few spams get through and (ironically) the receivers — the very NinerNet clients who have configured their email accounts here to forward to their free webmail provider — complain to the free webmail provider about the spam by clicking the convenient “this is spam” button. The free provider then responds by blocking or limiting mail from our server, making the reporting of the spam by the NinerNet client self-defeating!

Among other reasons, what people who do this are running into here is introducing multiple points of failure. If a message arrives on the NinerNet mail server, it’s made it! It has arrived where it was intended by the sender to be delivered. But now you’ve told our server to forward it somewhere else. It’s like telling a runner at the finish line that he has to do the same race again. And the runner might not make it the second time, just as your email might not make it into your Gmail account.

Right now there are a few dozen emails queued on our server waiting to be accepted by these free email services. Given that some of them have been queued for several days, most of them will likely bounce back to the senders within the next few hours. There is nothing unusual about this; we see it all the time, and it has little (if anything) to do with the mail server migration.

If webmail is your preferred way of accessing your email, we do (obviously) provide webmail on your own domain. (And non-Gmail webmail these days is way better than it used to be!) If you prefer the webmail offered by your free provider of choice, that’s fine, as long as you’re aware of the inherent risks of delayed and bounced email if you choose to forward everything.

If you’d like to discuss alternatives to forwarding your email, let us know and we can provide options to you or address any concerns you may have.

NC036: Migration update 14 — Microsoft blocks

6 June 2018 15:43:33 +0000

It seems that Microsoft blocks every IP address on the Internet by default, except those for which mail server administrators like NinerNet have to beg repeatedly to have removed. Our requests keep being ignored, despite the fact that we are members of both their Smart Network Data Service (SNDS) and their Junk Mail Reporting Program (JMRP), but we will keep trying.

Currently this means that we route Microsoft’s main domains — hotmail.com, outlook.com, msn.com and live.com — through our relay server which is not blacklisted as it pre-dates their aggressive blocking practices. However, if you send email to a non-Microsoft domain hosted by Outlook/Office365, you will almost certainly receive a bounce message that looks like this (if the domain you sent to hosted by Microsoft is “exampledomain.com”):

Remote-MTA: dns; exampledomain-com.mail.protection.outlook.com
Diagnostic-Code: smtp; 550 5.7.606 Access denied, banned sending IP
    [178.62.195.26]. To request removal from this list please visit
    https://sender.office.com/ and follow the directions. For more information
    please go to  http://go.microsoft.com/fwlink/?LinkID=526655 (AS16012609)

NC036: Migration update 13

6 June 2018 12:52:23 +0000

We will post a postmortem here in due course, hopefully with 24-48 hours, along with a thousand more apologies, but we are looking for feedback to ensure that all clients are able to connect to the server and download and send email, as this was not the case on Monday and Tuesday.

NC036: Migration update 12 — server back online

6 June 2018 12:28:04 +0000

The transfer of the mail spools has completed and server NC036 was brought back online at 12:12 UTC.

NC036: Migration update 12

6 June 2018 08:32:10 +0000

The transfer of mail data between the old and new data centres is still underway. Considering the transfer of the same amount of data took 50 minutes over the weekend, and we are now at the 5-hour mark, it was impossible for us to predict that this would take so long.

I can assure you that I understand the frustration that you are feeling with this situation, but given the network problems in southern and central Africa that necessitated this emergency move, we had no choice but to act immediately rather than forcing many clients to do without mail until the weekend.

My best estimate at this point, based on how much data has transferred so far (55%) and how much is left, is that the transfer will complete at approximately 13:00 UTC. Assuming this is the case, the server will be back online and accessible at about 13:30 UTC.

I sincerely and deeply apologise for this situation.

NC036: Migration update 11

6 June 2018 04:42:04 +0000

The transfer is taking significantly longer than we anticipated, likely due to the greater distance between the two data centres.

NC036: Migration update 10

6 June 2018 03:36:47 +0000

We’re almost done. Just waiting for the mail spools to finish transferring, then a few checks (double-checks) to ensure that all is in order, and we’ll re-enable all services. Then more checks to ensure that mail is flowing as it should, and then we await feedback.

NC036: Migration update 9

6 June 2018 01:22:20 +0000

We’ve finished planning this emergency migration of the new mail server (NC036), and will be shutting it down within the next five minutes.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: