NinerNet Communications™
System Status

Server and System Status

NC036: Migration update 25 — Final

18 June 2018 08:54:43 +0000

The migration of all email accounts from server NC027 to server NC036 is complete. In fact, it was successfully completed at 04:00 UTC on 4 June. What followed over the next few days was an unprecedented avalanche of misinformation and red herrings that resulted in our moving the new server to another data centre (a move that took ten times longer than the previous move from the data centre where NC027 was located) where the same “problems” experienced by only some of our clients magically reappeared.

We planned the migration to have absolutely no impact on existing email configurations. We did this by pointing legacy sub-domains of the niner.net domain that named server NC027 — e.g., smtp27.niner.net — to server NC036. At the conclusion of the migration these sub-domains were indeed pointing to the new server. In other words, on Monday morning (4 June) email programs would have thought they were still downloading mail from the same server, not realising (or needing to realise) that they were in fact downloading from a new server.

However, it turned out that a significant minority of email programs were somehow misconfigured with settings that worked on the old server, but stopped working when connecting to the new server. Those clients who were using the correct settings experienced no disruption at all, and when those clients with incorrect settings corrected them on the morning of Monday the 11th, the problems were fixed instantly.

Over the rest of that week (11-15 June) we helped a few clients with some issues unique to how they use email, especially where those practices clashed with current best practices for email transmission. We also dealt with some issues of senders whose mail servers were behaving improperly, causing their emails to be blocked because they looked like spammers. This notably affected email from the ZRA, but their emails are once again flowing unimpeded.

We’re monitoring the spam filtering on the new server. Any message that the server identifies as spam will have the subject of the message prefixed to add “[SPAM]“. You can use this to configure your email program or the webmail to deal with spam automatically, by filtering it into your “junk” folder or deleting it entirely. We recommend filtering to the junk folder so that you can catch the occasional legitimate message that is misclassified as spam.

Finally, in recognition of the fact that the emergency migration of the server to a new data centre on 6 June disrupted all clients’ email, and the fact that those clients with misconfigured email programs experienced a week of disruption before the issue was identified, we will be applying a one-week (quarter month) credit to the accounts of all clients hosted on server NC036. We apologise for the difficulties caused, and will apply what was learned this time to future migrations.

Thank-you, as always, for your custom and patience.

NC036: Migration update 7

4 June 2018 08:56:05 +0000

My apologies for not getting an update out sooner, but there have been some inevitable issues.

  • mail27.niner.net: First, if you have somehow been using the mail27.niner.net sub-domain in your email configuration, this was not a sub-domain intended to be used that way, and so was not pointed to the new server. However, we have reconsidered, and it is now pointing to the new server to assist those who apparently have it configured in their email.
  • Control panels: We are moving away from the unhelpful “cp” (control panel) sub-domains. The new mail control panel is located at mail.niner.net.
  • Webmail: This is now located at webmail.niner.net. We’ll be updating your domains with a “webmail” sub-domain to redirect to this in due course.
  • Control panel user names: If you used to log into the control panel with a user name that was your client code @mail27.niner.net, these no longer exist. We’ll be sending new log-in information within the next 24 hours. In the meantime, if you need to do anything in the control panel, please advise and we will assist.
  • Microsoft domains (Hotmail, Outlook, etc.): Despite the fact that we advised Microsoft through the proper channels that our new IP address is in their blacklist and their informing us that it would be removed, they are still blocking us. We are routing emails to their primary domains through our relay server to avoid emails being bounced, but it’s still possible that emails to these domains my end up in your correspondents’ spam folders.
  • Delays/losses: While the mail servers were down for approximately 7.5 hours — a lot longer than the four hours we optimistically forecast — incoming email to you sent while they were down will have been held on the sending server. Some of that backlog may take several hours to clear, depending on the policies of the senders’ mail servers, but the servers were down during an off-peak period anyway. Additionally, while we strive never to lose a single email during maintenance and migrations, we have seen several of our own emails to clients bounce. We will investigate and fix this issue.
  • Spam/viruses: On a more positive note, we’re already seeing this new server doing a much improved job of stopping spam and viruses.

Finally, one last apology for promising this migration three weeks ago and not actually delivering until now, and for not having all of the issues ironed out well in advance of Monday morning.

We’ll likely have additional information to post here later, but in the meantime you are welcome to contact us if you are having any issues not covered by the above.

NC027: Migration update 6

4 June 2018 04:25:43 +0000

Mail started flowing again at about 04:00 UTC. We will have some further details as time permits in the next couple of hours.

One apology: I’ve been using the wrong server number in the titles of the last few posts.

NC027: Migration update 5

4 June 2018 03:08:54 +0000

We are in the final stages of the migration to server NC036. We expect to have your email back online within the hour after finishing up testing.

NC027: Migration update 4 — we’re a go for launch!

3 June 2018 20:26:14 +0000

After numerous delays — not the least of which is that Microsoft never did actually remove our new mail server’s IP address from their blacklist despite their saying that they had — we are finally going to migrate all email from NC027 to NC036. This will start within the next few minutes.

During the migration you will not have access to either the new or old mail servers, and you will not be able to send email. Incoming email will be held on the sending servers until the new server is live and everything is pointing to it.

As our own email on our primacy domain (niner.net) will also be down during the migration, please email us at support.201806@ninernet.net if you have any urgent queries. Note the similar but different domain.

Thank-you.

NC027: Migration update 3 — postponed again

21 May 2018 01:19:39 +0000

We have, unfortunately, decided to postpone this migration yet again. For this we sincerely apologise.

The good news is that both of the issues identified last weekend were resolved. The bad news is that, while this migration will be quicker (once it gets going) with fewer complications than past migrations because of the fact that the new mail server runs newer versions of the same software, doing the migration this way actually introduces it’s own quirks and set of challenges. We really hope to address those over the coming week.

In the meantime, your email is operating as normal.

If you have any questions or concerns, please do let us know. Thank-you again for your patience.

NC027: Migration update 2 — postponed

13 May 2018 11:03:30 +0000

We have decided to postpone the migration of server NC027 to NC036 until our regular maintenance window next weekend, 19:00 UTC Saturday (19 May) to 19:00 UTC Sunday. Of course, the move won’t take 24 hours, but that is the full length of the weekend maintenance window. Through testing of the migration process a few times on a test server and optimising the process of getting millions of email messages safely from one server to another, intact — this much data required a re-evaluation of our normal procedures — we have managed to shave at least six hours off of the planned migration time.

We therefore estimate that the migration will take about four hours.

There are a couple of reasons for the delay:

  • One thing we haven’t got working properly yet is an anti-spam feature known as DKIM (DomainKeys Identified Mail). Because this technique involves use of the DNS (domain name system), it takes time for the authentication information published in the DNS to propagate, and if the wrong information is present (as is currently the case) all of your outbound email will be stopped in its tracks. This is, shall we say, undesirable.
  • Secondly, we have found out that the IP address of the new mail server is in the blacklist of a major email provider. Good thing we checked before going live! We’re at the mercy of said big corporation as far as the timing of the removal, and it’s optimistic to think that it will be removed in the next few hours. However, it’s reasonable to assume that they will remove the block of our new IP address, and that they will do so by next weekend.

For now, everything stays and keeps working exactly the same. Do not adjust your television sets … or your email programs, for that matter.

Having worked with the new mail system all week, we are pretty excited about getting it online due to some new features and some excellent spam-fighting tools. We know you’ll love it too; thanks for your patience.

NC027: Migration update 1

12 May 2018 12:47:05 +0000

The migration later today of all mail from server NC027 to NC036 should take at least six hours, possibly more. For this reason we will try to start it a bit later than previously stated.

NC027: Migration of mail server

4 May 2018 17:17:14 +0000

Server NC027 is being replaced by new mail server NC036. We will be migrating all email accounts to the new server during our regular weekly maintenance window next weekend, starting at 21:00 UTC on Saturday 12 May 2018.

We will be conducting tests over the next week to determine how long this should take, and will post an update here by the middle of next week with an estimate for how long your mail will be unavailable. Because this is a migration between servers running the same software, it’s much more straightforward that most migrations and should take less time than usual. Additionally, the configuration settings for your email will not change, making the process much smoother.

As mentioned last week, this is part of a two-phase plan — this is phase two — to significantly reduce downtime during migrations. Future migrations should be done in almost minutes rather than hours.

If you have any questions or concerns, please feel free to contact support. Thank-you for your patience.

NC027: Maintenance complete

28 April 2018 22:53:20 +0000

The maintenance on server NC027 is complete, and it was brought back online at 22:17 UTC. Thank-you for your patience.

While we did this maintenance we also patched the server for the Spectre vulnerability. This means that the maintenance to do that on 1 May is no longer necessary, and has been cancelled.

If you have any questions or concerns, please contact support.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: