Server and System Status

Over the weekend we successfully migrated all of the email accounts on old server NC027 to new server NC036. Except that for a large swathe of our clients, this migration was NOT a success.

We can tell you unequivocally that the new server is running and running well. It’s doing a much better job than the old server, and we splurged on a high-performance server with additional software to process mail quicker and to do a better job of filtering out spam and viruses. That part is all going great, and I can tell you that I am delighted about that part.

The part that’s not going great is that the data centre in which we placed the new server appears to have some serious networking issues for a large number of clients in southern and central Africa. We could spend the next week troubleshooting this and perhaps find the cause (and then work on addressing the cause), but you and we don’t have the luxury of that much time. Within the next few hours we will send troubleshooting instructions to affected clients, just in case.

What we are going to do to resolve this issue is use the wonders of modern technology to shut down the new server, take an image of it to preserve the time, effort and expense that has gone into creating it, and transfer it to new hardware in a different data centre. Ideally I would like to take a day to set up a test server in that data centre but, again, you can’t afford to have no or limited access to your email for a day.

Fortunately the process of moving to another data centre is quite straightforward, and will not require as much downtime as the full migration did. Copying the image from one data centre to another will be much like physically carrying the server to the new data centre; it’s already set up and configured, it just needs to be plugged in at the other end. The only thing that will not be quicker is that we have to use more traditional methods to transfer your mail spools to the new server. This took 50 minutes on the weekend, but with two equally powerful servers on both ends of the transfer it should be a bit quicker this time.

We will shut down the server at 01:00 UTC on Wednesday 6 June. Including the data transfer and some minor reconfiguration, I sincerely hope to have it back online by 03:00 UTC.

Please keep an eye on this status blog, where we will post important updates during the process.

Thank-you for your extraordinary patience.

Mail started flowing again at about 04:00 UTC. We will have some further details as time permits in the next couple of hours.

One apology: I’ve been using the wrong server number in the titles of the last few posts.

We are in the final stages of the migration to server NC036. We expect to have your email back online within the hour after finishing up testing.

After numerous delays — not the least of which is that Microsoft never did actually remove our new mail server’s IP address from their blacklist despite their saying that they had — we are finally going to migrate all email from NC027 to NC036. This will start within the next few minutes.

During the migration you will not have access to either the new or old mail servers, and you will not be able to send email. Incoming email will be held on the sending servers until the new server is live and everything is pointing to it.

As our own email on our primacy domain (niner.net) will also be down during the migration, please email us at support.201806@ninernet.net if you have any urgent queries. Note the similar but different domain.

Thank-you.

We have, unfortunately, decided to postpone this migration yet again. For this we sincerely apologise.

The good news is that both of the issues identified last weekend were resolved. The bad news is that, while this migration will be quicker (once it gets going) with fewer complications than past migrations because of the fact that the new mail server runs newer versions of the same software, doing the migration this way actually introduces it’s own quirks and set of challenges. We really hope to address those over the coming week.

In the meantime, your email is operating as normal.

If you have any questions or concerns, please do let us know. Thank-you again for your patience.

We have decided to postpone the migration of server NC027 to NC036 until our regular maintenance window next weekend, 19:00 UTC Saturday (19 May) to 19:00 UTC Sunday. Of course, the move won’t take 24 hours, but that is the full length of the weekend maintenance window. Through testing of the migration process a few times on a test server and optimising the process of getting millions of email messages safely from one server to another, intact — this much data required a re-evaluation of our normal procedures — we have managed to shave at least six hours off of the planned migration time.

We therefore estimate that the migration will take about four hours.

There are a couple of reasons for the delay:

• One thing we haven’t got working properly yet is an anti-spam feature known as DKIM (DomainKeys Identified Mail). Because this technique involves use of the DNS (domain name system), it takes time for the authentication information published in the DNS to propagate, and if the wrong information is present (as is currently the case) all of your outbound email will be stopped in its tracks. This is, shall we say, undesirable.
• Secondly, we have found out that the IP address of the new mail server is in the blacklist of a major email provider. Good thing we checked before going live! We’re at the mercy of said big corporation as far as the timing of the removal, and it’s optimistic to think that it will be removed in the next few hours. However, it’s reasonable to assume that they will remove the block of our new IP address, and that they will do so by next weekend.

For now, everything stays and keeps working exactly the same. Do not adjust your television sets … or your email programs, for that matter.

Having worked with the new mail system all week, we are pretty excited about getting it online due to some new features and some excellent spam-fighting tools. We know you’ll love it too; thanks for your patience.

The migration later today of all mail from server NC027 to NC036 should take at least six hours, possibly more. For this reason we will try to start it a bit later than previously stated.

Server NC027 is being replaced by new mail server NC036. We will be migrating all email accounts to the new server during our regular weekly maintenance window next weekend, starting at 21:00 UTC on Saturday 12 May 2018.

We will be conducting tests over the next week to determine how long this should take, and will post an update here by the middle of next week with an estimate for how long your mail will be unavailable. Because this is a migration between servers running the same software, it’s much more straightforward that most migrations and should take less time than usual. Additionally, the configuration settings for your email will not change, making the process much smoother.

As mentioned last week, this is part of a two-phase plan — this is phase two — to significantly reduce downtime during migrations. Future migrations should be done in almost minutes rather than hours.

If you have any questions or concerns, please feel free to contact support. Thank-you for your patience.