We are in the final stages of the migration to server NC036. We expect to have your email back online within the hour after finishing up testing.
We are in the final stages of the migration to server NC036. We expect to have your email back online within the hour after finishing up testing.
After numerous delays — not the least of which is that Microsoft never did actually remove our new mail server’s IP address from their blacklist despite their saying that they had — we are finally going to migrate all email from NC027 to NC036. This will start within the next few minutes.
During the migration you will not have access to either the new or old mail servers, and you will not be able to send email. Incoming email will be held on the sending servers until the new server is live and everything is pointing to it.
As our own email on our primacy domain (niner.net) will also be down during the migration, please email us at support.201806@ninernet.net if you have any urgent queries. Note the similar but different domain.
Thank-you.
We have, unfortunately, decided to postpone this migration yet again. For this we sincerely apologise.
The good news is that both of the issues identified last weekend were resolved. The bad news is that, while this migration will be quicker (once it gets going) with fewer complications than past migrations because of the fact that the new mail server runs newer versions of the same software, doing the migration this way actually introduces it’s own quirks and set of challenges. We really hope to address those over the coming week.
In the meantime, your email is operating as normal.
If you have any questions or concerns, please do let us know. Thank-you again for your patience.
We have decided to postpone the migration of server NC027 to NC036 until our regular maintenance window next weekend, 19:00 UTC Saturday (19 May) to 19:00 UTC Sunday. Of course, the move won’t take 24 hours, but that is the full length of the weekend maintenance window. Through testing of the migration process a few times on a test server and optimising the process of getting millions of email messages safely from one server to another, intact — this much data required a re-evaluation of our normal procedures — we have managed to shave at least six hours off of the planned migration time.
We therefore estimate that the migration will take about four hours.
There are a couple of reasons for the delay:
For now, everything stays and keeps working exactly the same. Do not adjust your television sets … or your email programs, for that matter.
Having worked with the new mail system all week, we are pretty excited about getting it online due to some new features and some excellent spam-fighting tools. We know you’ll love it too; thanks for your patience.
The migration later today of all mail from server NC027 to NC036 should take at least six hours, possibly more. For this reason we will try to start it a bit later than previously stated.
Server NC027 is being replaced by new mail server NC036. We will be migrating all email accounts to the new server during our regular weekly maintenance window next weekend, starting at 21:00 UTC on Saturday 12 May 2018.
We will be conducting tests over the next week to determine how long this should take, and will post an update here by the middle of next week with an estimate for how long your mail will be unavailable. Because this is a migration between servers running the same software, it’s much more straightforward that most migrations and should take less time than usual. Additionally, the configuration settings for your email will not change, making the process much smoother.
As mentioned last week, this is part of a two-phase plan — this is phase two — to significantly reduce downtime during migrations. Future migrations should be done in almost minutes rather than hours.
If you have any questions or concerns, please feel free to contact support. Thank-you for your patience.
The maintenance on server NC027 is complete, and it was brought back online at 22:17 UTC. Thank-you for your patience.
While we did this maintenance we also patched the server for the Spectre vulnerability. This means that the maintenance to do that on 1 May is no longer necessary, and has been cancelled.
If you have any questions or concerns, please contact support.
Server NC027 went down for maintenance at 21:05 UTC. We will post updates here as the work progresses and when the server is back online again.
Good news! NC027‘s IP address has been removed from the blacklist it was in, so at 12:41 UTC we switched mail processing back to the primary server. NC027 is in no blacklists of which we are aware.
Please note that the events of the last couple of days have pushed the notices we issued of upcoming scheduled maintenance on three of our servers down the page. If you have missed them please see:
Thank-you for your patience over the last 24 hours. If you have any questions or concerns, please do let us know.
As explained yesterday, a client’s compromised email account sent out thousands of spam emails before it was detected and stopped. This has happened before, but the circumstances this time are different.
Most blacklists are automated, both in adding IP addresses to the blacklist and in removing them. This is a double-edged sword. On the plus side, IP addresses that are the source of spam are quickly added, making it less likely that spam will get through in subsequent attempts from the same IP address. Most, if not all, automated blacklists then remove the bad IP address fairly quickly after the spam stops. They realise that stuff happens, and when the spam stops they assume the problem is fixed and remove the IP address. There is short-term pain, but it’s measured in hours and the block is generally removed within your business day.
On the negative side, organisations and people that run blacklists are generally unwilling to manually remove IP addresses before they automatically expire. In and of itself this isn’t actually a bad thing; many blacklist wouldn’t be able to function if they had to field pleas and demands that IP addresses be removed. Quick, automated removal when the problem that caused the listing in the first place is fixed is the cure.
Unfortunately this situation has exposed a blacklist that actually seems to be designed to punish mail servers that have had a temporary problem, even after the problem has been stopped. This is unusual in our experience, as it makes the blacklist less useful, by blocking legitimate email long after the problem has been addressed. Information on their website states that it could be “a week or more” before an IP address is removedif they determine the spam outbreak to be severe enough — without defining “severe” — even though it has stopped. And since the addition and removal of IP addresses is automated, “you cannot” get your IP address removed manually. Full stop.
Since this blacklist is still blocking our mail server’s IP address almost 24 hours later, at 05:01 UTC we started relaying all mail sent by clients through our relay mail server (NC023), which has a different IP address. We will continue to monitor the blacklist in question and reverse this once our IP address is removed.
It seems that most of the mail servers we’ve seen using this blacklist are in South Africa. Mail bounced using this blacklist will show a message like the following, using real email addresses, domains and IP addresses of course:
<destination@example.com>: host something.co.za[1.2.3.4] said:
550-rejected because 212.71.255.195 is in a black list at
truncate.gbudb.net 550 http://www.gbudb.com/truncate/ [212.71.255.195]
(in reply to RCPT TO command)
If you’ve seen this, we suggest that you contact the person to whom you sent the email and suggest that they tell their hosting company that they should stop using blacklists that don’t operate within the norms of most blacklists. Feel free to point them to this blog post.
With all of the above said, we will be setting up a new mail server and migrating all accounts to it within the next couple of weeks. The new server will be better equipped to spot and stop these outbreaks automatically before they become “severe”.
As always, we appreciate your patience, and we also appreciate those clients that keep their anti-virus software up to date. If you have any questions, please feel free to contact us. Thank-you.
Systems at a Glance:
| Loc. | System | Status | Ping |
|---|---|---|---|
| NC023 | Internal | Up? | |
| NC028 | Internal | Up? | |
| NC031 | Internal | Up? | |
| NC033 | Operational | Up? | |
| NC034 | Internal | Up? | |
| NC035 | Operational | Up? | |
| NC036 | Operational | Up? | |
| NC040 | Internal | Up? | |
| NC041 | Operational | Up? | |
| NC042 | Operational | Up? |
Subscriptions:
Search:
Recent Posts:
Archives:
Categories:
Links
Tags:
Resources:
On NinerNet: