NinerNet Communications™
System Status

Server and System Status

NC036: Migration update 16

7 June 2018 05:43:46 +0000

Are you wondering if our mail server is really up or if we’re “having problems”? We could be lying, but this third-party service will uncover our lies:

https://downforeveryoneorjustme.com

Every time we check, mail.niner.net and webmail.niner.net are up. Please check for yourself. In fact, we suggest contacting your ISP and asking them why you cannot reach a server that is alive and well.

You can also check the pop, imap and smtp sub-domains of niner.net, as well as the old pop27, imap27 and smtp27 sub-domains, all of which are working.

We actually do strongly urge you to contact your ISP about the fact that you can only intermittently connect to our mail server. They are the only ones who can help you with your connection to the Internet when it is not working properly.

NC036: Migration update 15 — Possible return of connection issues

6 June 2018 16:35:16 +0000

As happened on Monday, after initial success with setting up the new mail server, we’re again receiving reports from clients in Zambia (so far) that are starting to have their connections to the server dropped by their ISPs. This is incredibly frustrating, certainly for you, of course, but also for us.

If this is happening to you, you can help us help you by submitting what’s called an MTR report. MTR is a network diagnostic tool that gives us quantifiable information we can analyse and (hopefully) act on, showing where the network problem affecting your email lies so that we may possibly contact the organisation responsible for the problem.

If you’re running Windows, please download WinMTR. The file you will download is just a zip file that contains the actual program (32- and 64-bit versions). Read the “README.TXT” file for instructions on which version to use. All you need to do is double-click the appropriate file and the program runs; it’s not actually installed on your computer. Then type nc036.ninernet.net (yes, that’s ninernet.net, NOT niner.net) in the box as prompted, and select the start option to begin generating report data. Once you’ve gathered enough data, please use the copy or export functions to send us the report.

Please also send us the following information:

  • Your IP address.
  • The name of your ISP. If you have access to multiple connections, an MTR report from each ISP would be greatly appreciated.
  • If you have a reliable contact at that ISP, his/her name, position, phone number and email address.

Please email this to our usual email address, or send it via our contact form.

Thanks.

NC036: Migration update 13

6 June 2018 12:52:23 +0000

We will post a postmortem here in due course, hopefully with 24-48 hours, along with a thousand more apologies, but we are looking for feedback to ensure that all clients are able to connect to the server and download and send email, as this was not the case on Monday and Tuesday.

NC036: Migration update 12 — server back online

6 June 2018 12:28:04 +0000

The transfer of the mail spools has completed and server NC036 was brought back online at 12:12 UTC.

NC036: Migration update 12

6 June 2018 08:32:10 +0000

The transfer of mail data between the old and new data centres is still underway. Considering the transfer of the same amount of data took 50 minutes over the weekend, and we are now at the 5-hour mark, it was impossible for us to predict that this would take so long.

I can assure you that I understand the frustration that you are feeling with this situation, but given the network problems in southern and central Africa that necessitated this emergency move, we had no choice but to act immediately rather than forcing many clients to do without mail until the weekend.

My best estimate at this point, based on how much data has transferred so far (55%) and how much is left, is that the transfer will complete at approximately 13:00 UTC. Assuming this is the case, the server will be back online and accessible at about 13:30 UTC.

I sincerely and deeply apologise for this situation.

NC036: Migration update 11

6 June 2018 04:42:04 +0000

The transfer is taking significantly longer than we anticipated, likely due to the greater distance between the two data centres.

NC036: Migration update 10

6 June 2018 03:36:47 +0000

We’re almost done. Just waiting for the mail spools to finish transferring, then a few checks (double-checks) to ensure that all is in order, and we’ll re-enable all services. Then more checks to ensure that mail is flowing as it should, and then we await feedback.

NC036: Migration update 9

6 June 2018 01:22:20 +0000

We’ve finished planning this emergency migration of the new mail server (NC036), and will be shutting it down within the next five minutes.

NC036: Migration update 8 — Plan B

5 June 2018 23:32:15 +0000

Over the weekend we successfully migrated all of the email accounts on old server NC027 to new server NC036. Except that for a large swathe of our clients, this migration was NOT a success.

We can tell you unequivocally that the new server is running and running well. It’s doing a much better job than the old server, and we splurged on a high-performance server with additional software to process mail quicker and to do a better job of filtering out spam and viruses. That part is all going great, and I can tell you that I am delighted about that part.

The part that’s not going great is that the data centre in which we placed the new server appears to have some serious networking issues for a large number of clients in southern and central Africa. We could spend the next week troubleshooting this and perhaps find the cause (and then work on addressing the cause), but you and we don’t have the luxury of that much time. Within the next few hours we will send troubleshooting instructions to affected clients, just in case.

What we are going to do to resolve this issue is use the wonders of modern technology to shut down the new server, take an image of it to preserve the time, effort and expense that has gone into creating it, and transfer it to new hardware in a different data centre. Ideally I would like to take a day to set up a test server in that data centre but, again, you can’t afford to have no or limited access to your email for a day.

Fortunately the process of moving to another data centre is quite straightforward, and will not require as much downtime as the full migration did. Copying the image from one data centre to another will be much like physically carrying the server to the new data centre; it’s already set up and configured, it just needs to be plugged in at the other end. The only thing that will not be quicker is that we have to use more traditional methods to transfer your mail spools to the new server. This took 50 minutes on the weekend, but with two equally powerful servers on both ends of the transfer it should be a bit quicker this time.

We will shut down the server at 01:00 UTC on Wednesday 6 June. Including the data transfer and some minor reconfiguration, I sincerely hope to have it back online by 03:00 UTC.

Please keep an eye on this status blog, where we will post important updates during the process.

Thank-you for your extraordinary patience.

NC036: Migration update 7

4 June 2018 08:56:05 +0000

My apologies for not getting an update out sooner, but there have been some inevitable issues.

  • mail27.niner.net: First, if you have somehow been using the mail27.niner.net sub-domain in your email configuration, this was not a sub-domain intended to be used that way, and so was not pointed to the new server. However, we have reconsidered, and it is now pointing to the new server to assist those who apparently have it configured in their email.
  • Control panels: We are moving away from the unhelpful “cp” (control panel) sub-domains. The new mail control panel is located at mail.niner.net.
  • Webmail: This is now located at webmail.niner.net. We’ll be updating your domains with a “webmail” sub-domain to redirect to this in due course.
  • Control panel user names: If you used to log into the control panel with a user name that was your client code @mail27.niner.net, these no longer exist. We’ll be sending new log-in information within the next 24 hours. In the meantime, if you need to do anything in the control panel, please advise and we will assist.
  • Microsoft domains (Hotmail, Outlook, etc.): Despite the fact that we advised Microsoft through the proper channels that our new IP address is in their blacklist and their informing us that it would be removed, they are still blocking us. We are routing emails to their primary domains through our relay server to avoid emails being bounced, but it’s still possible that emails to these domains my end up in your correspondents’ spam folders.
  • Delays/losses: While the mail servers were down for approximately 7.5 hours — a lot longer than the four hours we optimistically forecast — incoming email to you sent while they were down will have been held on the sending server. Some of that backlog may take several hours to clear, depending on the policies of the senders’ mail servers, but the servers were down during an off-peak period anyway. Additionally, while we strive never to lose a single email during maintenance and migrations, we have seen several of our own emails to clients bounce. We will investigate and fix this issue.
  • Spam/viruses: On a more positive note, we’re already seeing this new server doing a much improved job of stopping spam and viruses.

Finally, one last apology for promising this migration three weeks ago and not actually delivering until now, and for not having all of the issues ironed out well in advance of Monday morning.

We’ll likely have additional information to post here later, but in the meantime you are welcome to contact us if you are having any issues not covered by the above.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC020, Chicago, United States of America, OPERATIONAL.NC020OperationalUp?
Server NC023, London, United Kingdom, OPERATIONAL.NC023OperationalUp?
Server NC028, Vancouver, Canada, INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America, OPERATIONAL.NC031OperationalUp?
Server NC033, Toronto, Canada, OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia, INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia, TESTING.NC035TestingUp?
Server NC036, Amsterdam, Netherlands, OPERATIONAL.NC036OperationalUp?
Server NC037, Seattle, United States of America, INTERNAL.NC037InternalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

General Information:

This blog provides information about the status of NinerNet Communications systems. Dates and times of posts to this blog are in the UTC time zone, and dates and times given for events are also in the UTC time zone, although conversions may be offered for some time zones common to our clients. Please use the World Time Server to ensure accurate conversion of dates and times to your own time zone.

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings configuration connection issues control panel database dns dos attack dot-zm domains down time email delivery error messages ftp hardware imap mailing lists mail relay mail server microsoft migration nameservers network networking outlook performance phplist pop power failure reboot smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server wordpress

Resources:

On NinerNet: