NinerNet Communications™
System Status

Server and System Status

Upgrade of NC031 to NC041: Update 4

18 November 2019 01:42:48 +0000

On Friday night / Saturday morning we completed the migration all websites from server NC031 to NC041. There are a couple of minor improvements for you:

  • We have doubled the disk space available to you from 10 GB to 20 GB, and
  • All websites that didn’t previously have an SSL certificate now have one.

In order to take advantage of encrypted connections to your website (and therefore see a green padlock in your browser) you will need to access it at https://yourdomain.com. All of your web pages will also need to call included items — such as images, style sheets, JavaScript files, etc. — over HTTPS connections as well. If they aren’t, a web browser will only show some sort of insecure or semi-secure indication — e.g., a non-green padlock with some sort of alert icon. If you need help with this, please consult your website manager.

All websites that can use it are now running version 7 of PHP. This is required by the current versions of some web apps. If your website is quite old, especially if it accesses a MySQL database, it made not be coded to be able to use PHP 7. If this is the case for your website it is running under PHP 5. However, version 5 will not be available forever, so your website will eventually stop working. In order to fix this you will need to engage someone to update the code for your website, or create a new one. We suggest that you consider doing this as soon as possible, but definitely within the next year.

The control panel for the new web server continues to be available at web.niner.net:10000. Some websites are still available on the old server; if you need access to the old control panel within the next 48 hours, please let us know. After that it will no longer be available.

All web apps that are installed through the control panel have been updated to their latest versions. If your web app — e.g., WordPress, os Commerce, Moodle, Mediawiki, etc. — is not installed through the control panel, please take steps to upgrade it as soon as possible to reduce your website’s exposure to being hacked.

Finally, if you have a contact form or some other process on your website that sends email to you or your users, please be aware that this server is on a “new” IP address that is relatively unknown to the world’s mail servers and blacklists. As a result we’re seeing that some outbound emails are being delayed. We strongly recommend that you configure such features to send email to a domain hosted by NinerNet rather than an external domain (especially gmail.com).

If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience and ongoing patronage.

Upgrade of NC031 to NC041: Update 3

30 August 2019 13:11:36 +0000

At the risk of being repetitive, work on this continues. We’re in no rush to get this done too quickly, especially as shortcuts taken under pressure when setting up server NC031 (the current web server) led to some inconsistencies in how some accounts were set up that we’re only now able to deal with fully, and this is the best opportunity. Once those are sorted out we will do a test migration of some non-client test domains, and only when it is clear that the test migrations work flawlessly will we move client domains.

For this reason we’re not going to commit to finishing the migration to NC041 this weekend. We are, however, alive to the issues that have prompted this migration in the first place, and we are working diligently to get it done without undue delay.

That said, we have set up some clients on the new server if they happen to be in the process of setting up a new website anyway, and others that are new clients. If you’re in the unique position of being on the cusp of setting up a new website and would prefer to abandon your old website on the old server and start afresh on the new server, please contact us and we will do that for you immediately.

Thank-you.

Upgrade of NC031 to NC041: Update 2

26 August 2019 03:42:55 +0000

Work on this continues, but instead of a straight migration we have set up the new server from scratch. We’re in the process of adding domains that will not be migrated from NC031 to NC041.

Upgrade of NC031 to NC041: Update 1

20 August 2019 06:38:48 +0000

Work on this continues. We will shortly set up a number of domains on the new server that are awaiting a newer version of PHP, but do not need to be migrated. Following that, domains will be migrated from server NC031 to NC041.

At this point we are not anticipating much, if any, down time. However, this is subject to revision. Please contact support if you would like further details as they may apply to your account.

Please note that this maintenance and migration affects only websites. It does not in any way affect email.

Upgrade of NC031 to NC041

16 August 2019 10:24:47 +0000

We will be finalising work this weekend on setting up a new server on which the latest version of PHP will be available for those web apps that require it. If all goes well, all websites will be migrated in one fell swoop. If there are issues to overcome this will be delayed to some point during next week or perhaps on the following weekend.

Please check back here for updates. Thank-you.

NC031: Emergency maintenance complete

26 December 2018 12:18:17 +0000

We took server NC031 (the primary web server) down for eleven minutes today to correct an issue related to memory usage. The server was offline between 10:32 and 10:43 UTC.

We will be replacing this server early next month, and migrating all websites to the new server.

Our apologies for this inconvenience. If you have any questions or concerns, please contact NinerNet support. Thank-you.

NC036: Migration update 25 — Final

18 June 2018 08:54:43 +0000

The migration of all email accounts from server NC027 to server NC036 is complete. In fact, it was successfully completed at 04:00 UTC on 4 June. What followed over the next few days was an unprecedented avalanche of misinformation and red herrings that resulted in our moving the new server to another data centre (a move that took ten times longer than the previous move from the data centre where NC027 was located) where the same “problems” experienced by only some of our clients magically reappeared.

We planned the migration to have absolutely no impact on existing email configurations. We did this by pointing legacy sub-domains of the niner.net domain that named server NC027 — e.g., smtp27.niner.net — to server NC036. At the conclusion of the migration these sub-domains were indeed pointing to the new server. In other words, on Monday morning (4 June) email programs would have thought they were still downloading mail from the same server, not realising (or needing to realise) that they were in fact downloading from a new server.

However, it turned out that a significant minority of email programs were somehow misconfigured with settings that worked on the old server, but stopped working when connecting to the new server. Those clients who were using the correct settings experienced no disruption at all, and when those clients with incorrect settings corrected them on the morning of Monday the 11th, the problems were fixed instantly.

Over the rest of that week (11-15 June) we helped a few clients with some issues unique to how they use email, especially where those practices clashed with current best practices for email transmission. We also dealt with some issues of senders whose mail servers were behaving improperly, causing their emails to be blocked because they looked like spammers. This notably affected email from the ZRA, but their emails are once again flowing unimpeded.

We’re monitoring the spam filtering on the new server. Any message that the server identifies as spam will have the subject of the message prefixed to add “[SPAM]“. You can use this to configure your email program or the webmail to deal with spam automatically, by filtering it into your “junk” folder or deleting it entirely. We recommend filtering to the junk folder so that you can catch the occasional legitimate message that is misclassified as spam.

Finally, in recognition of the fact that the emergency migration of the server to a new data centre on 6 June disrupted all clients’ email, and the fact that those clients with misconfigured email programs experienced a week of disruption before the issue was identified, we will be applying a one-week (quarter month) credit to the accounts of all clients hosted on server NC036. We apologise for the difficulties caused, and will apply what was learned this time to future migrations.

Thank-you, as always, for your custom and patience.

NC036: Migration update 13

6 June 2018 12:52:23 +0000

We will post a postmortem here in due course, hopefully with 24-48 hours, along with a thousand more apologies, but we are looking for feedback to ensure that all clients are able to connect to the server and download and send email, as this was not the case on Monday and Tuesday.

NC036: Migration update 12 — server back online

6 June 2018 12:28:04 +0000

The transfer of the mail spools has completed and server NC036 was brought back online at 12:12 UTC.

NC036: Migration update 12

6 June 2018 08:32:10 +0000

The transfer of mail data between the old and new data centres is still underway. Considering the transfer of the same amount of data took 50 minutes over the weekend, and we are now at the 5-hour mark, it was impossible for us to predict that this would take so long.

I can assure you that I understand the frustration that you are feeling with this situation, but given the network problems in southern and central Africa that necessitated this emergency move, we had no choice but to act immediately rather than forcing many clients to do without mail until the weekend.

My best estimate at this point, based on how much data has transferred so far (55%) and how much is left, is that the transfer will complete at approximately 13:00 UTC. Assuming this is the case, the server will be back online and accessible at about 13:30 UTC.

I sincerely and deeply apologise for this situation.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: