NinerNet Communications™
System Status

Server and System Status

Phone system down

4 June 2020 04:31:52 +0000

Clients should be aware that our North American phone system is down right now, and has been since shortly after 17:00 Pacific time today, 3 June (00:00 UTC on 4 June). Currently we have no estimate from our phone provider when this issue will be rectified.

However, of course, email is our primary method of communication and providing support, so please continue to use that. It’s working just fine.

NC041: Automated certificate renewals should be working again

12 January 2020 13:01:11 +0000

The issue with automated renewals of Let’s Encrypt certificates on server NC041 should now be fixed. We have taken the action suggested by the control panel vendor, and we are waiting for the first automated renewal to occur to see if everything is once again working. If, as we expect, the issue is fixed there will be no further posts on this subject.

If you receive any error messages related to certificate renewals — or any error emails at all — please contact NinerNet support and we’ll investigate further. Thank-you.

NC041: Automated renewals of Let’s Encrypt certificates failing

10 January 2020 03:21:00 +0000

Since about 1 January automated renewals of Let’s Encrypt SSL certificates have been failing on server NC041, and as a result you may receive automated emails from server NC041 that look like this:

Subject: Let’s Encrypt certificate renewal failed!

An error occurred requesting a new certificate for example.com, www.example.com from Let’s Encrypt : Web-based validation failed : <pre>sh: certonly: command not found
</pre>

We currently have a support ticket open with the control panel vendor to find out why this has suddenly started.

Since the renewal of Let’s Encrypt certificates is automatically scheduled to happen one month before they expire, the existing certificates will continue to be used and remain valid and in force. The issue will be resolved before these certificates start expiring in February.

If you have any concerns or questions about this, please contact NinerNet support. Thank-you.

NC033: Incident with primary nameserver resolved

15 November 2019 03:10:35 +0000

Following an operating system update that was applied at 00:37 UTC on 15 November the primary nameserver failed to restart. This was noticed immediately upon completion of the update, and we started troubleshooting.

After extensive research on the likely cause and resolution of the problem we determined that the quickest way to get the nameserver back online was to roll back the update and restore from a back-up. There were some issues with this, however, that necessitated implementing a back-up plan for company email in order for us to gain access to our account at the data centre where the nameserver is hosted. Once this was accomplished the restore was initiated.

The nameserver was back online at 02:28 UTC.

During the time the server was down domain information would have been cached by DNS resolvers that already had a cached copy of the DNS information for your domain. However, it is possible that some incoming email may have been delayed or possibly bounced, or people trying to browse your website would not have been able to see it. For this we apologise.

If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience.

Mail server latency issue resolved

30 September 2019 23:25:22 +0000

This issue on server NC036 appears to have been resolved, although we are awaiting official confirmation from the data centre. Should there be no change of status this issue will be closed with no further update here.

As always though, if you have any issues or concerns please contact support. Thank-you for your patience.

Email server (NC036) latency issue

30 September 2019 22:28:56 +0000

We are aware that connections to the primary mail server (NC036) are slow to the point they are timing out. We are told by the data centre where this server is located that this is an infrastructure issue of which they are aware and on which they are actively working.

We apologise for this temporary inconvenience. We will post further updates here as necessary or when the issue has been resolved. In the meantime, if you have any questions or concerns, please contact support. Thank-you.

NC031: Web server downtime

14 March 2019 21:34:49 +0000

The database on our primary web server (NC031) went down at 20:20 UTC, and was not brought back up until 21:00 UTC. This means that all database-driven websites were down during this 40-minute period, and would have displayed the message, “Error establishing a database connection”. We are still trying to determine why this happened and why the service was not automatically restarted, as should have been the case. Coincidentally — or perhaps not — at the same time the server was under a heavy load from hundred of thousands of requests on a single website, and we have now blocked the source of that traffic.

We sincerely apologise for this inconvenience. If you have any questions, please do contact support.

NC023: Secondary issue resolved

12 March 2019 20:26:33 +0000

The secondary issues referred to previously were not issues at all. A single test out of several failed, but the failure was unrelated to this server.

This issue is now closed. We apologise for any inconvenience this emergency maintenance may have caused while the server was offline between approximately 13:26 and 14:03 UTC.

If you have any questions, please contact support. Thank-you for your patience.

NC023: Emergency maintenance

12 March 2019 15:01:42 +0000

Due to issues affecting this server (NC023, the relay server), it has just undergone emergency maintenance. However, on boot there are new issues affecting it.

We are working to resolve the issue, and will post updates here. We apologise for this inconvenience.

NC031: Emergency maintenance complete

26 December 2018 12:18:17 +0000

We took server NC031 (the primary web server) down for eleven minutes today to correct an issue related to memory usage. The server was offline between 10:32 and 10:43 UTC.

We will be replacing this server early next month, and migrating all websites to the new server.

Our apologies for this inconvenience. If you have any questions or concerns, please contact NinerNet support. Thank-you.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), OPERATIONAL.NC023OperationalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC037, Seattle, United States of America (Mail server, secondary mail server), OPERATIONAL.NC037OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings configuration connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking outlook performance php phplist pop reboot smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: