NinerNet Communications™
System Status

Server and System Status

NC031: Emergency maintenance complete

26 December 2018 12:18:17 +0000

We took server NC031 (the primary web server) down for eleven minutes today to correct an issue related to memory usage. The server was offline between 10:32 and 10:43 UTC.

We will be replacing this server early next month, and migrating all websites to the new server.

Our apologies for this inconvenience. If you have any questions or concerns, please contact NinerNet support. Thank-you.

NC031: Database overload incident

14 December 2018 05:51:47 +0000

On 13 December (UTC) the database server on server NC031 (the primary web server) failed twice, the second time resulting in our deciding to reboot the server. We believe this to be the result of a marked increase in denial of service and hacking attempts against this server over the last few days.

The database server went down at 01:55. We immediately logged into the server to determine the cause, and restarted it at 02:37. Shortly after that the problem manifested itself again, we did a full reboot of the machine at 03:03, and the server was again online and fully functional at 03:06.

The database failure resulted in database-based websites — e.g., WordPress websites — generating “error connecting to database” errors.

This issue highlights an issue on this server that we intend to address very early in the New Year, that being a necessary upgrade of our firewall system to better handle such attacks in the future.

We apologise that this issue occurred. If you have any questions, please contact NinerNet support. Thank-you.

NC031: Data centre power failure

19 July 2018 05:09:49 +0000

At 23:04 UTC on 18 July connectivity issues were reported at the New York, US, data centre. The cause of this problem was later identified as a power outage. The data centre is currently reporting that power was restored to the data centre at 00:38 UTC on 19 July, but our own logs on the server itself indicate that it was only down between 22:55 and 23:50 UTC on 18 July. We are expecting a full report from the data centre and will post further details here when we receive that.

This affected our primary web server, hosting most clients’ websites, including our own primary website (www.niner.net).

We later soft-rebooted the server at 04:07 UTC on 19 July, and it was back online at 04:08.

This outage affected automated daily back-ups on the server. These have been restored and will run as scheduled at the next scheduled time, which is at midnight UTC on 20 July.

We apologise for this outage. The data centres we select are supposed to have redundant power systems to ensure that this kind of event never happens. However, clearly it did in this case and, as mentioned above, we are expecting a follow-up report from the data centre to explain this failure.

If you have any questions or concerns, please contact us to let us know. Thank-you, and we again apologise for this incident.

NC033: Maintenance complete

11 June 2018 00:45:06 +0000

Server NC033 is back online. It was down between 00:37 and 00:43 UTC.

NC033: Maintenance

11 June 2018 00:35:57 +0000

Server NC033 (the primary nameserver) is going down for maintenance in a few minutes for maintenance related to solving the mail server migration issue.

NC020: Reboot

2 June 2018 07:02:10 +0000

Server NC020 was rebooted at 06:19 UTC, and was back online again at 06:22.

NC031: Reboot

1 June 2018 04:59:41 +0000

As we work on diagnosing the back-up issue, server NC031 was rebooted at 04:24 UTC. It was back online at 04:25.

NC020: Reboot

8 May 2018 00:38:35 +0000

Due to an issue with the FTP server on NC020 we rebooted it at 00:27 UTC. It was back online at 00:28.

NC023: Maintenance complete

2 May 2018 21:03:31 +0000

Server NC023 was down between 20:06 and 20:23 UTC while a patch for the Spectre vulnerability was applied.

NC031: Maintenance complete

2 May 2018 06:08:55 +0000

Server NC031 was down between 05:03 and 05:17 UTC while a patch for the Spectre vulnerability was applied.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC020, Chicago, United States of America, OPERATIONAL.NC020OperationalUp?
Server NC023, London, United Kingdom, OPERATIONAL.NC023OperationalUp?
Server NC028, Vancouver, Canada, INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America, OPERATIONAL.NC031OperationalUp?
Server NC033, Toronto, Canada, OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia, INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia, TESTING.NC035TestingUp?
Server NC036, Amsterdam, Netherlands, OPERATIONAL.NC036OperationalUp?
Server NC037, Seattle, United States of America, INTERNAL.NC037InternalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

General Information:

This blog provides information about the status of NinerNet Communications systems. Dates and times of posts to this blog are in the UTC time zone, and dates and times given for events are also in the UTC time zone, although conversions may be offered for some time zones common to our clients. Please use the World Time Server to ensure accurate conversion of dates and times to your own time zone.

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings configuration connection issues control panel database dns dos attack dot-zm domains down time email delivery error messages ftp hardware imap mailing lists mail relay mail server microsoft migration nameservers network networking outlook performance phplist pop power failure reboot smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server wordpress

Resources:

On NinerNet: