Server and System Status

NC031: Emergency maintenance underway

27 April 2019 19:23:19 +0000

The previously scheduled maintenance to add memory to our primary web server (NC031) started at 19:01 UTC, and should take less time than we have scheduled. We will post updates here at the top of every hour.

Tags: down time, reboot, web server
Categories: Maintenance, NC031, Upgrades
Comments Off

NC031: Emergency server maintenance

26 April 2019 11:02:07 +0000

We will be upgrading server NC031 (the primary web server) with more memory to address memory shortage issues that have been affecting some websites recently. This maintenance is scheduled to start at 19:00 UTC on Saturday 27 April and is expected to take up to five hours, concluding before approximately 23:59 the same day. (Please use the World Time Server to convert this time to your time zone.) It is possible that the maintenance will take less time. During this time all websites (including the main NinerNet website) will be unavailable, for both browsing and editing.

This maintenance will not affect email.

We will post a notice here when the maintenance starts, progress updates on the hour every hour, and when the maintenance completes. We thank you for your patience as we work to improve our network’s reliability. If you have any questions or concerns, please contact NinerNet support.

Tags: down time, reboot, web server
Categories: Maintenance, NC031, Upgrades
Comments Off

NC031: Web server downtime

14 March 2019 21:34:49 +0000

The database on our primary web server (NC031) went down at 20:20 UTC, and was not brought back up until 21:00 UTC. This means that all database-driven websites were down during this 40-minute period, and would have displayed the message, “Error establishing a database connection”. We are still trying to determine why this happened and why the service was not automatically restarted, as should have been the case. Coincidentally — or perhaps not — at the same time the server was under a heavy load from hundred of thousands of requests on a single website, and we have now blocked the source of that traffic.

We sincerely apologise for this inconvenience. If you have any questions, please do contact support.

Tags: connection issues, database, dos attack, down time, reboot, web server
Categories: Incidents, NC031
Comments Off

Major WordPress upgrade

19 January 2019 08:35:33 +0000

You may have received automated emails from the web server (NC031) control panel recently advising of the availability of an upgrade for WordPress to version 5.0.3. Normally your installation of WordPress (or NinerNet) would upgrade WordPress for you, but because this is a “major upgrade” — from version 4.x to version 5.x — the automated upgrade process is not activated by WordPress and we also will not automatically upgrade WordPress installations installed through the control panel.

However, as you are no doubt aware, WordPress advises that you should always be running the latest version, and this is standard advice for any software, especially Internet-facing software. In fact, most websites run by web applications (such as WordPress) that are hacked at some point are hacked because they’re running an out-of-date version.

As such we strongly recommend that you upgrade your installation of WordPress to the latest version as soon as possible.

The most straightforward way to do this is to log into your installation of WordPress and follow the instructions there to run the upgrade. Or you can do so through the web server control panel at Install Scripts -> Installed Scripts.

You may take one of the following three courses of action at this time:

Upgrade your installation of WordPress yourself using one of the two methods stated immediately above.
Allow us to run an automated upgrade for you.
Choose not to upgrade. If you choose this course of action, please advise us so that we can exclude your website from an automated upgrade, and confirm that you accept the risk of running out-of-date software.

Please note that if your installation of WordPress was not installed through the control panel, then we cannot upgrade it for you. You can confirm whether or not WordPress was installed through the control panel by logging into the web server control panel, and navigating to Install Scripts -> Installed Scripts. If WordPress is listed there you can run the upgrade from there; if it is not listed, then you or your website manager or designer installed WordPress manually, and will need to upgrade it by logging into WordPress.

Any installations of WordPress that have not been upgraded by their owners by 00:01 UTC on Thursday 24 January will be upgraded by NinerNet through the control panel — unless you have asked to be excluded or your installation of WordPress was installed manually — shortly thereafter. To convert this deadline to your time zone please consult the World Time Server.

Thank-you for your attention to this matter. If you have any questions, please contact support.

Tags: web server, wordpress
Categories: Maintenance, NC031, Upgrades
Comments Off

NC031: Emergency maintenance complete

26 December 2018 12:18:17 +0000

We took server NC031 (the primary web server) down for eleven minutes today to correct an issue related to memory usage. The server was offline between 10:32 and 10:43 UTC.

We will be replacing this server early next month, and migrating all websites to the new server.

Our apologies for this inconvenience. If you have any questions or concerns, please contact NinerNet support. Thank-you.

Tags: down time, hardware, migration, performance, reboot, web server
Categories: Incidents, Maintenance, NC031
Comments Off

NC031: Database overload incident

14 December 2018 05:51:47 +0000

On 13 December (UTC) the database server on server NC031 (the primary web server) failed twice, the second time resulting in our deciding to reboot the server. We believe this to be the result of a marked increase in denial of service and hacking attempts against this server over the last few days.

The database server went down at 01:55. We immediately logged into the server to determine the cause, and restarted it at 02:37. Shortly after that the problem manifested itself again, we did a full reboot of the machine at 03:03, and the server was again online and fully functional at 03:06.

The database failure resulted in database-based websites — e.g., WordPress websites — generating “error connecting to database” errors.

This issue highlights an issue on this server that we intend to address very early in the New Year, that being a necessary upgrade of our firewall system to better handle such attacks in the future.

We apologise that this issue occurred. If you have any questions, please contact NinerNet support. Thank-you.

Tags: connection issues, database, dos attack, down time, reboot, web server
Categories: Incidents, NC031
Comments Off

NC031: Server back online

18 August 2018 01:28:10 +0000

Server NC031 is back online, although we are waiting for word from the data centre that the issue is definitely completely resolved.

Tags: connection issues, database, network, networking, web server
Categories: Incidents, NC031
Comments Off

NC031: Networking issues at data centre

18 August 2018 01:11:02 +0000

At 00:09 UTC on 18 August we became aware of a networking issue at the data centre where NC031 — the primary web server — is located. This is affecting almost all web hosting clients, including the primary NinerNet website. The web server itself is up and running, as far as we know, but the problem is that traffic to and from the data centre is down.

This issue does not affect any email services.

Data centre staff are working on restoring connectivity, and we will post an update as soon as we are aware that the web server is once gain accessible. We apologise for this issue.

Tags: connection issues, database, network, networking, web server
Categories: Incidents, NC031
Comments Off

NC031: Data centre power failure

19 July 2018 05:09:49 +0000

At 23:04 UTC on 18 July connectivity issues were reported at the New York, US, data centre. The cause of this problem was later identified as a power outage. The data centre is currently reporting that power was restored to the data centre at 00:38 UTC on 19 July, but our own logs on the server itself indicate that it was only down between 22:55 and 23:50 UTC on 18 July. We are expecting a full report from the data centre and will post further details here when we receive that.

This affected our primary web server, hosting most clients’ websites, including our own primary website (www.niner.net).

We later soft-rebooted the server at 04:07 UTC on 19 July, and it was back online at 04:08.

This outage affected automated daily back-ups on the server. These have been restored and will run as scheduled at the next scheduled time, which is at midnight UTC on 20 July.

We apologise for this outage. The data centres we select are supposed to have redundant power systems to ensure that this kind of event never happens. However, clearly it did in this case and, as mentioned above, we are expecting a follow-up report from the data centre to explain this failure.

If you have any questions or concerns, please contact us to let us know. Thank-you, and we again apologise for this incident.

Tags: back-up, connection issues, down time, power failure, reboot, web server
Categories: Incidents, NC031
Comments Off

NC031: Maintenance complete

3 June 2018 02:49:41 +0000

Scheduled maintenance on server NC031 is complete. The server was down between 01:54 and 02:36 UTC.

We anticipate that the intermittent back-up issue should be resolved, but we’ll be monitoring it more closely than usual for the next few days at least.

In addition to the database work, we have installed more memory and hard drive space on the server.

Thanks again for your patience.

Tags: back-up, database, down time, web server
Categories: Maintenance, NC031
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?