Server and System Status

NC023: Secondary issue resolved

12 March 2019 20:26:33 +0000

The secondary issues referred to previously were not issues at all. A single test out of several failed, but the failure was unrelated to this server.

This issue is now closed. We apologise for any inconvenience this emergency maintenance may have caused while the server was offline between approximately 13:26 and 14:03 UTC.

If you have any questions, please contact support. Thank-you for your patience.

Tags: down time, email delivery, mail relay, reboot, smtp
Categories: Email, Incidents, Maintenance, NC023
Comments Off

NC023: Emergency maintenance

12 March 2019 15:01:42 +0000

Due to issues affecting this server (NC023, the relay server), it has just undergone emergency maintenance. However, on boot there are new issues affecting it.

We are working to resolve the issue, and will post updates here. We apologise for this inconvenience.

Tags: down time, email delivery, mail relay, reboot, smtp
Categories: Email, Incidents, Maintenance, NC023
Comments Off

You may have received automated emails from the web server (NC031) control panel recently advising of the availability of an upgrade for WordPress to version 5.0.3. Normally your installation of WordPress (or NinerNet) would upgrade WordPress for you, but because this is a “major upgrade” — from version 4.x to version 5.x — the automated upgrade process is not activated by WordPress and we also will not automatically upgrade WordPress installations installed through the control panel.

However, as you are no doubt aware, WordPress advises that you should always be running the latest version, and this is standard advice for any software, especially Internet-facing software. In fact, most websites run by web applications (such as WordPress) that are hacked at some point are hacked because they’re running an out-of-date version.

As such we strongly recommend that you upgrade your installation of WordPress to the latest version as soon as possible.

The most straightforward way to do this is to log into your installation of WordPress and follow the instructions there to run the upgrade. Or you can do so through the web server control panel at Install Scripts -> Installed Scripts.

You may take one of the following three courses of action at this time:

Upgrade your installation of WordPress yourself using one of the two methods stated immediately above.
Allow us to run an automated upgrade for you.
Choose not to upgrade. If you choose this course of action, please advise us so that we can exclude your website from an automated upgrade, and confirm that you accept the risk of running out-of-date software.

Please note that if your installation of WordPress was not installed through the control panel, then we cannot upgrade it for you. You can confirm whether or not WordPress was installed through the control panel by logging into the web server control panel, and navigating to Install Scripts -> Installed Scripts. If WordPress is listed there you can run the upgrade from there; if it is not listed, then you or your website manager or designer installed WordPress manually, and will need to upgrade it by logging into WordPress.

Any installations of WordPress that have not been upgraded by their owners by 00:01 UTC on Thursday 24 January will be upgraded by NinerNet through the control panel — unless you have asked to be excluded or your installation of WordPress was installed manually — shortly thereafter. To convert this deadline to your time zone please consult the World Time Server.

Thank-you for your attention to this matter. If you have any questions, please contact support.

Tags: web server, wordpress
Categories: Maintenance, NC031, Upgrades
Comments Off

NC031: Emergency maintenance complete

26 December 2018 12:18:17 +0000

We took server NC031 (the primary web server) down for eleven minutes today to correct an issue related to memory usage. The server was offline between 10:32 and 10:43 UTC.

We will be replacing this server early next month, and migrating all websites to the new server.

Our apologies for this inconvenience. If you have any questions or concerns, please contact NinerNet support. Thank-you.

Tags: down time, hardware, migration, performance, reboot, web server
Categories: Incidents, Maintenance, NC031
Comments Off

NC036: Maintenance complete

15 December 2018 19:57:20 +0000

Our scheduled maintenance to increase the storage space available for email accounts on the mail server (NC036) is complete. The ability to send and receive email was not available between 19:17 and 19:44 UTC. The expanded disk space has been tested and all is running normally.

If you have any issues or questions, please contact NinerNet support. Thank-you.

Tags: down time, email delivery, mail relay, smtp, webmail
Categories: Email, Maintenance, NC036
Comments Off

NC036: Scheduled maintenance underway

15 December 2018 19:14:54 +0000

The scheduled maintenance to increase hard drive space on server NC036 is underway. We will post progress reports or notice that the maintenance is complete as necessary.

Tags: down time, email delivery, mail relay, smtp, webmail
Categories: Email, Maintenance, NC036
Comments Off

NC036: Scheduled mail server maintenance

14 December 2018 07:03:01 +0000

During this weekend’s maintenance window we will be adding hard drive storage to server NC036 to continue to provide more storage space for a growing number of growing email accounts. This maintenance is scheduled to start at 19:00 UTC and we anticipate it will last less than one hour.

During the maintenance the ability to send and receive email will not be available, both via standalone email programs (e.g., Outlook, Thunderbird, etc.) and the webmail. Incoming email will be queued on the sending servers until our server is back online again, after which it will then be delivered to our server and your email account. This may result in a delay longer than the planned hour of the maintenance though.

Please monitor this status page to be notified of the start and end of the maintenance. If you have any questions or concerns, please contact NinerNet support.

Thank-you for your patience as we continue to work to improve our services to you.

Tags: down time, email delivery, mail server, smtp, webmail
Categories: Email, Maintenance, NC036
1 Comment »

NC031: Database overload incident

14 December 2018 05:51:47 +0000

On 13 December (UTC) the database server on server NC031 (the primary web server) failed twice, the second time resulting in our deciding to reboot the server. We believe this to be the result of a marked increase in denial of service and hacking attempts against this server over the last few days.

The database server went down at 01:55. We immediately logged into the server to determine the cause, and restarted it at 02:37. Shortly after that the problem manifested itself again, we did a full reboot of the machine at 03:03, and the server was again online and fully functional at 03:06.

The database failure resulted in database-based websites — e.g., WordPress websites — generating “error connecting to database” errors.

This issue highlights an issue on this server that we intend to address very early in the New Year, that being a necessary upgrade of our firewall system to better handle such attacks in the future.

We apologise that this issue occurred. If you have any questions, please contact NinerNet support. Thank-you.

Tags: connection issues, database, dos attack, down time, reboot, web server
Categories: Incidents, NC031
Comments Off

NC036: Mail server blocked by Microsoft

14 December 2018 05:33:38 +0000

We are aware that the IP address of server NC036 (the primary mail server) has again been blocked by Microsoft’s various mail services, variously known as Outlook.com, MSN, Hotmail, Live.com, etc.

Although we are a member of their Smart Network Data Services programme and Junk Mail Reporting Program, which are supposed to allow us to proactively prevent these kinds of issues, we have been unable to use the service as advertised, or at least as we understand it’s supposed to work. We will continue to attempt to have this server’s IP address removed from their blacklist, and report here when we have success.

In the meantime, outgoing mail to their primary domains (hotmail.ca, hotmail.com, hotmail.co.uk, live.com, msn.com and outlook.com) is being routed through our relay server. If you receive a bounce message that reads similarly the one below to an email you’ve sent, it is probably for a private domain hosted by Microsoft of which we are not aware. Please contact us and we will add it to the list of domains for which email is routed through our relay server:

host 901e3cd0af6f44ab11b5a5e8a49da3.pamx1.hotmail.com[104.47.0.33] said: 550
5.7.1 Unfortunately, messages from [178.62.195.26] weren’t sent. Please
contact your Internet service provider since part of their network is on
our block list (S3140). You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors.
[HE1EUR01FT033.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
FROM command)

Please remember that all email you send through our mail server must be to recipients with whom you already have a business or personal relationship, and all mass email must be explicitly requested — i.e., Confirmed opt-in (COI) or Double opt-in (DOI) email.

Thanks for your cooperation, and our apologies for this inconvenience.

Tags: email delivery, mail relay, mail server, microsoft, spam
Categories: Email, Incidents, NC036
Comments Off

NC031: Server back online

18 August 2018 01:28:10 +0000

Server NC031 is back online, although we are waiting for word from the data centre that the issue is definitely completely resolved.

Tags: connection issues, database, network, networking, web server
Categories: Incidents, NC031
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?