Server and System Status

NC041: Automated renewals of Let’s Encrypt certificates failing

10 January 2020 03:21:00 +0000

Since about 1 January automated renewals of Let’s Encrypt SSL certificates have been failing on server NC041, and as a result you may receive automated emails from server NC041 that look like this:

Subject: Let’s Encrypt certificate renewal failed!

An error occurred requesting a new certificate for example.com, www.example.com from Let’s Encrypt : Web-based validation failed : <pre>sh: certonly: command not found
</pre>

We currently have a support ticket open with the control panel vendor to find out why this has suddenly started.

Since the renewal of Let’s Encrypt certificates is automatically scheduled to happen one month before they expire, the existing certificates will continue to be used and remain valid and in force. The issue will be resolved before these certificates start expiring in February.

If you have any concerns or questions about this, please contact NinerNet support. Thank-you.

Tags: control panel, ssl, ssl certificate, tls, tls certificate, web server
Categories: Incidents, NC041
1 Comment »

NC033: Incident with primary nameserver resolved

15 November 2019 03:10:35 +0000

Following an operating system update that was applied at 00:37 UTC on 15 November the primary nameserver failed to restart. This was noticed immediately upon completion of the update, and we started troubleshooting.

After extensive research on the likely cause and resolution of the problem we determined that the quickest way to get the nameserver back online was to roll back the update and restore from a back-up. There were some issues with this, however, that necessitated implementing a back-up plan for company email in order for us to gain access to our account at the data centre where the nameserver is hosted. Once this was accomplished the restore was initiated.

The nameserver was back online at 02:28 UTC.

During the time the server was down domain information would have been cached by DNS resolvers that already had a cached copy of the DNS information for your domain. However, it is possible that some incoming email may have been delayed or possibly bounced, or people trying to browse your website would not have been able to see it. For this we apologise.

If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience.

Tags: connection issues, dns, down time, nameservers
Categories: Incidents, Maintenance, NC033
Comments Off

Mail server latency issue resolved

30 September 2019 23:25:22 +0000

This issue on server NC036 appears to have been resolved, although we are awaiting official confirmation from the data centre. Should there be no change of status this issue will be closed with no further update here.

As always though, if you have any issues or concerns please contact support. Thank-you for your patience.

Tags: email, email delivery, mail, mail server, webmail
Categories: Email, Incidents, NC036
Comments Off

Email server (NC036) latency issue

30 September 2019 22:28:56 +0000

We are aware that connections to the primary mail server (NC036) are slow to the point they are timing out. We are told by the data centre where this server is located that this is an infrastructure issue of which they are aware and on which they are actively working.

We apologise for this temporary inconvenience. We will post further updates here as necessary or when the issue has been resolved. In the meantime, if you have any questions or concerns, please contact support. Thank-you.

Tags: email, email delivery, mail, mail server, webmail
Categories: Email, Incidents, NC036
Comments Off

NC031: Web server downtime

14 March 2019 21:34:49 +0000

The database on our primary web server (NC031) went down at 20:20 UTC, and was not brought back up until 21:00 UTC. This means that all database-driven websites were down during this 40-minute period, and would have displayed the message, “Error establishing a database connection”. We are still trying to determine why this happened and why the service was not automatically restarted, as should have been the case. Coincidentally — or perhaps not — at the same time the server was under a heavy load from hundred of thousands of requests on a single website, and we have now blocked the source of that traffic.

We sincerely apologise for this inconvenience. If you have any questions, please do contact support.

Tags: connection issues, database, dos attack, down time, reboot, web server
Categories: Incidents, NC031
Comments Off

NC023: Secondary issue resolved

12 March 2019 20:26:33 +0000

The secondary issues referred to previously were not issues at all. A single test out of several failed, but the failure was unrelated to this server.

This issue is now closed. We apologise for any inconvenience this emergency maintenance may have caused while the server was offline between approximately 13:26 and 14:03 UTC.

If you have any questions, please contact support. Thank-you for your patience.

Tags: down time, email delivery, mail relay, reboot, smtp
Categories: Email, Incidents, Maintenance, NC023
Comments Off

NC023: Emergency maintenance

12 March 2019 15:01:42 +0000

Due to issues affecting this server (NC023, the relay server), it has just undergone emergency maintenance. However, on boot there are new issues affecting it.

We are working to resolve the issue, and will post updates here. We apologise for this inconvenience.

Tags: down time, email delivery, mail relay, reboot, smtp
Categories: Email, Incidents, Maintenance, NC023
Comments Off

NC031: Emergency maintenance complete

26 December 2018 12:18:17 +0000

We took server NC031 (the primary web server) down for eleven minutes today to correct an issue related to memory usage. The server was offline between 10:32 and 10:43 UTC.

We will be replacing this server early next month, and migrating all websites to the new server.

Our apologies for this inconvenience. If you have any questions or concerns, please contact NinerNet support. Thank-you.

Tags: down time, hardware, migration, performance, reboot, web server
Categories: Incidents, Maintenance, NC031
Comments Off

NC031: Database overload incident

14 December 2018 05:51:47 +0000

On 13 December (UTC) the database server on server NC031 (the primary web server) failed twice, the second time resulting in our deciding to reboot the server. We believe this to be the result of a marked increase in denial of service and hacking attempts against this server over the last few days.

The database server went down at 01:55. We immediately logged into the server to determine the cause, and restarted it at 02:37. Shortly after that the problem manifested itself again, we did a full reboot of the machine at 03:03, and the server was again online and fully functional at 03:06.

The database failure resulted in database-based websites — e.g., WordPress websites — generating “error connecting to database” errors.

This issue highlights an issue on this server that we intend to address very early in the New Year, that being a necessary upgrade of our firewall system to better handle such attacks in the future.

We apologise that this issue occurred. If you have any questions, please contact NinerNet support. Thank-you.

Tags: connection issues, database, dos attack, down time, reboot, web server
Categories: Incidents, NC031
Comments Off

NC036: Mail server blocked by Microsoft

14 December 2018 05:33:38 +0000

We are aware that the IP address of server NC036 (the primary mail server) has again been blocked by Microsoft’s various mail services, variously known as Outlook.com, MSN, Hotmail, Live.com, etc.

Although we are a member of their Smart Network Data Services programme and Junk Mail Reporting Program, which are supposed to allow us to proactively prevent these kinds of issues, we have been unable to use the service as advertised, or at least as we understand it’s supposed to work. We will continue to attempt to have this server’s IP address removed from their blacklist, and report here when we have success.

In the meantime, outgoing mail to their primary domains (hotmail.ca, hotmail.com, hotmail.co.uk, live.com, msn.com and outlook.com) is being routed through our relay server. If you receive a bounce message that reads similarly the one below to an email you’ve sent, it is probably for a private domain hosted by Microsoft of which we are not aware. Please contact us and we will add it to the list of domains for which email is routed through our relay server:

host 901e3cd0af6f44ab11b5a5e8a49da3.pamx1.hotmail.com[104.47.0.33] said: 550
5.7.1 Unfortunately, messages from [178.62.195.26] weren’t sent. Please
contact your Internet service provider since part of their network is on
our block list (S3140). You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors.
[HE1EUR01FT033.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
FROM command)

Please remember that all email you send through our mail server must be to recipients with whom you already have a business or personal relationship, and all mass email must be explicitly requested — i.e., Confirmed opt-in (COI) or Double opt-in (DOI) email.

Thanks for your cooperation, and our apologies for this inconvenience.

Tags: email delivery, mail relay, mail server, microsoft, spam
Categories: Email, Incidents, NC036
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?