Server and System Status

NC036: Messages to Outlook-/Microsoft-/Hotmail-hosted domains delayed

6 July 2022 09:12:00 +0000

Please note that we are seeing messages from NC036 destined for private domains hosted by Microsoft delayed at this time. The cause is not clear, but it’s clearly on Microsoft’s end.

This is not in any way related to the internal server issue that happened yesterday.

Tags: email, email delivery, mail, mail server, microsoft
Categories: Email, NC036
Comments Off

NC036: Issue resolved

5 July 2022 23:54:13 +0000

The issue now appears to be completely resolved. If you’re having any email issues, please do contact NinerNet support and we will look into them. Thank-you.

Tags: connection issues, email, imap, mail, mail server, performance, smtp, webmail
Categories: Email, Incidents, NC036
Comments Off

NC036: Server is up, but some issues

5 July 2022 23:09:28 +0000

We have been informed by the data centre where server NC036 is located that, due to “an effect from ongoing data centre maintenance”, there are connectivity issues. We have been noticing intermittent issues with both direct email connections and the webmail, notably slower transactions (e.g., deleting messages, scanning multiple folders, etc.). However, email is still flowing as usual, perhaps with minor delays.

We have opened a ticket with the data centre to determine why the issues are ongoing. We will update you as we receive more information.

Tags: connection issues, email, imap, mail, mail server, performance, smtp, webmail
Categories: Email, Incidents, NC036
Comments Off

NC036: Disk space issue resolved

8 May 2022 06:19:09 +0000

Briefly between Saturday and Sunday the disk storing emails on server NC036 became full. This was resolved immediately upon discovery, but some incoming emails were delayed. Some may also have bounced. However, this issue is now fully resolved.

If you have any questions, please contact NinerNet support.

Tags: bounce messages, email, email delivery, mail, mail server
Categories: Email, Incidents, NC036
Comments Off

NC033: Data centre maintenance

22 March 2022 09:31:57 +0000

The data centre at which server NC033 (our primary nameserver, ns1.niner.net) is housed will be undergoing some maintenance tomorrow, 23 March, between 05:00 and 08:00 UTC. The work is on the data centre’s network infrastructure, and not on this server itself. The data centre released this statement:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be temporary loss of connectivity …. We will endeavour to keep any such impact to a minimum.

The data centre managers have had a good reputation for reliability in the six years we have been using them, so we do not anticipate any issues. However, if there are, we expect they will be very brief.

If you have any questions or concerns, please contact NinerNet support. Thank-you.

Tags: dns, nameservers, network
Categories: Maintenance, NC033
Comments Off

NC041: Maintenance post-mortem

8 February 2022 07:39:37 +0000

Our apologies for not following up this issue within the promised 48 hours.

This issue was traced to a problem with a single website hosted on this server, and the modifications made to this website’s file system. What was done within one account’s file system should not have affected the whole server though, and as a result we have filed a bug report with the control panel vendor.

We do again apologise for this issue, and will endeavour to ensure that it does not become an issue again.

If you have any questions at all about this maintenance, please let us know by contacting NinerNet support. Thank-you for your patience, and continued patronage.

Tags: connection issues, down time, web server
Categories: Incidents, Maintenance, NC041
Comments Off

NC041: Primary web server back online

24 January 2022 05:51:20 +0000

The primary web server (NC041) is again online. It was down for 39 minutes between 05:05 and 05:44 UTC. We’re investigating why this incident happened, and will post further details here within the next 48 hours. Our apologies for the inconvenience.

Tags: connection issues, down time, web server
Categories: Incidents, Maintenance, NC041
Comments Off

NC041: Primary web server is currently down

24 January 2022 05:11:56 +0000

The primary web server (NC041) is currently down following maintenance. We are investigating. This does not affect email.

Tags: connection issues, down time, web server
Categories: Incidents, Maintenance, NC041
Comments Off

All servers: Company SSL/TLS certificates on all servers renewed and updated

14 January 2022 12:26:04 +0000

The *.niner.net wildcard SSL/TLS certificate has been renewed and updated across our entire infrastructure. This should be seamless for all our clients. However, should you run into a situation in the next 24 hours where you’re told that the certificate has expired, please log out and reload whatever it is you’re trying to do. You may need to reboot, but this would be very unusual. Simply forcing a reload should clear things up. Via FTPS you may need to re-trust our certificate depending on your FTP client.

This doesn’t affect certificates on your own domain on your website, just NinerNet services such as control panels and connecting to the mail and FTP servers.

If you have any questions or concerns, please do contact NinerNet support with any error messages you may be seeing. Thank-you.

Tags: ssl, ssl certificate, tls, tls certificate
Categories: Maintenance, NC023, NC031, NC033, NC035, NC036, NC040, NC041
Comments Off

NC036: Maintenance port-mortem

16 October 2021 10:41:00 +0000

As stated previously, the maintenance on server NC036 is complete, and has been successful.

That said, it had to be carried out earlier than scheduled because disk space on the server was the issue that was causing a problem. To be clear, the disk space used by email accounts was not and is not the issue (that’s on a separate disk), it was the disk space used by the operating system and the logs in particular. When this problem was noticed earlier this week we projected that addressing the issue during our regular weekend maintenance window on Saturday evening UTC would be sufficient. Well, we were wrong. As soon as it became clear that our temporary mitigation efforts were failing to keep up with the rate at which disk space was being consumed (mostly by the aforementioned logs), we immediately implemented the maintenance that we had planned and already practised on a test server. This practice paid off, as instead of allowing an hour for the maintenance to complete, we were done in four minutes, including a server reboot.

That said, the lack of disk space on the server did cause disruption in the rate at which mail was processed, that led to some of you getting errors when sending email, and some incoming email being delayed. For this we apologise. The backed-up mail queue was fully processed by 09:18 UTC.

Had our prediction held up and the maintenance been carried out this evening as scheduled, we wouldn’t even have needed to post this detail. However, in order to clear up any concern, we have.

Early in 2022 server NC036 is due to be considered for replacement. The experience of this issue and other issues we have run into in recent months will inform our decisions about where to locate the replacement server (hint: it won’t be in the current data centre) and various configuration improvements we can make. We’ll also, of course, be using a more current version of the control panel.

If you have any questions at all about today’s maintenance, please let us know by contacting NinerNet support. Thank-you for your patience, and continued patronage.

Tags: down time, email, mail, mail server, reboot, webmail
Categories: Email, Incidents, Maintenance, NC036
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?