Server and System Status

NC023: Maintenance underway

27 March 2021 19:32:47 +0000

Maintenance on server NC023 is about to begin, about half an hour late. Apologies for the delay.

Tags: connection issues, down time, email, hardware, mail, mail relay, smtp
Categories: Maintenance, NC023
Comments Off

NC023: Relay server maintenance

26 March 2021 07:11:25 +0000

The relay mail server will be down briefly for some maintenance during our weekly maintenance window this weekend. This will start at 19:00 UTC on Saturday 27 March 2021, and should be complete within the hour.

We will post an update when the maintenance starts and when it is finished. This does not affect our primary mail server; it only affects you if you know that you use the relay server for outbound email.

If you have any questions, please contact NinerNet support. Thanks very much for your patience and understanding.

Tags: connection issues, down time, email, hardware, mail, mail relay, smtp
Categories: Maintenance, NC023
Comments Off

NC036: Post-mortem

21 September 2020 08:42:47 +0000

As noted in the previous two posts, there was a virus outbreak on server NC036 (the primary mail server) this morning. Apparently the machines associated with five email accounts on three domains were compromised, allowing criminals to use those accounts to send thousands of viruses. These were intercepted by our anti-virus scanner, but due to the volume of activity on the server we had to shut down the SMTP side of the mail server while we determined which email accounts were compromised, suspended them and removed their messages from the mail queue.

Please note that what happens in almost all cases when email accounts are compromised is that the computer (or one of the machines or devices on which those accounts are configured) is what is actually compromised; it is not the server. The account owner’s machine is usually infected with a virus or other malware, and the account’s password is then transmitted to the criminals behind the virus. They then launch an attack via the legitimate and correct password. It’s as if your car was stolen and the thief used it to commit a crime; the car behaved as it was told by the guy with the key, but is not responsible for the crime. On the other hand, the owner of the car may have left the key in their car and the door unlocked, contributing to the compromise. This is why it is vitally important that you have anti-virus software installed on your computer, and kept up-to-date.

If you have any questions about this, please feel free to contact NinerNet support, and we’ll be happy to answer your questions or concerns. Our apologies for the interruption.

Tags: connection issues, down time, email delivery, imap, mail, pop, smtp, viruses
Categories: Email, Incidents, NC036
Comments Off

NC036: Mail server is back online

21 September 2020 07:26:54 +0000

Our apologies. The sending side of the mail server (NC036) is back up. It was down for 21 minutes between 06:56 and 07:17 UTC. The ability to check your email account was not down.

We will post additional information and contact the affected clients shortly.

Tags: connection issues, down time, email delivery, imap, mail relay, pop, smtp, viruses
Categories: Email, Incidents, NC036
Comments Off

NC036: Mail server paused while we clean up a spam outbreak

21 September 2020 07:05:48 +0000

Server NC036 (the primary mail server) is temporarily paused while we clean up a huge virus outbreak. I will be back online as soon as possible.

Tags: connection issues, down time, email delivery, imap, mail server, pop, smtp, viruses
Categories: Email, Incidents, NC036
Comments Off

NC033: Incident with primary nameserver resolved

15 November 2019 03:10:35 +0000

Following an operating system update that was applied at 00:37 UTC on 15 November the primary nameserver failed to restart. This was noticed immediately upon completion of the update, and we started troubleshooting.

After extensive research on the likely cause and resolution of the problem we determined that the quickest way to get the nameserver back online was to roll back the update and restore from a back-up. There were some issues with this, however, that necessitated implementing a back-up plan for company email in order for us to gain access to our account at the data centre where the nameserver is hosted. Once this was accomplished the restore was initiated.

The nameserver was back online at 02:28 UTC.

During the time the server was down domain information would have been cached by DNS resolvers that already had a cached copy of the DNS information for your domain. However, it is possible that some incoming email may have been delayed or possibly bounced, or people trying to browse your website would not have been able to see it. For this we apologise.

If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience.

Tags: connection issues, dns, down time, nameservers
Categories: Incidents, Maintenance, NC033
Comments Off

NC031: Web server downtime

14 March 2019 21:34:49 +0000

The database on our primary web server (NC031) went down at 20:20 UTC, and was not brought back up until 21:00 UTC. This means that all database-driven websites were down during this 40-minute period, and would have displayed the message, “Error establishing a database connection”. We are still trying to determine why this happened and why the service was not automatically restarted, as should have been the case. Coincidentally — or perhaps not — at the same time the server was under a heavy load from hundred of thousands of requests on a single website, and we have now blocked the source of that traffic.

We sincerely apologise for this inconvenience. If you have any questions, please do contact support.

Tags: connection issues, database, dos attack, down time, reboot, web server
Categories: Incidents, NC031
Comments Off

NC031: Database overload incident

14 December 2018 05:51:47 +0000

On 13 December (UTC) the database server on server NC031 (the primary web server) failed twice, the second time resulting in our deciding to reboot the server. We believe this to be the result of a marked increase in denial of service and hacking attempts against this server over the last few days.

The database server went down at 01:55. We immediately logged into the server to determine the cause, and restarted it at 02:37. Shortly after that the problem manifested itself again, we did a full reboot of the machine at 03:03, and the server was again online and fully functional at 03:06.

The database failure resulted in database-based websites — e.g., WordPress websites — generating “error connecting to database” errors.

This issue highlights an issue on this server that we intend to address very early in the New Year, that being a necessary upgrade of our firewall system to better handle such attacks in the future.

We apologise that this issue occurred. If you have any questions, please contact NinerNet support. Thank-you.

Tags: connection issues, database, dos attack, down time, reboot, web server
Categories: Incidents, NC031
Comments Off

NC031: Server back online

18 August 2018 01:28:10 +0000

Server NC031 is back online, although we are waiting for word from the data centre that the issue is definitely completely resolved.

Tags: connection issues, database, network, networking, web server
Categories: Incidents, NC031
Comments Off

NC031: Networking issues at data centre

18 August 2018 01:11:02 +0000

At 00:09 UTC on 18 August we became aware of a networking issue at the data centre where NC031 — the primary web server — is located. This is affecting almost all web hosting clients, including the primary NinerNet website. The web server itself is up and running, as far as we know, but the problem is that traffic to and from the data centre is down.

This issue does not affect any email services.

Data centre staff are working on restoring connectivity, and we will post an update as soon as we are aware that the web server is once gain accessible. We apologise for this issue.

Tags: connection issues, database, network, networking, web server
Categories: Incidents, NC031
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Degraded	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?