Server and System Status

NC031: Web server downtime

14 March 2019 21:34:49 +0000

The database on our primary web server (NC031) went down at 20:20 UTC, and was not brought back up until 21:00 UTC. This means that all database-driven websites were down during this 40-minute period, and would have displayed the message, “Error establishing a database connection”. We are still trying to determine why this happened and why the service was not automatically restarted, as should have been the case. Coincidentally — or perhaps not — at the same time the server was under a heavy load from hundred of thousands of requests on a single website, and we have now blocked the source of that traffic.

We sincerely apologise for this inconvenience. If you have any questions, please do contact support.

Tags: connection issues, database, dos attack, down time, reboot, web server
Categories: Incidents, NC031
Comments Off

NC031: Database overload incident

14 December 2018 05:51:47 +0000

On 13 December (UTC) the database server on server NC031 (the primary web server) failed twice, the second time resulting in our deciding to reboot the server. We believe this to be the result of a marked increase in denial of service and hacking attempts against this server over the last few days.

The database server went down at 01:55. We immediately logged into the server to determine the cause, and restarted it at 02:37. Shortly after that the problem manifested itself again, we did a full reboot of the machine at 03:03, and the server was again online and fully functional at 03:06.

The database failure resulted in database-based websites — e.g., WordPress websites — generating “error connecting to database” errors.

This issue highlights an issue on this server that we intend to address very early in the New Year, that being a necessary upgrade of our firewall system to better handle such attacks in the future.

We apologise that this issue occurred. If you have any questions, please contact NinerNet support. Thank-you.

Tags: connection issues, database, dos attack, down time, reboot, web server
Categories: Incidents, NC031
Comments Off

London network restored

27 December 2015 10:17:45 +0000

The data centre informs us that connectivity on their network was restored at 09:24 UTC.

We have been able to confirm this by accessing all of our servers in this data centre:

Incoming mail is once again being processed immediately by server NC027, although any incoming mail sent during the disruption may be delayed a little longer until the sending mail servers try again. This is not under our control.
Server NC023 is also processing email immediately.

We apologise for this interruption. Denial of service attacks (especially distributed denial of service attacks) are extremely difficult and expensive to fight. We have confidence that the data centre did everything they could as quickly as they could to combat this attack.

Tags: connection issues, dos attack, email delivery, mail relay
Categories: Email, Incidents, NC023, NC027
Comments Off

London data centre DDoS update

27 December 2015 08:30:17 +0000

We have been updated that the denial of service attack against the London data centre is actually a large distributed denial of service (DDoS) attack, and it is still underway. Efforts are being made at multiple levels on the network to combat this attack, and we are seeing some movement of email through server NC027, although it took an hour and forty-seven minutes for a test message to get through. This indicates, as mentioned earlier, that the server is up and processing email, but that other servers (and our clients, obviously) are having problems connecting to the server.

We apologise for these problems, but this is affecting a wide range and large number of companies hosted at this data centre, not just NinerNet.

Updates will continue to be posted here as we learn more.

Tags: connection issues, dos attack, email delivery, mail relay
Categories: Email, Incidents, NC023, NC027
Comments Off

Connectivity issue at London data centre

27 December 2015 06:52:18 +0000

The London data centre has been suffering from the effects of a couple of denial of service (DoS) attacks over the last 12 or so hours. The attack late yesterday (26 December) lasted about an hour and a half. Today’s attack started at 05:37 UTC and is currently ongoing. The data centre staff assure us that they have identified the problem and are implementing a fix for it.

This affects server NC023 (the relay server) and server NC027 (the main mail server), as well as virtual private servers hosted in London. As far as we know, the servers themselves are up and connected to the network, but the attack is causing them to be inaccessible.

We’ll update here as soon as we know more.

Tags: connection issues, dos attack, email delivery, mail relay
Categories: Email, Incidents, NC023, NC027
Comments Off

NC018: Web server restarted

19 June 2014 07:08:10 +0000

We have restarted the web server and are keeping an eye on it, specifically the sources of traffic to it.

Tags: dos attack, down time, performance, web server
Categories: Incidents, NC018
Comments Off

NC018: Web server down

19 June 2014 06:57:04 +0000

We have taken down the web server while we investigate an overload condition. We will have it back up as soon as possible, and will post further updates here.

Email and other services on server NC018 are not affected.

Tags: dos attack, down time, performance, web server
Categories: Incidents, NC018
Comments Off

NC018: Investigation findings

8 March 2014 00:57:23 +0000

The web server on server NC018 sustained a denial-of-service attack between approximately 14:00 and 18:00 UTC on 7 March. This is why websites on this server were unavailable during this period. No other services (email or nameservers) were affected.

We have implemented improved monitoring of this activity on this server. Additionally, we are in discussions with the data centre to move this server to an improved network infrastructure, a move we hope to implement before the end of March.

We apologise for this disruption.

Tags: dos attack, down time, web server
Categories: Incidents, NC018
Comments Off

Server NC018 web server restarted

24 August 2013 08:21:57 +0000

We have identified the particular websites that are being attacked and have made adjustments to try and deflect the attack. In the meantime we have restarted the web server and will continue to monitor the attack.

Additionally, the mail queue has now cleared.

Tags: connection issues, dos attack, email delivery, performance
Categories: Incidents, NC018
Comments Off

NC018 web server stopped

24 August 2013 07:24:12 +0000

We have temporarily stopped the NC018 web server in an attempt to fight this attack.

Tags: connection issues, dos attack, down time, performance
Categories: Incidents, NC018
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?