NinerNet Communications™
System Status

Server and System Status

NC031: Web server downtime

14 March 2019 21:34:49 +0000

The database on our primary web server (NC031) went down at 20:20 UTC, and was not brought back up until 21:00 UTC. This means that all database-driven websites were down during this 40-minute period, and would have displayed the message, “Error establishing a database connection”. We are still trying to determine why this happened and why the service was not automatically restarted, as should have been the case. Coincidentally — or perhaps not — at the same time the server was under a heavy load from hundred of thousands of requests on a single website, and we have now blocked the source of that traffic.

We sincerely apologise for this inconvenience. If you have any questions, please do contact support.

NC031: Database overload incident

14 December 2018 05:51:47 +0000

On 13 December (UTC) the database server on server NC031 (the primary web server) failed twice, the second time resulting in our deciding to reboot the server. We believe this to be the result of a marked increase in denial of service and hacking attempts against this server over the last few days.

The database server went down at 01:55. We immediately logged into the server to determine the cause, and restarted it at 02:37. Shortly after that the problem manifested itself again, we did a full reboot of the machine at 03:03, and the server was again online and fully functional at 03:06.

The database failure resulted in database-based websites — e.g., WordPress websites — generating “error connecting to database” errors.

This issue highlights an issue on this server that we intend to address very early in the New Year, that being a necessary upgrade of our firewall system to better handle such attacks in the future.

We apologise that this issue occurred. If you have any questions, please contact NinerNet support. Thank-you.

London network restored

27 December 2015 10:17:45 +0000

The data centre informs us that connectivity on their network was restored at 09:24 UTC.

We have been able to confirm this by accessing all of our servers in this data centre:

  • Incoming mail is once again being processed immediately by server NC027, although any incoming mail sent during the disruption may be delayed a little longer until the sending mail servers try again. This is not under our control.
  • Server NC023 is also processing email immediately.

We apologise for this interruption. Denial of service attacks (especially distributed denial of service attacks) are extremely difficult and expensive to fight. We have confidence that the data centre did everything they could as quickly as they could to combat this attack.

London data centre DDoS update

27 December 2015 08:30:17 +0000

We have been updated that the denial of service attack against the London data centre is actually a large distributed denial of service (DDoS) attack, and it is still underway. Efforts are being made at multiple levels on the network to combat this attack, and we are seeing some movement of email through server NC027, although it took an hour and forty-seven minutes for a test message to get through. This indicates, as mentioned earlier, that the server is up and processing email, but that other servers (and our clients, obviously) are having problems connecting to the server.

We apologise for these problems, but this is affecting a wide range and large number of companies hosted at this data centre, not just NinerNet.

Updates will continue to be posted here as we learn more.

Connectivity issue at London data centre

27 December 2015 06:52:18 +0000

The London data centre has been suffering from the effects of a couple of denial of service (DoS) attacks over the last 12 or so hours. The attack late yesterday (26 December) lasted about an hour and a half. Today’s attack started at 05:37 UTC and is currently ongoing. The data centre staff assure us that they have identified the problem and are implementing a fix for it.

This affects server NC023 (the relay server) and server NC027 (the main mail server), as well as virtual private servers hosted in London. As far as we know, the servers themselves are up and connected to the network, but the attack is causing them to be inaccessible.

We’ll update here as soon as we know more.

NC018: Web server restarted

19 June 2014 07:08:10 +0000

We have restarted the web server and are keeping an eye on it, specifically the sources of traffic to it.

NC018: Web server down

19 June 2014 06:57:04 +0000

We have taken down the web server while we investigate an overload condition. We will have it back up as soon as possible, and will post further updates here.

Email and other services on server NC018 are not affected.

NC018: Investigation findings

8 March 2014 00:57:23 +0000

The web server on server NC018 sustained a denial-of-service attack between approximately 14:00 and 18:00 UTC on 7 March. This is why websites on this server were unavailable during this period. No other services (email or nameservers) were affected.

We have implemented improved monitoring of this activity on this server. Additionally, we are in discussions with the data centre to move this server to an improved network infrastructure, a move we hope to implement before the end of March.

We apologise for this disruption.

Server NC018 web server restarted

24 August 2013 08:21:57 +0000

We have identified the particular websites that are being attacked and have made adjustments to try and deflect the attack. In the meantime we have restarted the web server and will continue to monitor the attack.

Additionally, the mail queue has now cleared.

NC018 web server stopped

24 August 2013 07:24:12 +0000

We have temporarily stopped the NC018 web server in an attempt to fight this attack.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC020, Chicago, United States of America, OPERATIONAL.NC020OperationalUp?
Server NC023, London, United Kingdom, OPERATIONAL.NC023OperationalUp?
Server NC028, Vancouver, Canada, INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America, OPERATIONAL.NC031OperationalUp?
Server NC033, Toronto, Canada, OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia, INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia, OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands, OPERATIONAL.NC036OperationalUp?
Server NC037, Seattle, United States of America, INTERNAL.NC037InternalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

General Information:

This blog provides information about the status of NinerNet Communications systems. Dates and times of posts to this blog are in the UTC time zone, and dates and times given for events are also in the UTC time zone, although conversions may be offered for some time zones common to our clients. Please use the World Time Server to ensure accurate conversion of dates and times to your own time zone.

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings configuration connection issues control panel database dns dos attack dot-zm domains down time email delivery error messages ftp hardware imap mailing lists mail relay mail server microsoft migration nameservers network networking outlook performance phplist pop power failure reboot smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server wordpress

Resources:

On NinerNet: