NinerNet Communications™
System Status

Server and System Status

London network restored

27 December 2015 10:17:45 +0000

The data centre informs us that connectivity on their network was restored at 09:24 UTC.

We have been able to confirm this by accessing all of our servers in this data centre:

  • Incoming mail is once again being processed immediately by server NC027, although any incoming mail sent during the disruption may be delayed a little longer until the sending mail servers try again. This is not under our control.
  • Server NC023 is also processing email immediately.

We apologise for this interruption. Denial of service attacks (especially distributed denial of service attacks) are extremely difficult and expensive to fight. We have confidence that the data centre did everything they could as quickly as they could to combat this attack.

London data centre DDoS update

27 December 2015 08:30:17 +0000

We have been updated that the denial of service attack against the London data centre is actually a large distributed denial of service (DDoS) attack, and it is still underway. Efforts are being made at multiple levels on the network to combat this attack, and we are seeing some movement of email through server NC027, although it took an hour and forty-seven minutes for a test message to get through. This indicates, as mentioned earlier, that the server is up and processing email, but that other servers (and our clients, obviously) are having problems connecting to the server.

We apologise for these problems, but this is affecting a wide range and large number of companies hosted at this data centre, not just NinerNet.

Updates will continue to be posted here as we learn more.

Connectivity issue at London data centre

27 December 2015 06:52:18 +0000

The London data centre has been suffering from the effects of a couple of denial of service (DoS) attacks over the last 12 or so hours. The attack late yesterday (26 December) lasted about an hour and a half. Today’s attack started at 05:37 UTC and is currently ongoing. The data centre staff assure us that they have identified the problem and are implementing a fix for it.

This affects server NC023 (the relay server) and server NC027 (the main mail server), as well as virtual private servers hosted in London. As far as we know, the servers themselves are up and connected to the network, but the attack is causing them to be inaccessible.

We’ll update here as soon as we know more.

NC018: Web server restarted

19 June 2014 07:08:10 +0000

We have restarted the web server and are keeping an eye on it, specifically the sources of traffic to it.

NC018: Web server down

19 June 2014 06:57:04 +0000

We have taken down the web server while we investigate an overload condition. We will have it back up as soon as possible, and will post further updates here.

Email and other services on server NC018 are not affected.

NC018: Investigation findings

8 March 2014 00:57:23 +0000

The web server on server NC018 sustained a denial-of-service attack between approximately 14:00 and 18:00 UTC on 7 March. This is why websites on this server were unavailable during this period. No other services (email or nameservers) were affected.

We have implemented improved monitoring of this activity on this server. Additionally, we are in discussions with the data centre to move this server to an improved network infrastructure, a move we hope to implement before the end of March.

We apologise for this disruption.

Server NC018 web server restarted

24 August 2013 08:21:57 +0000

We have identified the particular websites that are being attacked and have made adjustments to try and deflect the attack. In the meantime we have restarted the web server and will continue to monitor the attack.

Additionally, the mail queue has now cleared.

NC018 web server stopped

24 August 2013 07:24:12 +0000

We have temporarily stopped the NC018 web server in an attempt to fight this attack.

Server NC018 under attack again

24 August 2013 07:08:51 +0000

As the title says, it appears that server NC018 is again the target of an attack. We are working to address the issue as quickly as possible.

Server NC018 denial of service attack

23 August 2013 20:55:10 +0000

Server NC018 suffered a denial of service attack beginning at approximately 07:40 on 23 August, UTC. This continued until the server succumbed under the load at about 12:40. This affected all functions of the server: websites, sending and receiving emails, databases, etc.

Unfortunately we did not respond to this situation in a timely manner. This was partially because automated monitoring of server functions had been turned off due to an increasing number of false alarms from the system.

Once the server was rebooted at 13:47 and the attack mitigated, the server then still had a backlog of several thousand emails to process. This took several hours, with the queue being cleared by about 18:35. While emails were “missing” for several hours, no emails should have been lost; they were just delivered several hours late.

We sincerely apologise for the disruption caused by this event. We will reinstate monitoring and the associated actions that are taken when monitoring alarms go off.

We will shortly be responding to all emails and phone messages regarding this incident. If you have any further questions about this event, please contact us.

NinerNet home page

Systems at a Glance:


SystemStatusPing
NC020OperationalPing
NC023OperationalPing
NC024InternalPing
NC027OperationalPing
NC028InternalPing
NC031OperationalPing
NC033OperationalPing

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

General Information:

This blog provides information about the status of NinerNet Communications systems. Dates and times of posts to this blog are in the UTC time zone, and dates and times given for events are also in the UTC time zone, although conversions may be offered for some time zones common to our clients. Please use the World Time Server to ensure accurate conversion of dates and times to your own time zone.

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains browser warnings connection issues dns dos attack dot-zm domains down time email delivery ftp hardware mailing lists mail relay migration nameservers performance phplist reboot smtp spam spamassassin ssl ssl certificate tls tls certificate web server

Resources:

Unable to open ninernet_xsite: Access denied for user 'ninernet_x_site'@'nc020.ninernet.net' to database 'ninernet_xsite'