NinerNet Communications™
System Status

Server and System Status

Server NC018 denial of service attack

23 August 2013 20:55:10 +0000

Server NC018 suffered a denial of service attack beginning at approximately 07:40 on 23 August, UTC. This continued until the server succumbed under the load at about 12:40. This affected all functions of the server: websites, sending and receiving emails, databases, etc.

Unfortunately we did not respond to this situation in a timely manner. This was partially because automated monitoring of server functions had been turned off due to an increasing number of false alarms from the system.

Once the server was rebooted at 13:47 and the attack mitigated, the server then still had a backlog of several thousand emails to process. This took several hours, with the queue being cleared by about 18:35. While emails were “missing” for several hours, no emails should have been lost; they were just delivered several hours late.

We sincerely apologise for the disruption caused by this event. We will reinstate monitoring and the associated actions that are taken when monitoring alarms go off.

We will shortly be responding to all emails and phone messages regarding this incident. If you have any further questions about this event, please contact us.

Server NC018 issues

23 August 2013 15:54:34 +0000

Server NC018 appears to be suffering the effects of a denial of service attack. We are working to address the issue.

Server NC018 replacement: Update 10

8 May 2013 01:41:27 +0000

It took 20 hours, but we have transferred all of the data from the old server (NC018) to the new server (NC026). However, now we have some manual configuring to do to address the errors in the automated process.

As always, we’ll post further updates here.

Server NC018 replacement: Update 9

7 May 2013 07:12:49 +0000

Apparently the first 50% was the quick part. The next 50% is going to take much longer. However, the good news is that most of the processing has now moved to the new server, and none of the operations on the current (old) server have been affected.

Server NC018 replacement: Update 8

7 May 2013 06:07:22 +0000

We are just over 50% of the way through our test. At the current rate, we should be finished by about 06:55 UTC.

Server NC018 replacement: Update 7

7 May 2013 05:04:51 +0000

We are currently running a migration test on server NC018. You may experience intermittent access to services.

We will provide updates here as the test progresses, and when it is concluded.

Server NC018 replacement: Update 6

3 May 2013 07:11:33 +0000

We are ready to begin the process of migrating accounts from server NC018 to NC026. We will start with websites on Friday, 3 May at 19:00 UTC.

Once the websites have been migrated, we anticipate that the reduced load on server NC018 will permit re-enabling anti-spam measures. This, I’m sure, will be a welcome relief for all.

Server NC018 replacement: Update 5

22 April 2013 05:42:31 +0000

We have spent this weekend configuring and testing the new server (NC026). Our next step is to conduct a test migration to determine how long it will take and what possible issues may arise that we’ll need to address.

When we have done this, we will formulate a migration plan, determine a window in which the migration will take place, and communicate that information to you.

Once that’s done, we’re looking forward to having a much newer, much faster, more resilient, more secure, and more configurable server. I know you are too.

Server NC018 replacement: Update 4

20 April 2013 22:55:35 +0000

On Thursday the 18th we set up a new server in a new data centre. We’re excited about the innovations in security that this new configuration will bring to your hosting account.

We are working diligently to configure and test this server, and test the migration process to minimise the time it will take. Due to issues on the old server, our priority is to get the migration done as soon as possible. While we had hoped to do this over this weekend, we may need to do it during the week. Either way we will ensure that any disruption will be kept to a minimum.

We will post further updates here over the course of the next few days.

Server NC018 replacement: Update 3

21 March 2013 18:59:03 +0000

Contrary to previous information, the replacement server will not be ready until mid to late next week.

The good mews is that we have taken further steps to mitigate the performance issues, and we have seen no evidence of performance problems in the last 48 hours.

We’ll post another update here when we have firmer time lines to tell you about.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: