NinerNet Communications™
System Status

Server and System Status

Capacity issues on server NC031

7 April 2016 07:42:20 +0000

We are experiencing problems with server NC031. These are caused by a very busy website that we transferred to this server about 48 hours ago.

We are assessing the situation and will post updates here. We apologise for the problem.

Yahoo delivery working

31 March 2016 07:29:23 +0000

Currently email to Yahoo domains is getting through. We are closely monitoring this situation, but only expect to provide major updates from this point forwards.

Thank-you again for your patience with this issue.

Yahoo delivery update

31 March 2016 04:42:03 +0000

Yahoo finally replied to our request for information about why they are blocking us by providing a canned response which didn’t tell us anything we didn’t already read on their website. But they did apologise: “I want to apologize for any inconvenience this issue has caused you.”

Shortly after 01:30 UTC on 31 March we did see all emails still queued on our server for addresses on Yahoo and Yahoo-hosted domains clear — i.e., they were delivered. However, a few hours after that the block at Yahoo based on “user complaints” seems to have been reinstated, and so we are back to square one.

We have again written to Yahoo and are continuing to research possible workarounds … while we wait another day or two for their possible response.

In addition to researching workarounds, we’re taking some of Yahoo’s reasonable advice from their canned email and moving up certain changes to mail server configuration policies that they suggest. However, these cannot be implemented overnight across all clients’ domains without causing even more problems for our clients, so they will be introduced gradually.

We sincerely apologise for this inconvenience, but we cannot take responsibility for Draconian policies implemented at a very large email provider that has drastic effects on small hosting companies like NinerNet and our clients, and that are seemingly out of step with comparable large email providers like Gmail, Hotmail, AOL, etc. We’ve been in business for two decades — only two years less than Yahoo — and are not an upstart spamming operation that opened business last night and will be gone by next week. Clearly this kind of information is not being taken into account by Yahoo.

This is yet another example of why — on a network that requires cooperation between disparate entities, which is the very definition of the Internet — we shouldn’t do business with massive companies that can steamroll their way over the ability of smaller entities to participate. Signing up for a service from Yahoo (free or paid) or the parent companies of any of the other corporations mentioned in this post might be quick and easy (and probably even “free” of monetary cost), but there is always a price to be paid down the road. We and you are paying it today.

The workaround we’re working on may solve this issue, but there is still the risk that Yahoo will notice what we’ve done and block that avenue too. If you’re forwarding email from your domain to a Yahoo account, we strongly suggest that you stop doing that. If you’re doing that or using personal, free accounts for business use on Yahoo or other free email providers, we also strongly suggest you stop doing that. If you need help achieving — with your own domain and the hosting services you’re already paying for — what you’re currently doing with free email accounts with third-party providers, talk to us and we’ll help you do it a better way.

Further updates will continue to be posted here as they become available.

Delivery issues to Yahoo domains

27 March 2016 12:11:55 +0000

We have become aware of the fact that Yahoo is either delaying or, in come cases, outright refusing email from our primary mail server. This seems very odd, as our mail server’s IP address is not in any of the major anti-spam blacklists, and other major email providers (e.g., AOL, Gmail, Hotmail, Apple, etc.) are not blocking us.

We have been unsuccessful in trying to contact Yahoo about this as, like most massive companies, they put up massive barriers to communication from small companies or individuals.

An obvious exception to this that we have happily dealt with before to solve similar issues is AOL, who provide an excellent website where companies like NinerNet can look up the “reputation” of their mail servers’ IP addresses, and initiate communication with people who are willing and able to help address issues. Interestingly, this is AOL’s current report on the reputation of the mail server that Yahoo is blocking:

212.71.255.195 [nc027.ninernet.net] Reputation is Good

What does a good reputation mean?

The IP address has a good reputation. This means AOL has enough data to determine that the IP address is a known mail sender with a good reputation. You should not be experiencing bounce or reject errors when sending mail from this IP.

Reputation of NinerNet mail server IP address with AOL.

Reputation of NinerNet mail server IP address with AOL.

We will monitor this situation and make further attempts to contact Yahoo and have them explain or remove this block. In the meantime, this serves as yet another example of why using “free” email on monolithic systems run by massive companies is a bad idea.

Web server upgrade

12 March 2016 12:33:12 +0000

We’ll be sending out an email shortly to informĀ NinerNet clients that over the next few weeks we will be moving all of the shared hosting websites hosted on servers NC018, NC020 and NC026 to a new server, NC031. While this might look like one server is taking on the workload of three, that’s not quite the case; servers NC020 and NC026 are currently hosting only a relatively few websites that had outgrown the limitations of NC018. On top of that, server NC031 is a shiny new server, running the latest commercially available versions of all software, with far more resources than the servers it is replacing, and an increasing emphasis on secure connections.

The new server is also running a new control panel. The one on server NC018 (Plesk) has served us well. In fact, server NC026 was running the latest version of Plesk when it was set up a couple of years ago, but our experience with the new version has — to put it simply — been disastrous. In fact, we’ve gone out of our way to avoid clients actually using it. We evaluated another popular control panel — cPanel — but we came away with much the same opinion as we’ve developed of Plesk, but for different reasons.

The new control panel we’re using is called Virtualmin, and is based on a control panel (Webmin) that we first used privately in 2006 but which has been around since 1997 … almost as long as NinerNet! There will no doubt be a period of adjustment as we switch — for you and for us, especially as we develop support documentation — but we’re confident that Virtualmin is far better than other control panels that we have evaluated, and we’ve been impressed by their responsive support so far.

With that background information out of the way, here are a few points you need to know:

  • As we set up your domain(s) in the new control panel you will receive an automated email with the log-in information you will need for the control panel and for FTPS (FTP Secure) for uploading files to your server. Please keep this information in a safe and secure place, for future reference. Your website will be moved shortly after that, but not necessarily immediately.
  • Where possible we will move your website without any coordination with you. Where coordination is necessary or desirable, we will contact you at least 24 hours in advance.
  • If you or your web manager, designer or developer would like to contact us in advance of moving your website to discuss any particular needs or requirements, please feel free to do so as soon as possible.
  • We will be starting with websites on server NC026, then server NC018, and lastly server NC020.
  • All files in the “web root” on the old server will be moved. Where necessary (rarely) we’ll rename existing directories if their names clash with directories set up by default on the new server — unless this will cause a problem with your website, in which case we’ll rename the directories on the new server. If you have any files stored outside of the “web root” — the “httpdocs” directory on NC018 and NC026, or the “htdocs” directory on server NC020 — please either download them to your local computer, or contact us so that we’re aware that they need to be moved too.
  • Speaking of the web root, the file path on the server to your website will change. The new path will be /home/USERNAME/public_html , where you will replace “USERNAME” with your actual user name, of course. Most of the time this is irrelevant, but for some web applications it might be important to know this.
  • If the nameservers for your domain are hosted elsewhere — i.e., we cannot access them to update the information for your domain to point it to the new web server — you will need to do this yourself. The IP address of the new server is 173.255.231.250; you will need to know this to point your domain to it after we transfer the files.
  • We will manually check to ensure that your site is online after the move and check a few additional pages, but please check yourself to make sure everything is good. Once a day we’ll email the owners of the websites that were moved that day.
  • Your website will not be down during the transfer. Where there is the possibility of complications, we will leave your domain pointing to the old website until any complications have been resolved.
  • This migration has nothing to do with email in any way. The email server is not involved, and your email will continue to flow uninterrupted while we migrate websites.

One other item to note is that we’re really emphasising the importance of complex passwords. These have always been important to us, but our older systems haven’t always enforced complex passwords. Complex passwords are vital to the security of everything you do online that requires you to log in to view private data (such as your email or bank account) or manage services that unauthorised users should not access (such as the back-end of your website). This is why we set longer (a minimum of 12 characters) and more complex passwords by default. To manage long and complex passwords we use and recommend a program called KeePass, which has versions and clones made for all major systems — e.g., Windows, Linux, Mac, Android and iOS (iPhone/iPad). We strongly recommend this program or one like it. It will make your life so much easier, and more secure too.

As always, we strive to ensure that your hosting (and the move to the new server) are without any headaches, and that attitude will apply to this move as well. Please feel free to contact us if you have any questions or concerns. Thank-you.

London network restored

27 December 2015 10:17:45 +0000

The data centre informs us that connectivity on their network was restored at 09:24 UTC.

We have been able to confirm this by accessing all of our servers in this data centre:

  • Incoming mail is once again being processed immediately by server NC027, although any incoming mail sent during the disruption may be delayed a little longer until the sending mail servers try again. This is not under our control.
  • Server NC023 is also processing email immediately.

We apologise for this interruption. Denial of service attacks (especially distributed denial of service attacks) are extremely difficult and expensive to fight. We have confidence that the data centre did everything they could as quickly as they could to combat this attack.

London data centre DDoS update

27 December 2015 08:30:17 +0000

We have been updated that the denial of service attack against the London data centre is actually a large distributed denial of service (DDoS) attack, and it is still underway. Efforts are being made at multiple levels on the network to combat this attack, and we are seeing some movement of email through server NC027, although it took an hour and forty-seven minutes for a test message to get through. This indicates, as mentioned earlier, that the server is up and processing email, but that other servers (and our clients, obviously) are having problems connecting to the server.

We apologise for these problems, but this is affecting a wide range and large number of companies hosted at this data centre, not just NinerNet.

Updates will continue to be posted here as we learn more.

Connectivity issue at London data centre

27 December 2015 06:52:18 +0000

The London data centre has been suffering from the effects of a couple of denial of service (DoS) attacks over the last 12 or so hours. The attack late yesterday (26 December) lasted about an hour and a half. Today’s attack started at 05:37 UTC and is currently ongoing. The data centre staff assure us that they have identified the problem and are implementing a fix for it.

This affects server NC023 (the relay server) and server NC027 (the main mail server), as well as virtual private servers hosted in London. As far as we know, the servers themselves are up and connected to the network, but the attack is causing them to be inaccessible.

We’ll update here as soon as we know more.

NC020: Earlier server reboot

21 November 2015 02:16:43 +0000

We have determined from monitoring logs that server NC020 was only offline for a few minutes, and was rebooted at 09:46 UTC on 20 November within a few minutes of the problem being discovered.

We apologise to those few clients still using server NC020 who may have been inconvenienced at that time.

If you have any questions about this, please let us know. Thanks.

NC020: Server rebooted

20 November 2015 10:07:39 +0000

We have rebooted server NC020 after it was overwhelmed. We’re looking into it right now and will post details here as soon as we have them.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: