We are experiencing problems with server NC031. These are caused by a very busy website that we transferred to this server about 48 hours ago.
We are assessing the situation and will post updates here. We apologise for the problem.
We are experiencing problems with server NC031. These are caused by a very busy website that we transferred to this server about 48 hours ago.
We are assessing the situation and will post updates here. We apologise for the problem.
Currently email to Yahoo domains is getting through. We are closely monitoring this situation, but only expect to provide major updates from this point forwards.
Thank-you again for your patience with this issue.
Yahoo finally replied to our request for information about why they are blocking us by providing a canned response which didn’t tell us anything we didn’t already read on their website. But they did apologise: “I want to apologize for any inconvenience this issue has caused you.”
Shortly after 01:30 UTC on 31 March we did see all emails still queued on our server for addresses on Yahoo and Yahoo-hosted domains clear — i.e., they were delivered. However, a few hours after that the block at Yahoo based on “user complaints” seems to have been reinstated, and so we are back to square one.
We have again written to Yahoo and are continuing to research possible workarounds … while we wait another day or two for their possible response.
In addition to researching workarounds, we’re taking some of Yahoo’s reasonable advice from their canned email and moving up certain changes to mail server configuration policies that they suggest. However, these cannot be implemented overnight across all clients’ domains without causing even more problems for our clients, so they will be introduced gradually.
We sincerely apologise for this inconvenience, but we cannot take responsibility for Draconian policies implemented at a very large email provider that has drastic effects on small hosting companies like NinerNet and our clients, and that are seemingly out of step with comparable large email providers like Gmail, Hotmail, AOL, etc. We’ve been in business for two decades — only two years less than Yahoo — and are not an upstart spamming operation that opened business last night and will be gone by next week. Clearly this kind of information is not being taken into account by Yahoo.
This is yet another example of why — on a network that requires cooperation between disparate entities, which is the very definition of the Internet — we shouldn’t do business with massive companies that can steamroll their way over the ability of smaller entities to participate. Signing up for a service from Yahoo (free or paid) or the parent companies of any of the other corporations mentioned in this post might be quick and easy (and probably even “free” of monetary cost), but there is always a price to be paid down the road. We and you are paying it today.
The workaround we’re working on may solve this issue, but there is still the risk that Yahoo will notice what we’ve done and block that avenue too. If you’re forwarding email from your domain to a Yahoo account, we strongly suggest that you stop doing that. If you’re doing that or using personal, free accounts for business use on Yahoo or other free email providers, we also strongly suggest you stop doing that. If you need help achieving — with your own domain and the hosting services you’re already paying for — what you’re currently doing with free email accounts with third-party providers, talk to us and we’ll help you do it a better way.
Further updates will continue to be posted here as they become available.
We have become aware of the fact that Yahoo is either delaying or, in come cases, outright refusing email from our primary mail server. This seems very odd, as our mail server’s IP address is not in any of the major anti-spam blacklists, and other major email providers (e.g., AOL, Gmail, Hotmail, Apple, etc.) are not blocking us.
We have been unsuccessful in trying to contact Yahoo about this as, like most massive companies, they put up massive barriers to communication from small companies or individuals.
An obvious exception to this that we have happily dealt with before to solve similar issues is AOL, who provide an excellent website where companies like NinerNet can look up the “reputation” of their mail servers’ IP addresses, and initiate communication with people who are willing and able to help address issues. Interestingly, this is AOL’s current report on the reputation of the mail server that Yahoo is blocking:
212.71.255.195 [nc027.ninernet.net] Reputation is Good
What does a good reputation mean?
The IP address has a good reputation. This means AOL has enough data to determine that the IP address is a known mail sender with a good reputation. You should not be experiencing bounce or reject errors when sending mail from this IP.
We will monitor this situation and make further attempts to contact Yahoo and have them explain or remove this block. In the meantime, this serves as yet another example of why using “free” email on monolithic systems run by massive companies is a bad idea.
We’ll be sending out an email shortly to informĀ NinerNet clients that over the next few weeks we will be moving all of the shared hosting websites hosted on servers NC018, NC020 and NC026 to a new server, NC031. While this might look like one server is taking on the workload of three, that’s not quite the case; servers NC020 and NC026 are currently hosting only a relatively few websites that had outgrown the limitations of NC018. On top of that, server NC031 is a shiny new server, running the latest commercially available versions of all software, with far more resources than the servers it is replacing, and an increasing emphasis on secure connections.
The new server is also running a new control panel. The one on server NC018 (Plesk) has served us well. In fact, server NC026 was running the latest version of Plesk when it was set up a couple of years ago, but our experience with the new version has — to put it simply — been disastrous. In fact, we’ve gone out of our way to avoid clients actually using it. We evaluated another popular control panel — cPanel — but we came away with much the same opinion as we’ve developed of Plesk, but for different reasons.
The new control panel we’re using is called Virtualmin, and is based on a control panel (Webmin) that we first used privately in 2006 but which has been around since 1997 … almost as long as NinerNet! There will no doubt be a period of adjustment as we switch — for you and for us, especially as we develop support documentation — but we’re confident that Virtualmin is far better than other control panels that we have evaluated, and we’ve been impressed by their responsive support so far.
With that background information out of the way, here are a few points you need to know:
One other item to note is that we’re really emphasising the importance of complex passwords. These have always been important to us, but our older systems haven’t always enforced complex passwords. Complex passwords are vital to the security of everything you do online that requires you to log in to view private data (such as your email or bank account) or manage services that unauthorised users should not access (such as the back-end of your website). This is why we set longer (a minimum of 12 characters) and more complex passwords by default. To manage long and complex passwords we use and recommend a program called KeePass, which has versions and clones made for all major systems — e.g., Windows, Linux, Mac, Android and iOS (iPhone/iPad). We strongly recommend this program or one like it. It will make your life so much easier, and more secure too.
As always, we strive to ensure that your hosting (and the move to the new server) are without any headaches, and that attitude will apply to this move as well. Please feel free to contact us if you have any questions or concerns. Thank-you.
The data centre informs us that connectivity on their network was restored at 09:24 UTC.
We have been able to confirm this by accessing all of our servers in this data centre:
We apologise for this interruption. Denial of service attacks (especially distributed denial of service attacks) are extremely difficult and expensive to fight. We have confidence that the data centre did everything they could as quickly as they could to combat this attack.
We have been updated that the denial of service attack against the London data centre is actually a large distributed denial of service (DDoS) attack, and it is still underway. Efforts are being made at multiple levels on the network to combat this attack, and we are seeing some movement of email through server NC027, although it took an hour and forty-seven minutes for a test message to get through. This indicates, as mentioned earlier, that the server is up and processing email, but that other servers (and our clients, obviously) are having problems connecting to the server.
We apologise for these problems, but this is affecting a wide range and large number of companies hosted at this data centre, not just NinerNet.
Updates will continue to be posted here as we learn more.
The London data centre has been suffering from the effects of a couple of denial of service (DoS) attacks over the last 12 or so hours. The attack late yesterday (26 December) lasted about an hour and a half. Today’s attack started at 05:37 UTC and is currently ongoing. The data centre staff assure us that they have identified the problem and are implementing a fix for it.
This affects server NC023 (the relay server) and server NC027 (the main mail server), as well as virtual private servers hosted in London. As far as we know, the servers themselves are up and connected to the network, but the attack is causing them to be inaccessible.
We’ll update here as soon as we know more.
We have determined from monitoring logs that server NC020 was only offline for a few minutes, and was rebooted at 09:46 UTC on 20 November within a few minutes of the problem being discovered.
We apologise to those few clients still using server NC020 who may have been inconvenienced at that time.
If you have any questions about this, please let us know. Thanks.
We have rebooted server NC020 after it was overwhelmed. We’re looking into it right now and will post details here as soon as we have them.
Systems at a Glance:
Loc. | System | Status | Ping |
---|---|---|---|
NC023 | Internal | Up? | |
NC028 | Internal | Up? | |
NC031 | Internal | Up? | |
NC033 | Operational | Up? | |
NC034 | Internal | Up? | |
NC035 | Operational | Up? | |
NC036 | Operational | Up? | |
NC040 | Internal | Up? | |
NC041 | Operational | Up? | |
NC042 | Operational | Up? |
Subscriptions:
Search:
Recent Posts:
Archives:
Categories:
Links
Tags:
Resources:
On NinerNet: