There is a significant number of messages on our mail server (NC036) queued up to be delivered to domains (mostly in South Africa) that use the Mimecast mail filtering service. These will be delivered when their service is back online again.
There is a significant number of messages on our mail server (NC036) queued up to be delivered to domains (mostly in South Africa) that use the Mimecast mail filtering service. These will be delivered when their service is back online again.
There is planned network maintenance, “to improve performance and scalability”, for the data centre in Amsterdam where server NC036 (our primary mail server) is located. This maintenance is not expected to result in any downtime, but if any issues arise “affected [servers] may experience increased latency or a brief disruption in network traffic.” This will take place on Tuesday 27 February 2024 from 16:00 to 20:00 UTC. This is 08:00 to 12:00 Pacific Time and 18:00 to 22:00 Central Africa Time on the same date.
If you’d like to calculate this for any other location, please use the World Time Server.
We only post this to ensure you’re aware of the reason for any possible issues in advance. However, based on previous experience, we don’t anticipate any issues.
If you have any questions, please do contact NinerNet support. Thank-you.
We’ve been meaning to post about this for several years (since 2020), but we haven’t because there’s always something more important to do and it’s not our job to highlight how badly a competitor is running their mail servers. However, after the issue had gone away for a time we thought that perhaps Xneelo (formerly Hetzner South Africa) had resolved their problem. But now it’s back.
The problem is that one of the multiple mail servers they run is in multiple anti-spam blacklists/blocklists. We don’t know how many outbound mail servers they run, but let’s say (for the sake of example) it’s ten. If the IP address of one of those servers is in a blacklist, and all of their outbound mail is equally spread among those ten servers, then 10% of their outbound messages will not be delivered to anyone that is using the blacklists in which that server is listed.
The problem for us is that we end up devoting a significant percentage of our support resources to answering questions from clients who don’t understand what is going on, and who think that we are the problem. But we’re not!
Getting into an anti-spam blacklist is a significant event for any company, but the bigger you are and the more servers you run the less of an issue it is. NinerNet is not big, and so if one of our servers gets into a blacklist it’s a big deal, and we jump around to fix the problem and have our IP address removed from the blacklist. Considering how long one of Xneelo’s mail servers has been in a major blacklist, and how many of their clients we (or our clients) have told to take the problem to Xneelo, it’s shocking to us that they don’t seem to have done a thing about it. So we’re making this blog post to try and bring their attention to it publicly and to point our clients here whenever they have a question about the problem.
If one of your correspondents tells you that their email messages to you are not getting through to you, and they’re a Xneelo customer, this is almost certainly the reason. Your correspondent — the Xneelo client — needs to go to Xneelo to demand that Xneelo resolve their years-old problem in order to serve their own clientele properly.
Update, 2024-03-01: A Xneelo client that was failing to correspond with one of our clients engaged Xneelo support on 28 February 2024, and we are in ongoing discussions with Xneelo for them to fix this problem. Xneelo have acknowledged that at least two of their outgoing mail servers (197.189.244.82 and 197.189.244.90) are in six blacklists between them, so the situation is actually far worse than the hypothetical example given above. When the issue is resolved we will post an update here.
Update, 2024-03-05: Almost a week later and Xneelo are still spouting excuses and fabrications, and assuming that the people they are addressing (including us!) are too stupid to understand how email works. This may or may not be resolved at some point, but at this point it seems it likely won’t be. Email service providers should cooperate with one another to reduce spam across the Internet, and they should communicate truthfully with one another, but that seems to be too much of a challenge for Xneelo. We’d love to run our mail servers without having to consult blacklists of spammers, but your email account would become instantly useless if we did. All we can suggest you do is to advise your correspondents to use email hosting providers that take care of the reputations of their IP addresses, as otherwise their email will be tainted with the same negative reputation. This is how the combined actions of hosting clients are supposed to drive the bad actors out of business.
Update, 2024-03-14: It has emerged that Xneelo doesn’t even handle greylisting properly! One of the parties involved in the ticket with Xneelo decided to communicate with us directly (to become a client, ironically), but because Xneelo’s mail servers don’t respond correctly to NinerNet’s use of the well-established anti-spam technique greylisting, their messages to us are now bouncing! Apparently this client of Xneelo has done the right thing and told Xneelo that they will be moving their hosting business away if they don’t fix the problem. It blows us away how badly Xneelo is running their mail servers!
Update 2, 2024-03-14: We have added various IP addresses, server names and domains to various whitelists on our mail servers. Adding the blacklisted IP address will not have any effect on messages from Xneelo’s blacklisted servers that are listed in the blacklists our anti-spam system consults, but these additions are, as far as we can tell so far, having a positive effect on Xneelo’s inability to handle greylisting. If you communicate with a South African correspondent who is hosted by Xneelo and their emails to you are bouncing or taking hours to come through, please log into the mail server control panel and add their domain at Domains and Accounts -> YOUR_DOMAIN -> Greylisting -> Do not apply greylisting on listed senders. If your correspondent is susie@example.com, you need to add “@example.com” (with the leading @ symbol) to that text box, and click the green “Save changes” button.
Update, 2024-03-25: We have not heard back from Xneelo in three weeks, and it seems clear that we won’t. So our advice remains the same: If one of your correspondents tells you that their email messages to you are not getting through to you, and they’re a Xneelo customer, this (everything explained above) is almost certainly the reason. Your correspondent — the Xneelo client — needs to go to Xneelo to demand that Xneelo resolve their years-old problem in order to serve their own clientele properly. But please do also contact NinerNet support and we will assist you to do all that can be done to help the Xneelo mail servers get their messages through to you. However, the situation described above — the two IP addresses above are still in multiple blacklist — is still just as true today as it was when we first posted this in January.
Update, 2024-12-07: When Xneelo customers contact Xneelo support now, Xneelo are blaming NinerNet for this problem by stating:
The cause of the delivery failure is due to the recipient blacklisting your email address. This can be seen in the delivery failure itself: Recipient address rejected: Blacklisted.
This is not an issue on your phone or an issue with the server. The error would need to be investigated by the recipient themselves as they are the ones rejecting the emails.
As is very clearly demonstrated above, this is completely false, and shows a complete lack of understanding on the part of Xneelo of how Internet email and widely circulated and used blacklists work. This is surprising considering this untruth is being perpetuated by someone who labels himself as a “Senior Technical Support Consultant”! The fact is that NinerNet does not normally block individual email addresses, as this is generally pointless because spammers change their email addresses all the time, often with every email message. We block the IP addresses of mail servers (including Xneelo’s) that spew spam and malware.
If you’re hosting your domain with Xneelo, you’re entrusting it to incompetents. That’s a poor business decision.
The *.niner.net wildcard SSL/TLS certificate has been renewed and updated across our entire infrastructure, although we apologise for being about four hours late. This should be seamless for all our clients. However, should you run into a situation in the next 24 hours where you’re told that the certificate has expired, please log out and reload whatever it is you’re trying to do. You may need to reboot, but this would be very unusual. Simply forcing a reload should clear things up. Via FTPS you may need to re-trust our certificate depending on your FTP client.
This doesn’t affect certificates on your own domain on your website, just NinerNet services such as control panels and connecting to the mail and FTP servers.
If you have any questions or concerns, please do contact NinerNet support with any error messages you may be seeing. Thank-you.
Server NC036, our primary mail server, was apparently down for 34 minutes yesterday, 8 January. Our server itself was not down, but the data centre experienced what seems to have been a partial connectivity issue. In fact, you may not even have noticed it, as we didn’t receive any complaints about the downtime.
According to the data centre’s report, the issue was resolved and it is not expected to recur. We apologise to anyone who was affected.
Server NC041 has been rebooted. This appears to have dealt with the database server issue. We will continue to look for the root cause to ensure this doesn’t happen again.
The database server on server NC041 appears to be having some issues. We are looking into it and will post an update about what we find shortly.
Today we received a number of complaints of connectivity problems to our mail server. However, they were all from Zambia, from the Luangwa Valley to the Copperbelt to Lusaka; nobody else had any issues.
Below is a screenshot of NinerNet’s live status page that was taken moments ago that shows that the mail server (NC036) has been up for 100% of the last 90 days.
In the absence of any other information we can only assume that there was some sort of widespread, country-wide connectivity problem in Zambia late today, for which NinerNet is clearly not responsible. The NinerNet mail servers continued to process inbound email during this time, so even if you couldn’t connect to the mail server, incoming email to you was still collecting on the mail server for your eventual collection.
If you have any questions or concerns, please feel free to contact NinerNet support. Thank-you.
Update, 2023-05-19: One of our clients contacted a Zambian ISP and the problem was blamed on Zambia’s international gateway. Sorry for that!
The *.niner.net wildcard SSL/TLS certificate has been renewed and updated across our entire infrastructure. This should be seamless for all our clients. However, should you run into a situation in the next 24 hours where you’re told that the certificate has expired, please log out and reload whatever it is you’re trying to do. You may need to reboot, but this would be very unusual. Simply forcing a reload should clear things up. Via FTPS you may need to re-trust our certificate depending on your FTP client.
This doesn’t affect certificates on your own domain on your website, just NinerNet services such as control panels and connecting to the mail and FTP servers.
If you have any questions or concerns, please do contact NinerNet support with any error messages you may be seeing. Thank-you.
Please note that we are seeing messages from NC036 destined for private domains hosted by Microsoft delayed at this time. The cause is not clear, but it’s clearly on Microsoft’s end.
This is not in any way related to the internal server issue that happened yesterday.
Systems at a Glance:
Loc. | System | Status | Ping |
---|---|---|---|
NC023 | Internal | Up? | |
NC028 | Internal | Up? | |
NC031 | Internal | Up? | |
NC033 | Operational | Up? | |
NC034 | Internal | Up? | |
NC035 | Operational | Up? | |
NC036 | Operational | Up? | |
NC040 | Internal | Up? | |
NC041 | Operational | Up? | |
NC042 | Operational | Up? |
Subscriptions:
Search:
Recent Posts:
Archives:
Categories:
Links
Tags:
Resources:
On NinerNet: