NinerNet Communications™
System Status

Server and System Status

NC036: Evaluating the effectiveness of being listed in the UCEPROTECT whitelist

19 July 2024 03:57:16 +0000

Just about a month ago we became aware of the problem with delivering email to Microsoft-hosted domains. At that time, one of the actions we took was to pay to have our mail server’s IP address listed in the UCEPROTECT whitelist that effectively removed our IP address from a huge list of blacklisted IP addresses that are only listed because of the lackadaisical approach of our data centre (Digital Ocean) to removing spammers from their network of data centres.

We weren’t certain at the time that this action would achieve anything and, to be frank, we have no concrete evidence to believe that our doing so has had the desired effect. We think there is strong evidence that it has, but there is no way we can determine that definitively. The strong evidence is that we have had very few problems with mail to Microsoft-hosted domains since a couple of days after the problem started on 20 June.

Our subscription to this whitelist expires on Monday the 22nd. To test the effectiveness of this subscription we are going to allow it to lapse. If we see a sudden uptick in email messages to Microsoft-hosted domains bouncing then we’ll take that as evidence that the subscription is working, and we’ll immediately renew it. In that case, messages you send to such domains will bounce temporarily, so please forward those bounce messages to NinerNet support and we will immediately divert future messages to those domains via our secondary outbound mail servers. This is a temporary and planned test; if these bounces happen it is not evidence that our systems are failing for any reason. If they don’t bounce then we will have learned that our paying for the listing is a waste of money.

The purpose of this message is to let you know of this test in advance. We expect that if messages start bouncing again and we renew the subscription, everything will be back to normal within about 24 hours, and immediately for domains we add to our mail server configuration to have messages go out via our secondary mail server. That said, we have also found that retrying if you get a bounce in this situation sometimes succeeds! It’s bizarre.

Thank-you for your continued patience with issues like this thrown at us by massive providers that don’t care about you, people who choose to host with their competitors.

NC036: Significant issue with delivery of email to Microsoft-hosted domains

21 June 2024 03:51:33 +0000

Yesterday, 20 June 2024, multiple clients began contacting us to report that email they were sending to certain domains was bouncing. We responded as usual by routing messages to the problem domains via our secondary SMTP server. In other words, this wasn’t an unusual experience, and we mitigated it immediately as we always do.

However, it very quickly became apparent that all of the problem destination domains in these multiple reports were hosted by a single hosting provider: Microsoft … or Outlook, or Hotmail, or however they’d like to be known today.

As we’ve said to our clients for many years, fighting spam is a never-ending battle. It’s a big issue for hosting companies big and small; however, the power of small hosting companies like NinerNet to deal with massive companies like Microsoft, Gmail, Yahoo, etc., is almost non-existent. Actually, it’s not almost non-existent, it is totally non-existent. For many years NinerNet has been (and still is) a member of or participant in the Outlook.com “Smart Network Data Services” system. This was supposed to give small providers like NinerNet access to the decision makers at Microsoft (the so-called postmaster[s]) so that we could work out issues together as if we were all grown-ups. However, it has never actually worked that way. Instead, the big hosting providers listed above treat companies like NinerNet with disdain. After all, we’re competitors, and every client who hosts with NinerNet takes away revenue from the big boys.

The rest of this blog post is lengthy, and goes into a fair bit of detail. The summary is that a huge email hosting provider (Microsoft) has suddenly made sending email to them very difficult for us, but NinerNet has done and is doing everything we can to provide a working service to our clientele.

Today we find we’re in a situation where one of the biggest email hosting companies in the world — owned and run by Microsoft — is refusing email from companies like NinerNet. This is anti-competitive, which you wouldn’t expect from a company headquartered in a country where the competitive marketplace is supposed to trump (pardon the pun) everything else.

The bounce messages generated by the failed deliveries offer a “delisting” service, purely because Microsoft seems to actually realise that they have acted with a very heavy hand in this instance. However, when we tried for the third time to get our mail server’s IP address delisted, the automated response we got was that, “The IP address in question is not currently blocked in our system.” This is interesting.

What we believe has happened here is that Microsoft are using a blacklist that includes every single one of the IP addresses owned by the company where a number of our servers (including our primary mail server) are physically located, and have been located for about eight years. This company is Digital Ocean. Why are all of Digital Ocean’s IP addresses blacklisted? Good question. The summary seems to be that Digital Ocean has no interest in dedicating resources to keeping spammers off of their servers. This results in their telling their customers (like NinerNet) that they should send all email out via third parties. This is a ridiculous and expensive requirement, of course, because that is not how the Internet was designed several decades ago, and it’s not how NinerNet operates or has ever operated. When this requirement was forced on us by another data centre company many years ago (Interland), we refused and moved our business elsewhere. For sometime now we have known that the data centre for our next mail server would not be a Digital Ocean data centre but, strangely enough, Microsoft didn’t give us any notice of this change in their practices. And as you know if you’ve been a NinerNet client for any length of time, moving email hosting to a new server is no small undertaking.

The result of Digital Ocean doing nothing to keep spammers out of their data centres is that their IP addresses (including ours) have been elevated from UCEPROTECT Level 0, to Level 1, to Level 2 and finally (over time) to Level 3. UCEPROTECT describes Level 3 as listing the “IP Space of the worst ASNs”. (An ASN is a “Autonomous System Number”, “an identifier for a collection of IP networks and routers under the control of one entity”. [Wikipedia.]) So NinerNet’s mail server is in a blacklist, not because of something we or one of our clients have done (or not done), but because Digital Ocean fails to do anything to keep spammers off of their systems.

For sometime we have known about the fact that UCEPROTECT has a system by which companies like NinerNet, who have no track record of providing safe harbour to spammers, can have their IP address(es) whitelisted, so that we are essentially excluded from the Level 3 blacklisting of all Digital Ocean IP addresses. Previously we chose not to do this because of the added expense, and we preferred to spend money on other ways (described in the first paragraph of this post) of mitigating this problem. However, we have broken down and paid a fee to UCEPROTECT to have our IP address whitelisted.

Therefore, if we are correct in deducing the cause of the current problem, we expect that email to domains hosted by Microsoft will be delivered without hindrance starting by about 04:18 UTC today, 21 June 2024.


Update, 2024-06-22: We thought that our having paid for an exception to the UCEPROTECT blacklist had solved the problem. And it does seem to have solved the problem, for the most part. However, very oddly, messages to only some Microsoft-hosted domains are still being blocked with the exact same bounce message that directs senders to their article, “External senders – Use the delist portal to remove yourself from the blocked senders list and address 5.7.511 Access denied errors” at http://go.microsoft.com/fwlink/?LinkID=526655, which redirects to https://learn.microsoft.com/en-gb/defender-office-365/external-senders-use-the-delist-portal-to-unblock-yourself?redirectedfrom=MSDN. (The 5.7.511 error in the title does not appear to apply to the messages bounced from our server, as those errors are 5.7.1.) However, every time we try to have our mail server’s IP address delisted, the response we receive is, “The IP address in question is not currently blocked in our system.” So why are messages being blocked?!

This seems to be a ridiculous game of cat-and-mouse that Microsoft are playing instead of being open with people about what they are doing, and companies like NinerNet cannot do anything to counter that. It makes absolutely no sense, and doesn’t serve Microsoft, their customers, or NinerNet or our customers.

So in these circumstances, if you’re still having messages to Microsoft-hosted domains bounced — you will know if you see references to Outlook(.com) and Microsoft(.com) in the bounce message — please forward the bounce message(s) to NinerNet support and we will add the problem domains to the mail server configuration that redirects messages sent to those domains via our secondary SMTP server. This is the same procedure that we followed previously, but we were hoping to avoid that procedure by buying our way out of the UCEPROTECT blacklist. However, at least now the number of Microsoft-hosted domains that we have to add to our mail server configuration should be far less than previously.

Again, we apologise to you, our clients, for this non-consensual position in which Microsoft has put us and many small hosting companies around the world.

Update, 2024-06-28: Over the last week we have added a grand total of 21 domains to our mail server’s configuration to redirect outgoing messages to them via our secondary mail server. In that time we have learned that there is no consistency to the problem. Sometimes mails that are blocked are delivered five minutes later if the sender retries, without our adding that domain to our mail server’s configuration. And delivery succeeds to some Microsoft-hosted domains consistently without any intervention by us. There’s nothing more frustrating than an inconsistent problem that is not possible to troubleshoot.

So at this point it seems that we are back to the point we were at before this incident started. Here is a summary of what has transpired:

  • All emails to Microsoft-hosted domains started bouncing.
  • We paid to be removed from a blacklist that we thought might be the cause.
  • This seemed to help to some extent, but over the course of the next few days we added 21 destination domains to our mail server’s configuration to direct messages to those domains via our secondary mail server.
  • We are still exploring alternative ways of automatically determining the MX record of destination domains and automatically redirecting mail to Microsoft-hosted domains via our secondary mail server.
  • We are also still looking for ways to contact Microsoft to determine the cause of this issue, but we hold out little hope of doing that.
  • This server will be replaced in the near term. To that end we will be looking for a data centre where we won’t run into the issue of all of their IP addresses being blacklisted as is the case where our primary mail server is currently located on a Digital Ocean IP address.

As always, if you have mail you send bounced by Microsoft, please forward the bounce message to support and we will add the destination domain to our mail server’s configuration. We appreciate your patience and continued patronage.

NC023: Maintenance complete

27 March 2021 20:32:10 +0000

Maintenance on server NC023 has completed successfully. We have tested its relay features and they are working correctly.

That said, there will be more upgrades performed on this server in the not-too-distant future. However, there will be plenty of notices of this work.

Thank-you for your patience. If you have any question, please contact NinerNet support.

NC023: Maintenance underway

27 March 2021 19:32:47 +0000

Maintenance on server NC023 is about to begin, about half an hour late. Apologies for the delay.

NC023: Relay server maintenance

26 March 2021 07:11:25 +0000

The relay mail server will be down briefly for some maintenance during our weekly maintenance window this weekend. This will start at 19:00 UTC on Saturday 27 March 2021, and should be complete within the hour.

We will post an update when the maintenance starts and when it is finished. This does not affect our primary mail server; it only affects you if you know that you use the relay server for outbound email.

If you have any questions, please contact NinerNet support. Thanks very much for your patience and understanding.

Emails to Microsoft domains delayed

12 March 2021 04:16:41 +0000

Since about 05:00 UTC on 11 March we have noticed that emails to Microsoft’s free email domains — hotmail.com, outlook.com, msn.com, live.com and various other domains in various other TLDs — have been and are being significantly delayed, sometimes by twelve hours or more. This is not a problem with our mail servers; it is Microsoft that are deferring and refusing deliveries. If you look at their “service health” page, they claim that “Everything is up and running”, but this is not accurate — or true.

We are monitoring the situation, but there is little to nothing that we can do. We ask you to have patience when communicating with people who use this free email service, and if Microsoft cared to communicate with the public, we’re sure they’d ask that too.

NC036: Mail server is back online

21 September 2020 07:26:54 +0000

Our apologies. The sending side of the mail server (NC036) is back up. It was down for 21 minutes between 06:56 and 07:17 UTC. The ability to check your email account was not down.

We will post additional information and contact the affected clients shortly.

NC036: Emails to Hotmail/Outlook being redirected

22 May 2020 07:03:49 +0000

Some clients may have had emails they sent to Hotmail, Outlook and other Microsoft email service domains bounced in the last 24 hours. This is because the primary mail server’s IP address has been blacklisted by Microsoft, despite our being enrolled in their “Junk Mail Reporting Program”. Our being enrolled is supposed to result in our being informed of spam complaints (which are overwhelmingly false) before our IP address is blacklisted, but their system seems to have been broken for the last few years.

Until we can get in touch with Microsoft support to have our IP address removed, or until our listing expires, we are automatically directing all email to their domains through our relay server, which is not blacklisted.

If you had an email bounced, please resend it. Thank-you, and sorry for the hassle. If you have any questions please contact NinerNet support.

NC036: Maintenance complete

10 June 2019 15:11:11 +0000

Our scheduled maintenance to increase the storage space available for email accounts on the mail server (NC036) is complete, after a delay completing the final step. The ability to send and receive email was not available between 19:23 and 20:19 UTC on 8 June. The expanded disk space has been tested and all is running normally.

If you have any issues or questions, please contact NinerNet support. Thank-you.

NC036: Scheduled maintenance underway

8 June 2019 19:15:56 +0000

The scheduled maintenance to increase hard drive space on server NC036 is underway. We will post progress reports or notice that the maintenance is complete as necessary.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: