Server and System Status

Yesterday, 20 June 2024, multiple clients began contacting us to report that email they were sending to certain domains was bouncing. We responded as usual by routing messages to the problem domains via our secondary SMTP server. In other words, this wasn’t an unusual experience, and we mitigated it immediately as we always do.

However, it very quickly became apparent that all of the problem destination domains in these multiple reports were hosted by a single hosting provider: Microsoft … or Outlook, or Hotmail, or however they’d like to be known today.

As we’ve said to our clients for many years, fighting spam is a never-ending battle. It’s a big issue for hosting companies big and small; however, the power of small hosting companies like NinerNet to deal with massive companies like Microsoft, Gmail, Yahoo, etc., is almost non-existent. Actually, it’s not almost non-existent, it is totally non-existent. For many years NinerNet has been (and still is) a member of or participant in the Outlook.com “Smart Network Data Services” system. This was supposed to give small providers like NinerNet access to the decision makers at Microsoft (the so-called postmaster[s]) so that we could work out issues together as if we were all grown-ups. However, it has never actually worked that way. Instead, the big hosting providers listed above treat companies like NinerNet with disdain. After all, we’re competitors, and every client who hosts with NinerNet takes away revenue from the big boys.

The rest of this blog post is lengthy, and goes into a fair bit of detail. The summary is that a huge email hosting provider (Microsoft) has suddenly made sending email to them very difficult for us, but NinerNet has done and is doing everything we can to provide a working service to our clientele.

Today we find we’re in a situation where one of the biggest email hosting companies in the world — owned and run by Microsoft — is refusing email from companies like NinerNet. This is anti-competitive, which you wouldn’t expect from a company headquartered in a country where the competitive marketplace is supposed to trump (pardon the pun) everything else.

The bounce messages generated by the failed deliveries offer a “delisting” service, purely because Microsoft seems to actually realise that they have acted with a very heavy hand in this instance. However, when we tried for the third time to get our mail server’s IP address delisted, the automated response we got was that, “The IP address in question is not currently blocked in our system.” This is interesting.

What we believe has happened here is that Microsoft are using a blacklist that includes every single one of the IP addresses owned by the company where a number of our servers (including our primary mail server) are physically located, and have been located for about eight years. This company is Digital Ocean. Why are all of Digital Ocean’s IP addresses blacklisted? Good question. The summary seems to be that Digital Ocean has no interest in dedicating resources to keeping spammers off of their servers. This results in their telling their customers (like NinerNet) that they should send all email out via third parties. This is a ridiculous and expensive requirement, of course, because that is not how the Internet was designed several decades ago, and it’s not how NinerNet operates or has ever operated. When this requirement was forced on us by another data centre company many years ago (Interland), we refused and moved our business elsewhere. For sometime now we have known that the data centre for our next mail server would not be a Digital Ocean data centre but, strangely enough, Microsoft didn’t give us any notice of this change in their practices. And as you know if you’ve been a NinerNet client for any length of time, moving email hosting to a new server is no small undertaking.

The result of Digital Ocean doing nothing to keep spammers out of their data centres is that their IP addresses (including ours) have been elevated from UCEPROTECT Level 0, to Level 1, to Level 2 and finally (over time) to Level 3. UCEPROTECT describes Level 3 as listing the “IP Space of the worst ASNs”. (An ASN is a “Autonomous System Number”, “an identifier for a collection of IP networks and routers under the control of one entity”. [Wikipedia.]) So NinerNet’s mail server is in a blacklist, not because of something we or one of our clients have done (or not done), but because Digital Ocean fails to do anything to keep spammers off of their systems.

For sometime we have known about the fact that UCEPROTECT has a system by which companies like NinerNet, who have no track record of providing safe harbour to spammers, can have their IP address(es) whitelisted, so that we are essentially excluded from the Level 3 blacklisting of all Digital Ocean IP addresses. Previously we chose not to do this because of the added expense, and we preferred to spend money on other ways (described in the first paragraph of this post) of mitigating this problem. However, we have broken down and paid a fee to UCEPROTECT to have our IP address whitelisted.

Therefore, if we are correct in deducing the cause of the current problem, we expect that email to domains hosted by Microsoft will be delivered without hindrance starting by about 04:18 UTC today, 21 June 2024.

Update, 2024-06-22: We thought that our having paid for an exception to the UCEPROTECT blacklist had solved the problem. And it does seem to have solved the problem, for the most part. However, very oddly, messages to only some Microsoft-hosted domains are still being blocked with the exact same bounce message that directs senders to their article, “External senders – Use the delist portal to remove yourself from the blocked senders list and address 5.7.511 Access denied errors” at http://go.microsoft.com/fwlink/?LinkID=526655, which redirects to https://learn.microsoft.com/en-gb/defender-office-365/external-senders-use-the-delist-portal-to-unblock-yourself?redirectedfrom=MSDN. (The 5.7.511 error in the title does not appear to apply to the messages bounced from our server, as those errors are 5.7.1.) However, every time we try to have our mail server’s IP address delisted, the response we receive is, “The IP address in question is not currently blocked in our system.” So why are messages being blocked?!

This seems to be a ridiculous game of cat-and-mouse that Microsoft are playing instead of being open with people about what they are doing, and companies like NinerNet cannot do anything to counter that. It makes absolutely no sense, and doesn’t serve Microsoft, their customers, or NinerNet or our customers.

So in these circumstances, if you’re still having messages to Microsoft-hosted domains bounced — you will know if you see references to Outlook(.com) and Microsoft(.com) in the bounce message — please forward the bounce message(s) to NinerNet support and we will add the problem domains to the mail server configuration that redirects messages sent to those domains via our secondary SMTP server. This is the same procedure that we followed previously, but we were hoping to avoid that procedure by buying our way out of the UCEPROTECT blacklist. However, at least now the number of Microsoft-hosted domains that we have to add to our mail server configuration should be far less than previously.

Again, we apologise to you, our clients, for this non-consensual position in which Microsoft has put us and many small hosting companies around the world.

Update, 2024-06-28: Over the last week we have added a grand total of 21 domains to our mail server’s configuration to redirect outgoing messages to them via our secondary mail server. In that time we have learned that there is no consistency to the problem. Sometimes mails that are blocked are delivered five minutes later if the sender retries, without our adding that domain to our mail server’s configuration. And delivery succeeds to some Microsoft-hosted domains consistently without any intervention by us. There’s nothing more frustrating than an inconsistent problem that is not possible to troubleshoot.

So at this point it seems that we are back to the point we were at before this incident started. Here is a summary of what has transpired:

• All emails to Microsoft-hosted domains started bouncing.
• We paid to be removed from a blacklist that we thought might be the cause.
• This seemed to help to some extent, but over the course of the next few days we added 21 destination domains to our mail server’s configuration to direct messages to those domains via our secondary mail server.
• We are still exploring alternative ways of automatically determining the MX record of destination domains and automatically redirecting mail to Microsoft-hosted domains via our secondary mail server.
• We are also still looking for ways to contact Microsoft to determine the cause of this issue, but we hold out little hope of doing that.
• This server will be replaced in the near term. To that end we will be looking for a data centre where we won’t run into the issue of all of their IP addresses being blacklisted as is the case where our primary mail server is currently located on a Digital Ocean IP address.

As always, if you have mail you send bounced by Microsoft, please forward the bounce message to support and we will add the destination domain to our mail server’s configuration. We appreciate your patience and continued patronage.