Server and System Status

NC036: Update 4

13 June 2025 01:23:56 +0000

The issue on our primary mail server has finally been resolved, and all messages in the queue have been delivered. As expected, once we had access it only took a few seconds.

We will post a post-mortem in the next couple of days … hopefully. I can’t exaggerate the extent to which numerous unrelated events have piled on top of one another — even in the last few minutes! — to prevent an earlier resolution of this problem, and at this point I can’t predict whether or not more issues will prevent the posting of the post-mortem. However, I’m finally taking a breath, as this issue (amongst other things) is finally resolved.

I do once again extend my heartfelt apology for this incident, and I will do everything in my power to review the cascading failures — all not even related to the mail server itself! — that led to this not being resolved much, much sooner.

Tags: bounce messages, email, email delivery, error messages, imap, mail, mail server, pop
Categories: Email, Incidents, NC036
No Comments »

NC036: Update 3

12 June 2025 13:45:11 +0000

Words cannot express my frustration at this point. 🙁

It will be another few hours again before this situation can be resolved. It just cannot go beyond tonight, UTC. By that time my computer will be completely reset with a fully updated operating system installed.

Sorry.

Tags: bounce messages, email, email delivery, error messages, imap, mail, mail server, pop
Categories: Email, Incidents, NC036
No Comments »

NC036: Update 2

12 June 2025 09:53:21 +0000

Let me explain the situation we’re in. It’s an illustration of the fact that sometimes too much is, in fact, too much.

My primary workstation stopped working late Wednesday afternoon (UTC). It stopped working because I could not log in after performing a maintenance/security operation that I routinely run, but I ran it in a certain way that was sightly different to how I usually run it with no problems.

At about the same time I received a report from a client about a problem with the mail server. I received it by email (of course) which I read on my phone. I hadn’t seen anything similar before, so I asked him for screenshots. In the meantime I had an idea of what the cause of the problem could be based on monitoring I had done the day before, but without access to my workstation I could not log in and check and fix the problem … which would (and will) take all of about 60 seconds if I am correct. Reports and my experience since have almost confirmed my suspicions.

So, given the fact that it is the middle of the night where I am I cannot do anything until business hours, which will be about 06:00 local, 13:00 UTC.

My local workstation is, of course, fully backed up, so it’s not a problem of a loss of data. The “problem” is with the additional security on logging into the server which we have purposely put into place in order to protect our infrastructure and your email. Because of that I cannot log into the mail server from the machine I am currently using, and will only have access to the resources I require in the morning, local time.

I cannot apologise enough for this situation that we have caused. We will calculate a credit that will be applied to all invoices of clients who host their email with us.

In the meantime, we apologise but this issue will continue until about 13:00 UTC. At that time I should have access to the server to fully and permanently address the problem. I will post an update here, on the status blog, when this issue is resolved. My humble and sincere apologies once again.

Tags: bounce messages, email, email delivery, error messages, imap, mail, mail server, pop
Categories: Email, Incidents, NC036
No Comments »

NC036: Update 1

12 June 2025 07:54:49 +0000

We continue to work on resolving this issue. The problem we’re having has nothing to do with the server itself, but our access to it.

One thing we can tell you for now is that one of the issues you may encounter is that incoming messages to your accounts may be duplicated, which is something I’m certainly experiencing. It’s frustrating and annoying for me, so I assume it is for you as well. Again, we apologise.

Tags: bounce messages, email, error messages, imap, mail, mail server, pop
Categories: Email, Incidents, NC036
No Comments »

NC036: Mail server issues

12 June 2025 05:39:50 +0000

The mail server is having an issue at the moment. You will see different symptoms, but they are all “unusual”.

This has apparently been happening for about 24 hours now. Under normal circumstance we would have worked on the problem close to 24 hours ago now, but unfortunately our office network is currently not fully usable at the moment, so we can’t.

What we can tell you at this point is that outgoing messages are going out, even if you see errors that seem to indicate otherwise.

We are currently working on all three of the “perfect storms” to address this issue. I hope I’ll have an update for you within the next couple of hours.

Tags: bounce messages, email, error messages, imap, mail, mail server
Categories: Email, Incidents, NC036
No Comments »

Email failures from Apple email addresses

25 March 2025 04:33:20 +0000

Due to support requests recently, it has become obvious that Apple have mis-configured their mail servers, either intentionally or because they’re stupid (both are one and the same in this case, but I’m leaning towards the latter of course), so that they don’t play well with an anti-spam technology that NinerNet and millions around the world use called greylisting. (If you want to more thoroughly understand greylisting, we recommend you read the page at that link.)

It seems that some (many?) users of Apple-administered mail servers on domains like mac.com, me.com, icloud.com (maybe even apple.com itself) and possibly other domains that we’re not aware of receive bounce messages for email messages they send to NinerNet-hosted domains that contain this text:

<email-address@ninernet-hosted-domain.com>: host mx.niner.net[178.62.195.26] said:
451 4.7.1 <*****@*****************>: Recipient address rejected:
Intentional policy rejection, please try again later (in reply to RCPT TO command)

At this time we don’t know if this also applies to domains running on any of the servers that use an Apple operating system, but we haven’t seen evidence of this because even those servers generally run applications (e.g., Postfix [which we use], Qmail, etc.) that are not Apple products and comply with email standards. We also don’t know the frequency with which email sent to NinerNet-hosted domains fail.

Here’s the explanation we have sent to recent clients:

The four (hundred) codes (451 and 4.7.1) tell the sending server that the error is temporary, and that the sending server should (as it also says in plain English), “try again later”. A four-hundred error code is not a permanent error; those are five-hundred codes. Email messages should not be bounced by four-hundred error codes, so this is why the sending (Apple) server is behaving incorrectly according to email standards. Email doesn’t work if organisations (i.e., Apple) ignore standards.

Greylisting isn’t something we invented while I was bored last weekend; it’s an extremely widely used and also extremely successful anti-spam technique that we have been using for about as long as NinerNet has been in business, which is thirty years.

So the sender needs to bring this to the attention of the support department at Apple. NinerNet is not the cause of this problem.

As we’ve also explained, “standards” are not vague suggestions; they’re the “laws” on which servers and clients agree to operate so that they can work together.

In order to compensate for the stupidity of the brain trust at Apple we have whitelisted the following domains server-wide, so that greylisting is not applied to email from these domains:

apple.com
icloud.com
me.com
mac.com

This, of course, means that spam from these domains will more likely reach our clients, and for this we humbly apologise. We hope that Apple are doing a better job of keeping spammers off of their systems than they are at actually running the mail servers themselves.

If your correspondents on other Apple-hosted domains report similar problems sending email to you, please let us know and we’ll add those domains too to our mail servers’ whitelists.

If you have any questions about this, please feel free to contact NinerNet support. Thanks for your attention.

Tags: apple, bounce messages, email, email delivery, error messages, mail, postfix, qmail
Categories: Email, NC036
Comments Off

NC036: Mail server is back to normal

20 December 2024 13:51:23 +0000

Server NC036 (the primary mail server) is back to normal. All mail in the mail queue has been cleared and delivered to all local accounts and remote/foreign domains.

We do not currently have an explanation for this occurrence, but we are looking into it. Our apologies for the interruption.

If you have any specific questions or issues with this now-resolved event, please contact NinerNet support. Thank-you, and we apologise for this incident.

Tags: email, email delivery, mail, mail server, performance
Categories: Incidents, NC036
Comments Off

NC036: Mail server update

20 December 2024 13:28:50 +0000

Server NC036 is recovering. Delayed email is being delivered and the mail queue is decreasing.

Tags: email, email delivery, mail, mail server, performance
Categories: Incidents, NC036
Comments Off

NC036: Server is temporarily struggling

20 December 2024 13:07:52 +0000

Server NC036 (the primary mail server) is temporarily struggling under an unusually large mail load. We are working to determine the cause of this and bring everything back to normal.

In the meantime you may experience delays in sending or receiving messages.

We will update as and when there is new information.

Tags: email, email delivery, mail, mail server, performance
Categories: Incidents, NC036
Comments Off

NC036: Evaluating the effectiveness of being listed in the UCEPROTECT whitelist

19 July 2024 03:57:16 +0000

Just about a month ago we became aware of the problem with delivering email to Microsoft-hosted domains. At that time, one of the actions we took was to pay to have our mail server’s IP address listed in the UCEPROTECT whitelist that effectively removed our IP address from a huge list of blacklisted IP addresses that are only listed because of the lackadaisical approach of our data centre (Digital Ocean) to removing spammers from their network of data centres.

We weren’t certain at the time that this action would achieve anything and, to be frank, we have no concrete evidence to believe that our doing so has had the desired effect. We think there is strong evidence that it has, but there is no way we can determine that definitively. The strong evidence is that we have had very few problems with mail to Microsoft-hosted domains since a couple of days after the problem started on 20 June.

Our subscription to this whitelist expires on Monday the 22nd. To test the effectiveness of this subscription we are going to allow it to lapse. If we see a sudden uptick in email messages to Microsoft-hosted domains bouncing then we’ll take that as evidence that the subscription is working, and we’ll immediately renew it. In that case, messages you send to such domains will bounce temporarily, so please forward those bounce messages to NinerNet support and we will immediately divert future messages to those domains via our secondary outbound mail servers. This is a temporary and planned test; if these bounces happen it is not evidence that our systems are failing for any reason. If they don’t bounce then we will have learned that our paying for the listing is a waste of money.

The purpose of this message is to let you know of this test in advance. We expect that if messages start bouncing again and we renew the subscription, everything will be back to normal within about 24 hours, and immediately for domains we add to our mail server configuration to have messages go out via our secondary mail server. That said, we have also found that retrying if you get a bounce in this situation sometimes succeeds! It’s bizarre.

Thank-you for your continued patience with issues like this thrown at us by massive providers that don’t care about you, people who choose to host with their competitors.

Tags: bounce messages, email, email delivery, mail, mail relay, mail server, microsoft, spam
Categories: Email, NC036
Comments Off

System	Status	Ping
NC023	Internal	Up?
NC028	Internal	Up?
NC031	Internal	Up?
NC033	Operational	Up?
NC034	Internal	Up?
NC035	Operational	Up?
NC036	Operational	Up?
NC040	Internal	Up?
NC041	Operational	Up?
NC042	Operational	Up?