After about an hour and a half of managing resources on the mail server (NC036), we have cleared the mail queue and processed all of the email that was delayed.
As far as we can tell, these incidents are caused by the mail server being overloaded. Recent statistics showed that at certain times about 80% of the mail received by our mail server was spam. I don’t want to appear like I am cherry-picking statistics to make it look worse than it really is, but the statistics amaze even me, who has been running mail servers for 29 years. At other times the spam load decreases, of course, but at other times, of the 100 messages we receive in any given unit of time, 80 of the messages are spam. So at those times the server cannot keep up with the flow of email into the server, because the server is busy scanning all the messages to determine if they are spam, and while it’s doing that ten more new messages arrive. Then it finishes processing that spam message and starts processing the first of the ten new messages waiting, and so the process goes.
Yes, we could increase the memory on the server, but this isn’t cheap, isn’t instant and can be further disruptive, and we know for some of our bigger clients, your invoices have increased this year. The money available to us is not unlimited, just as your wallets are not unlimited.
The mail server woes are not related to the nameserver work we’re doing. They are three different machines in three different data centres on three different continents. Gone are the days back in the nineties when we ran everything on one physical server, when we had problems communicating with our clients if the one server had a problem unrelated to email; when that happened, the unrelated problem meant we couldn’t also update this server blog (which didn’t exists back then!), so everybody was in the dark until we fixed the problem. Now, we at least have the real-time monitor and this status blog, but even those have limitations; you’ll notice that the statistics for server NC036 (the mail server) show it as having been up for 99.997% in the last 24 hours, which supposedly means the mail server was down for only 2.5 seconds! If that’s all it was down for, people wouldn’t have time to write emails explaining their problems! So the real-time monitor isn’t 100% accurate (even though it is close), and even the monitoring didn’t alert us when it should have. This resulted in the delay in clearing up the mail queue.
The point of this long blog post is to describe the challenges that we face some days, and to let you know we’re not sitting back doing nothing, while you wonder what’s happening to your email. Thank-you for your attention.
