This seems to be a disk space issue. We have freed up enough space that backlogged email has been released. There should be no problems sending and receiving email now, but we are still working to implement a permanent fix. Apologies for the interruption.
We will have further updates as we implement the permanent fix.
We have been informed that some clients (probably all) are encountering issues with the mail server. We are looking into the cause now.
The *.niner.net wildcard SSL/TLS certificate has been renewed and updated across our entire infrastructure. This should be seamless for all our clients. However, should you run into a situation in the next 24 hours where you’re told that the certificate has expired, please log out and reload whatever it is you’re trying to do. You may need to reboot, but this would be very unusual. Simply forcing a reload should clear things up. Via FTPS you may need to re-trust our certificate depending on your FTP client.
This doesn’t affect certificates on your own domain on your website, just NinerNet services such as control panels and connecting to the mail and FTP servers. There is also an ongoing issue on dns.niner.net, but bypassing the SSL warning is perfectly safe.
If you have any questions or concerns, please do contact NinerNet support with any error messages you may be seeing. Thank-you.
After about an hour and a half of managing resources on the mail server (NC036), we have cleared the mail queue and processed all of the email that was delayed.
As far as we can tell, these incidents are caused by the mail server being overloaded. Recent statistics showed that at certain times about 80% of the mail received by our mail server was spam. I don’t want to appear like I am cherry-picking statistics to make it look worse than it really is, but the statistics amaze even me, who has been running mail servers for 29 years. At other times the spam load decreases, of course, but at other times, of the 100 messages we receive in any given unit of time, 80 of the messages are spam. So at those times the server cannot keep up with the flow of email into the server, because the server is busy scanning all the messages to determine if they are spam, and while it’s doing that ten more new messages arrive. Then it finishes processing that spam message and starts processing the first of the ten new messages waiting, and so the process goes.
Yes, we could increase the memory on the server, but this isn’t cheap, isn’t instant and can be further disruptive, and we know for some of our bigger clients, your invoices have increased this year. The money available to us is not unlimited, just as your wallets are not unlimited.
The mail server woes are not related to the nameserver work we’re doing. They are three different machines in three different data centres on three different continents. Gone are the days back in the nineties when we ran everything on one physical server, when we had problems communicating with our clients if the one server had a problem unrelated to email; when that happened, the unrelated problem meant we couldn’t also update this server blog (which didn’t exists back then!), so everybody was in the dark until we fixed the problem. Now, we at least have the real-time monitor and this status blog, but even those have limitations; you’ll notice that the statistics for server NC036 (the mail server) show it as having been up for 99.997% in the last 24 hours, which supposedly means the mail server was down for only 2.5 seconds! If that’s all it was down for, people wouldn’t have time to write emails explaining their problems! So the real-time monitor isn’t 100% accurate (even though it is close), and even the monitoring didn’t alert us when it should have. This resulted in the delay in clearing up the mail queue.
The point of this long blog post is to describe the challenges that we face some days, and to let you know we’re not sitting back doing nothing, while you wonder what’s happening to your email. Thank-you for your attention.
This is one of those operations that takes only a few minutes if you’ve done it a thousand times before. 🙂 But we’ve sort of addressed the imminent possible problem of our secondary mail relay service changing one of their requirements. To be frank, although this maintenance needed (and still needs) to be done, we’ve instead done a lot of manual work that has addressed the immediate problem before the weekend.
But we do need to spin up two new nameservers; it’s been at the top of our list for some time, and will greatly improve how you will be able to manage the nameservice for your domains. Since we last did this a few years ago we’ve found that we have greater choice of software and methods than we had before, and so we’re going to take some time over the weekend to evaluate a number of different options. Finishing the maintenance over the weekend will also ensure that we avoid stepping on anybody’s toes in the middle of their changing their DNS.
We will finish over the weekend and continue to post further updates here. Thank-you for your patience.
We need to replace our two nameservers (NC033 and NC035) in short order. Under normal circumstances this would be a seamless operation, of which we could just notify you after the fact. However, we always notify you of planned maintenance in advance anyway, but in this case we are under a tight deadline to do this due to changed requirements at our secondary mail relay service.
Our plan is to carry out this operation over the next few hours. It’s unclear whether SMTP2GO’s deadline is in the UTC time zone or in New Zealand where the company is located. If the deadline is in the UTC time zone then we’ll be finished long before the deadline. There will be no downtime, but if there is any interruption if will be for some outbound email. We will work to avoid this and implement workarounds if necessary.
There will be a new look with a new control panel next time you log into dns.niner.net. In one of our next few posts we will post screenshots of the new control panel so that you know what to expect.
If you have any questions, please contact NinerNet support. We will post updates as necessary as work progresses. Thank-you.
| Loc. | System | Status | Ping |
|---|---|---|---|
 | NC023 | Internal | Up? |
 | NC028 | Internal | Up? |
 | NC031 | Internal | Up? |
| NC033 | Operational | Up? |
 | NC034 | Internal | Up? |
 | NC035 | Operational | Up? |
 | NC036 | Operational | Up? |
| NC040 | Internal | Up? |
| NC041 | Operational | Up? |
| NC042 | Operational | Up? |
