Server NC036, our primary mail server, was apparently down for 34 minutes yesterday, 8 January. Our server itself was not down, but the data centre experienced what seems to have been a partial connectivity issue. In fact, you may not even have noticed it, as we didn’t receive any complaints about the downtime.
According to the data centre’s report, the issue was resolved and it is not expected to recur. We apologise to anyone who was affected.
Today we received a number of complaints of connectivity problems to our mail server. However, they were all from Zambia, from the Luangwa Valley to the Copperbelt to Lusaka; nobody else had any issues.
Below is a screenshot of NinerNet’s live status page that was taken moments ago that shows that the mail server (NC036) has been up for 100% of the last 90 days.
NinerNet live status screenshot, 2023-05-18.
In the absence of any other information we can only assume that there was some sort of widespread, country-wide connectivity problem in Zambia late today, for which NinerNet is clearly not responsible. The NinerNet mail servers continued to process inbound email during this time, so even if you couldn’t connect to the mail server, incoming email to you was still collecting on the mail server for your eventual collection.
If you have any questions or concerns, please feel free to contact NinerNet support. Thank-you.
Update, 2023-05-19: One of our clients contacted a Zambian ISP and the problem was blamed on Zambia’s international gateway. Sorry for that!
The *.niner.net wildcard SSL/TLS certificate has been renewed and updated across our entire infrastructure. This should be seamless for all our clients. However, should you run into a situation in the next 24 hours where you’re told that the certificate has expired, please log out and reload whatever it is you’re trying to do. You may need to reboot, but this would be very unusual. Simply forcing a reload should clear things up. Via FTPS you may need to re-trust our certificate depending on your FTP client.
This doesn’t affect certificates on your own domain on your website, just NinerNet services such as control panels and connecting to the mail and FTP servers.
If you have any questions or concerns, please do contact NinerNet support with any error messages you may be seeing. Thank-you.
Please note that we are seeing messages from NC036 destined for private domains hosted by Microsoft delayed at this time. The cause is not clear, but it’s clearly on Microsoft’s end.
This is not in any way related to the internal server issue that happened yesterday.
The issue now appears to be completely resolved. If you’re having any email issues, please do contact NinerNet support and we will look into them. Thank-you.
We have been informed by the data centre where server NC036 is located that, due to “an effect from ongoing data centre maintenance”, there are connectivity issues. We have been noticing intermittent issues with both direct email connections and the webmail, notably slower transactions (e.g., deleting messages, scanning multiple folders, etc.). However, email is still flowing as usual, perhaps with minor delays.
We have opened a ticket with the data centre to determine why the issues are ongoing. We will update you as we receive more information.
The *.niner.net wildcard SSL/TLS certificate has been renewed and updated across our entire infrastructure. This should be seamless for all our clients. However, should you run into a situation in the next 24 hours where you’re told that the certificate has expired, please log out and reload whatever it is you’re trying to do. You may need to reboot, but this would be very unusual. Simply forcing a reload should clear things up. Via FTPS you may need to re-trust our certificate depending on your FTP client.
This doesn’t affect certificates on your own domain on your website, just NinerNet services such as control panels and connecting to the mail and FTP servers.
If you have any questions or concerns, please do contact NinerNet support with any error messages you may be seeing. Thank-you.
As stated previously, the maintenance on server NC036 is complete, and has been successful.
That said, it had to be carried out earlier than scheduled because disk space on the server was the issue that was causing a problem. To be clear, the disk space used by email accounts was not and is not the issue (that’s on a separate disk), it was the disk space used by the operating system and the logs in particular. When this problem was noticed earlier this week we projected that addressing the issue during our regular weekend maintenance window on Saturday evening UTC would be sufficient. Well, we were wrong. As soon as it became clear that our temporary mitigation efforts were failing to keep up with the rate at which disk space was being consumed (mostly by the aforementioned logs), we immediately implemented the maintenance that we had planned and already practised on a test server. This practice paid off, as instead of allowing an hour for the maintenance to complete, we were done in four minutes, including a server reboot.
That said, the lack of disk space on the server did cause disruption in the rate at which mail was processed, that led to some of you getting errors when sending email, and some incoming email being delayed. For this we apologise. The backed-up mail queue was fully processed by 09:18 UTC.
Had our prediction held up and the maintenance been carried out this evening as scheduled, we wouldn’t even have needed to post this detail. However, in order to clear up any concern, we have.
Early in 2022 server NC036 is due to be considered for replacement. The experience of this issue and other issues we have run into in recent months will inform our decisions about where to locate the replacement server (hint: it won’t be in the current data centre) and various configuration improvements we can make. We’ll also, of course, be using a more current version of the control panel.
If you have any questions at all about today’s maintenance, please let us know by contacting NinerNet support. Thank-you for your patience, and continued patronage.
The maintenance on server NC036 (the primary mail server) is complete and the server is back online. It was unavailable between 08:28 and 08:32 UTC. According to some reports there have been issues with delayed emails and “lookup failure” errors, but these have now cleared.
We’ll be posting some further information shortly with more details.
| Loc. | System | Status | Ping |
|---|---|---|---|
 | NC023 | Internal | Up? |
 | NC028 | Internal | Up? |
 | NC031 | Internal | Up? |
| NC033 | Operational | Up? |
 | NC034 | Internal | Up? |
 | NC035 | Operational | Up? |
 | NC036 | Operational | Up? |
| NC040 | Internal | Up? |
| NC041 | Operational | Up? |
| NC042 | Operational | Up? |
