NinerNet Communications™
System Status

Server and System Status

NC041: Web server rebooted

19 June 2023 18:53:54 +0000

Server NC041 has been rebooted. This appears to have dealt with the database server issue. We will continue to look for the root cause to ensure this doesn’t happen again.

NC041: Web server having some database issues

19 June 2023 17:35:03 +0000

The database server on server NC041 appears to be having some issues. We are looking into it and will post an update about what we find shortly.

NC036: Connectivity problems in Zambia

18 May 2023 19:54:44 +0000

Today we received a number of complaints of connectivity problems to our mail server. However, they were all from Zambia, from the Luangwa Valley to the Copperbelt to Lusaka; nobody else had any issues.

Below is a screenshot of NinerNet’s live status page that was taken moments ago that shows that the mail server (NC036) has been up for 100% of the last 90 days.

NinerNet live status screenshot, 2023-05-18.

NinerNet live status screenshot, 2023-05-18.

In the absence of any other information we can only assume that there was some sort of widespread, country-wide connectivity problem in Zambia late today, for which NinerNet is clearly not responsible. The NinerNet mail servers continued to process inbound email during this time, so even if you couldn’t connect to the mail server, incoming email to you was still collecting on the mail server for your eventual collection.

If you have any questions or concerns, please feel free to contact NinerNet support. Thank-you.


Update, 2023-05-19: One of our clients contacted a Zambian ISP and the problem was blamed on Zambia’s international gateway. Sorry for that!

NC036: Issue resolved

5 July 2022 23:54:13 +0000

The issue now appears to be completely resolved. If you’re having any email issues, please do contact NinerNet support and we will look into them. Thank-you.

NC036: Server is up, but some issues

5 July 2022 23:09:28 +0000

We have been informed by the data centre where server NC036 is located that, due to “an effect from ongoing data centre maintenance”, there are connectivity issues. We have been noticing intermittent issues with both direct email connections and the webmail, notably slower transactions (e.g., deleting messages, scanning multiple folders, etc.). However, email is still flowing as usual, perhaps with minor delays.

We have opened a ticket with the data centre to determine why the issues are ongoing. We will update you as we receive more information.

NC036: Disk space issue resolved

8 May 2022 06:19:09 +0000

Briefly between Saturday and Sunday the disk storing emails on server NC036 became full. This was resolved immediately upon discovery, but some incoming emails were delayed. Some may also have bounced. However, this issue is now fully resolved.

If you have any questions, please contact NinerNet support.

NC041: Maintenance post-mortem

8 February 2022 07:39:37 +0000

Our apologies for not following up this issue within the promised 48 hours.

This issue was traced to a problem with a single website hosted on this server, and the modifications made to this website’s file system. What was done within one account’s file system should not have affected the whole server though, and as a result we have filed a bug report with the control panel vendor.

We do again apologise for this issue, and will endeavour to ensure that it does not become an issue again.

If you have any questions at all about this maintenance, please let us know by contacting NinerNet support. Thank-you for your patience, and continued patronage.

NC041: Primary web server back online

24 January 2022 05:51:20 +0000

The primary web server (NC041) is again online. It was down for 39 minutes between 05:05 and 05:44 UTC. We’re investigating why this incident happened, and will post further details here within the next 48 hours. Our apologies for the inconvenience.

NC041: Primary web server is currently down

24 January 2022 05:11:56 +0000

The primary web server (NC041) is currently down following maintenance. We are investigating. This does not affect email.

NC036: Maintenance port-mortem

16 October 2021 10:41:00 +0000

As stated previously, the maintenance on server NC036 is complete, and has been successful.

That said, it had to be carried out earlier than scheduled because disk space on the server was the issue that was causing a problem. To be clear, the disk space used by email accounts was not and is not the issue (that’s on a separate disk), it was the disk space used by the operating system and the logs in particular. When this problem was noticed earlier this week we projected that addressing the issue during our regular weekend maintenance window on Saturday evening UTC would be sufficient. Well, we were wrong. As soon as it became clear that our temporary mitigation efforts were failing to keep up with the rate at which disk space was being consumed (mostly by the aforementioned logs), we immediately implemented the maintenance that we had planned and already practised on a test server. This practice paid off, as instead of allowing an hour for the maintenance to complete, we were done in four minutes, including a server reboot.

That said, the lack of disk space on the server did cause disruption in the rate at which mail was processed, that led to some of you getting errors when sending email, and some incoming email being delayed. For this we apologise. The backed-up mail queue was fully processed by 09:18 UTC.

Had our prediction held up and the maintenance been carried out this evening as scheduled, we wouldn’t even have needed to post this detail. However, in order to clear up any concern, we have.

Early in 2022 server NC036 is due to be considered for replacement. The experience of this issue and other issues we have run into in recent months will inform our decisions about where to locate the replacement server (hint: it won’t be in the current data centre) and various configuration improvements we can make. We’ll also, of course, be using a more current version of the control panel.

If you have any questions at all about today’s maintenance, please let us know by contacting NinerNet support. Thank-you for your patience, and continued patronage.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings configuration connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: