Server and System Status

We have had ongoing problems with delivery to the kcm.co.zm (Konkola Copper Mines) domain since time immemorial. Konkola appears to host their own mail server over a very slow Zamtel connection. This means their connection appears to be extremely limited in what it can handle. While sporadic small emails without attachments generally go through within a day or two (yes, that’s days, not seconds, minutes or even hours), larger emails with attachments either stall or take so long to deliver that they affect delivery of other emails.

That is the case today. Three large emails from two clients to over one hundred kcm.co.zm email addresses effectively plugged up the mail server while it waited for the kcm.co.zm mail server to accept delivery, preventing delivery of hundreds of other emails. We had no choice but to delete these emails. As soon as we did, the backlog of several hundred emails that had built up over the course of about an hour were successfully delivered within minutes.

If you email people on the kcm.co.zm domain, please consider reducing the sizes of attachments and reducing the number of kcm.co.zm email addresses you include in your distribution lists. You might also want to point them to this post so that they might become aware of the problem and therefore do something to fix it.

It appears that email to microlink.zm email addresses is experiencing delays at the moment. We have quite a few emails queued on the server at the moment going back to 3 July. It seems that the Microlink mail server is currently down.

Early this week we started receiving reports from Blackberry users with different wireless carriers on different continents, that late last week they started having problems using their Blackberries to access their email on their domains hosted with NinerNet. However, they could access their email accounts through regular means — i.e., POP (with an email program) or webmail.

An initial look at the problem suggested that the two common denominators were NinerNet and Research in Motion (RIM), the latter being the “Blackberry company” through which all email on Blackberry devices is handled. It was (and is) unlikely that individual carriers were the cause of the problem.

After considerable research and consultation over the course of a couple of days, we did not have (and still do not have) any definitive reason for the problem. One reason for the lack of a definitive cause is the complete lack of information from RIM, who seem not to have any useful information about their email processing methods on their website for hosting companies like NinerNet, and who make it very difficult for people to contact them. (One good thing that can be said about a behemoth like AOL is that they maintain a website and a channel for hosting companies like NinerNet to contact them in the event of email delivery issues, as happened last week. That issue was successfully resolved thanks to dialogue with AOL.)

In the spirit of trial and error we tried two different configuration changes to our mail server based on the connection activity we were seeing from RIM servers. In hindsight we should have implemented these separately to see which one was actually responsible for fixing the issue, but the focus was, of course, on fixing the issue, not messing around. That said, a third issue was that, after making the configuration changes, we restarted the mail server. This had the effect of closing down about two hundred connections to our mail server from RIM servers. About fifty connections were quickly re-established, and we expected this number to grow again to about two hundred … but it didn’t. After more than twenty-four hours the number of connections remains steady at about fifty. This is interesting in itself, as simply restarting the server and not making the configuration changes may actually have fixed the problem anyway, making us wonder why RIM had so many connections open to our server in the first place.

So for now the issue appears to be resolved. We’re continuing to monitor connections to our mail server by RIM servers, and we ask that, if you have any further problems with accessing your NinerNet-hosted email using your Blackberry, you contact us with the details as soon as possible. Thanks very much.

We’ve made a couple of configuration changes to the mail server and restarted it. By all accounts this appears to have fixed the problem, but we’re not convinced that this is the end of the issue. We’ll post another update here within the next 48 hours, and we’ll also be sending an email to all clients about this when that update is posted.

Thanks for your patience. As mentioned previously, we’re still looking for more information from Blackberry users about this, so if there is any information you can provide to us that you think might be useful, we’re all ears.

We’ve had reports from Blackberry users with different networks on different continents that they are unable to access email on their domains hosted with NinerNet via their Blackberry devices. We’re still looking into this issue to determine if the problem is on our end or with the Blackberry / Research in Motion (RIM, the Blackberry company) systems. Based on our understanding of the relationship between Blackberry/RIM and individual network carriers, this is probably not an issue with the carriers themselves. However, we don’t know whether or not individual subscribers are able to contact Blackberry/RIM directly for support.

In the meantime, we’ve been informed by one client that they were able to regain access to their email account through their Blackberry by changing their configuration to use IMAP on port 143 and a non-SSL connection. If this issue is affecting you, please try this.

We are interested in feedback and diagnostic information on this issue from clients. If you are able to send us copies of error messages or information directly from Blackberry/RIM (the company) support or your carrier, we would appreciate it. Thanks.

We’ll post updates here as we work on this issue.

We’ve had the following response from AOL to our query about incoming email from AOL addresses being bounced back to the senders:

We had few issues with our Outbound IP’s timing out with few mail servers last week. Are you still seeing reports of our users not able to connect to your servers? Please send me the headers and I can check the logs at our end. With regards to Spam Flag Yes- we are seeing lots of AOL accounts being compromised forcing us to send mails through our aberrant IP’s. These mails might be rejected at our end too if we suspect compromise but do issue a bounce message.

This would seem to indicate that AOL has solved the issue of legitimate email sent to NinerNet clients being returned as undeliverable. We’ve had no reports of incoming email from AOL senders being bounced for over a week now, so we’ll consider this matter closed.

While we have not heard from AOL yet, in recent experiments all email from AOL got through. However, these experiments were on a very small scale, so may not accurately reflect reality.

Domains hosted by Realtime appear to be back up again after being down for about five days. That email backlog has been cleared.

Email to the iwayafrica.com domain appears to be having problems. There isn’t much of this backlogged on the mail servers, but there is some. Same with a couple of domains hosted by Coppernet.

If you’re trying to communicate with people with unreliable email, why not refer them to NinerNet? We pay referral fees!

We have had some reports of incoming email from AOL users being bounced back to the senders. After some extensive research over the last few days we believe we have managed to address this problem, adjusting the configuration of one of our anti-spam measures (greylisting) to allow for the fact that AOL’s handling of outbound email doesn’t play well with greylisting.

If you are a NinerNet client and have any reports from correspondents with AOL email addresses indicating that email from them to you has been returned as undeliverable, please contact support.

Update 2010-06-15 07:28 UTC: Apparently this issue has not yet been resolved. We contacted AOL yesterday to find out why some of their mail servers are bouncing email prematurely and without reason, but their auto-response says it could take seven days or more for them to respond, it at all. We will post updates as they become available.

Clients sending email to domains hosted by Realtime are advised that messages to these domains have been backing up on our mail servers over the weekend, and are still not being delivered. A cursory analysis would seem to indicate that there is a routing problem at or near the Realtime mail servers. If you have concerns about this, please contact the intended recipients via a medium other than email and have them contact Realtime for an explanation.

Update 2010-06-15 07:26 UTC: This issue seems to be ongoing. Domains hosted by Realtime appear to be down still, and email to them continues to be queued on our mail servers.