Server and System Status

We have had ongoing problems with delivery to the kcm.co.zm (Konkola Copper Mines) domain since time immemorial. Konkola appears to host their own mail server over a very slow Zamtel connection. This means their connection appears to be extremely limited in what it can handle. While sporadic small emails without attachments generally go through within a day or two (yes, that’s days, not seconds, minutes or even hours), larger emails with attachments either stall or take so long to deliver that they affect delivery of other emails.

That is the case today. Three large emails from two clients to over one hundred kcm.co.zm email addresses effectively plugged up the mail server while it waited for the kcm.co.zm mail server to accept delivery, preventing delivery of hundreds of other emails. We had no choice but to delete these emails. As soon as we did, the backlog of several hundred emails that had built up over the course of about an hour were successfully delivered within minutes.

If you email people on the kcm.co.zm domain, please consider reducing the sizes of attachments and reducing the number of kcm.co.zm email addresses you include in your distribution lists. You might also want to point them to this post so that they might become aware of the problem and therefore do something to fix it.

It appears that email to microlink.zm email addresses is experiencing delays at the moment. We have quite a few emails queued on the server at the moment going back to 3 July. It seems that the Microlink mail server is currently down.

Early this week we started receiving reports from Blackberry users with different wireless carriers on different continents, that late last week they started having problems using their Blackberries to access their email on their domains hosted with NinerNet. However, they could access their email accounts through regular means — i.e., POP (with an email program) or webmail.

An initial look at the problem suggested that the two common denominators were NinerNet and Research in Motion (RIM), the latter being the “Blackberry company” through which all email on Blackberry devices is handled. It was (and is) unlikely that individual carriers were the cause of the problem.

After considerable research and consultation over the course of a couple of days, we did not have (and still do not have) any definitive reason for the problem. One reason for the lack of a definitive cause is the complete lack of information from RIM, who seem not to have any useful information about their email processing methods on their website for hosting companies like NinerNet, and who make it very difficult for people to contact them. (One good thing that can be said about a behemoth like AOL is that they maintain a website and a channel for hosting companies like NinerNet to contact them in the event of email delivery issues, as happened last week. That issue was successfully resolved thanks to dialogue with AOL.)

In the spirit of trial and error we tried two different configuration changes to our mail server based on the connection activity we were seeing from RIM servers. In hindsight we should have implemented these separately to see which one was actually responsible for fixing the issue, but the focus was, of course, on fixing the issue, not messing around. That said, a third issue was that, after making the configuration changes, we restarted the mail server. This had the effect of closing down about two hundred connections to our mail server from RIM servers. About fifty connections were quickly re-established, and we expected this number to grow again to about two hundred … but it didn’t. After more than twenty-four hours the number of connections remains steady at about fifty. This is interesting in itself, as simply restarting the server and not making the configuration changes may actually have fixed the problem anyway, making us wonder why RIM had so many connections open to our server in the first place.

So for now the issue appears to be resolved. We’re continuing to monitor connections to our mail server by RIM servers, and we ask that, if you have any further problems with accessing your NinerNet-hosted email using your Blackberry, you contact us with the details as soon as possible. Thanks very much.

We’ve made a couple of configuration changes to the mail server and restarted it. By all accounts this appears to have fixed the problem, but we’re not convinced that this is the end of the issue. We’ll post another update here within the next 48 hours, and we’ll also be sending an email to all clients about this when that update is posted.

Thanks for your patience. As mentioned previously, we’re still looking for more information from Blackberry users about this, so if there is any information you can provide to us that you think might be useful, we’re all ears.