NinerNet Communications™
System Status

Server and System Status

NC036: Migration update 20 — Solutions

11 June 2018 04:08:41 +0000

Summary

We suspect that clients having problems sending or receiving email have very old legacy configuration settings. Please see the “Email server settings” section below for the definitively correct settings.

Situation

Over the weekend we took a deep breath and stepped back to re-analyse this problem, and consult with a number of you. Between…

  • a move to a new server in a new data centre,
  • and then to another data centre to try to outrun the phantom issues at the first data centre,
  • the conclusion that Zambian ISPs were somehow complicit (and then even having that explicitly confirmed [and then retracted] by someone in the IT Security Department at MTN!) because we’d had no complaints from other countries,
  • and the fact that the new server was somehow processing just as many messages as it normally does despite so many people apparently being unable to send and/or receive,

.. we were awash in red herrings to an extent I have never seen in 22 years.

We’ve taken a look at the behaviour of two of the most used email programs (Thunderbird and Outlook) and come to some conclusions about what might be happening:

  • The fact that most clients carried on connecting with no problems tells us that (a) the server was operating normally, but (b) some clients were using old (in some cases very old) settings that were permitted (but not recommended) on the old server, but no longer permitted on the new server due to the ever-increasing need to raise the bar on server security.
  • Some email programs (notably Thunderbird and various incarnations of the Apple Mail app) tend to funnel all outgoing email through a single SMTP (outgoing) account. This can lead to situations where someone might be trying to send an email from you@domain1.com, but trying to log in as other-address@domain2.net. Again, with the ongoing need to tighten email security, this is no longer permissible with just about every mail service provider in the world.
  • A lot (probably most, actually) of email programs and apps try to second-guess your selection of a port number, often after you think you’ve saved your email configuration.
  • Over the years we’ve seen some email programs and apps treat SSL and TLS in odd and unpredictable ways. The existing settings we’ve always given out still work, but in the interests of getting everyone on the same page we’re starting with a clean slate.

So, if you’re having problems sending, it will likely be worth your while to check your SMTP (outgoing) settings; if you’re having problems receiving, it will likely be worth your time to check your POP or IMAP (incoming) settings. I wanted to have some screenshots ready for this post, but I’d rather get the words up now and post screenshots shortly afterwards, so here are the settings you need to use:

Email server settings

  • Email address: you@yourdomain.com
  • User name: you@yourdomain.com (same as your email address)
  • Password: The correct password on your email account. If you’re not sure what it is, please reset it to a new one through the email control panel (admins only). It can also be reset through the webmail.
  • Password type: Plain
  • Incoming (POP/IMAP) mail:
    • Server: pop.niner.net or imap.niner.net
    • Port: 110 (POP) or 143 (IMAP)
    • Encryption: STARTTLS
  • Outgoing (SMTP) mail:
    • Server: smtp.niner.net
    • Port: 587
    • Encryption: STARTTLS
    • Authentication: Turned on
  • To send email, you must log in with the same user name (address) as the address you’re sending from.
  • Some older mail programs may not offer STARTTLS; if that’s the case for you, try TLS and/or SSL, in that order.

Additional information

I can’t emphasise strongly enough how important it is for you to be precise in setting up this configuration. No setting is “close enough”, and your computer is not smart enough to figure it out; it will just tell you there is an error. Although, having said that, I’d like to emphasise that the niner.net sub-domains with “27” in them — i.e., pop27.niner.net, imap27.niner.net and smtp27.niner.net — do still also work, but they will be phased out; do not use them.

In the case of those email programs that like to railroad you into sending all email through a single SMTP account by default, we suggest that you start with a clean slate there too by deleting all of the saved SMTP accounts (unless you have some on systems that are completely separate from NinerNet) and creating a new one for each of your email accounts. Because your email program may not let you delete the “default” SMTP account, you’ll need to make a new SMTP account the new default, and then delete the old default.

We will post helpful screenshots as soon as possible. In the meantime, please check (and, if necessary, update) your email account settings and ensure that they are correct.

Thank-you.

NC036: Migration update 19 — MTN blocking us

8 June 2018 07:23:34 +0000

I have just got off the phone with someone in IT security at MTN head office in Lusaka, and they confirm that they have been blocking our new mail server as part of a wrong-headed plan to prevent MTN users from sending spam. It is likely that the first new mail server was also being actively blocked. He says that our IP addresses will be unblocked within the next ten minutes.

This raises the significant question of whether or not this is now an Africa-wide policy with many other ISPs. Other countries manage to prevent their users from sending spam without holding the keys to a gateway to the Internet, forcing companies like NinerNet to supplicate themselves to the likes of big companies like MTN when we find our businesses held hostage.

This is why we sent the questionnaire out yesterday asking you for details on whether nor not you are still having problems, and for the details of your ISP. Please reply to those emails so that we may determine which ISPs are actively blocking our servers and take the appropriate action.

NC036: Migration update 18 — MTN LTE

7 June 2018 11:12:19 +0000

We have had this report from a client:

I have now reset my LTE unit in our office to factory default and mails are working again on MTN, weird…We will monitor and see if it goes off again

NC036: Migration update 17

7 June 2018 09:12:19 +0000

We continue to track the intermittent connections in Zambia. They simply don’t make sense. For example, some MTN customers have no problems connecting, but some do. And some people can connect on MTN, but not Realtime/HAI, or they can connect on Paratus, but not MTN.

But we are slowly managing to narrow things down with a resolution in mind.

We did receive a call from a client who has talked to at least one ISP up on the Copperbelt, and they informed him that they allow some connections but not others, and they allow some connections intermittently such that it works one minute and stops working the next. This is exactly the behaviour our clients are seeing, and it seems to be intentional on the part of at least one Zambian ISP! Now, these are very vague statements, but our client asked us for an email explaining how our system works and is configured that he could send to them. Herewith a copy of our email:

Thanks for your phone call. As I said on the phone, this mail server operates in exactly the same way as the old mail server. There is simply no way to operate a mail server on the Internet that does not conform to the same interoperability standards as every other mail server on the Internet. Sure, the are minor variations on how some things are done internally on all servers, but for server A to talk to server B and deliver an email — or for a personal computer or phone to get that email to server A in the first place — they all have to be talking the same language.

Also, I find it very difficult to understand an ISP saying that they allow some standard behaviour and disallow other standard behaviour. And it’s even more bizarre that they say they allow some behaviour intermittently; what’s the point of that?!

With that editorial out of the way, this is the configuration of both the old and new mail servers:

SOFTWARE:

  • MTA (mail transfer agent, i.e., mail server software, SMTP): Postfix
  • MDA (mail delivery agent, i.e., POP and IMAP): Dovecot
  • Web server (control panel and webmail): Nginx

PORTS (all TLS/SSL):

  • POP: 110/995
  • IMAP: 143/993
  • SMTP: 587
  • Web: 443

This is a 100% standard configuration, and as I’ve said before, is exactly the same as it was on the old server … EXACTLY the same.

Any ISP is welcome to contact me directly, by email or phone, to explain why users on our system should be subject to some sort of arbitrary blocking of anything. And they’re welcome to contact me just to ask questions or for a friendly chat. Everyone in the world (barring repressive dictatorships, which I don’t think Zambia has become just yet) uses these same port numbers and configurations.

Please keep me informed. Thanks.

Craig

NC036: Migration update 16

7 June 2018 05:43:46 +0000

Are you wondering if our mail server is really up or if we’re “having problems”? We could be lying, but this third-party service will uncover our lies:

https://downforeveryoneorjustme.com

Every time we check, mail.niner.net and webmail.niner.net are up. Please check for yourself. In fact, we suggest contacting your ISP and asking them why you cannot reach a server that is alive and well.

You can also check the pop, imap and smtp sub-domains of niner.net, as well as the old pop27, imap27 and smtp27 sub-domains, all of which are working.

We actually do strongly urge you to contact your ISP about the fact that you can only intermittently connect to our mail server. They are the only ones who can help you with your connection to the Internet when it is not working properly.

NC036: Migration update 15 — Possible return of connection issues

6 June 2018 16:35:16 +0000

As happened on Monday, after initial success with setting up the new mail server, we’re again receiving reports from clients in Zambia (so far) that are starting to have their connections to the server dropped by their ISPs. This is incredibly frustrating, certainly for you, of course, but also for us.

If this is happening to you, you can help us help you by submitting what’s called an MTR report. MTR is a network diagnostic tool that gives us quantifiable information we can analyse and (hopefully) act on, showing where the network problem affecting your email lies so that we may possibly contact the organisation responsible for the problem.

If you’re running Windows, please download WinMTR. The file you will download is just a zip file that contains the actual program (32- and 64-bit versions). Read the “README.TXT” file for instructions on which version to use. All you need to do is double-click the appropriate file and the program runs; it’s not actually installed on your computer. Then type nc036.ninernet.net (yes, that’s ninernet.net, NOT niner.net) in the box as prompted, and select the start option to begin generating report data. Once you’ve gathered enough data, please use the copy or export functions to send us the report.

Please also send us the following information:

  • Your IP address.
  • The name of your ISP. If you have access to multiple connections, an MTR report from each ISP would be greatly appreciated.
  • If you have a reliable contact at that ISP, his/her name, position, phone number and email address.

Please email this to our usual email address, or send it via our contact form.

Thanks.

NC036: Migration update 13

6 June 2018 12:52:23 +0000

We will post a postmortem here in due course, hopefully with 24-48 hours, along with a thousand more apologies, but we are looking for feedback to ensure that all clients are able to connect to the server and download and send email, as this was not the case on Monday and Tuesday.

NC036: Migration update 12 — server back online

6 June 2018 12:28:04 +0000

The transfer of the mail spools has completed and server NC036 was brought back online at 12:12 UTC.

NC036: Migration update 12

6 June 2018 08:32:10 +0000

The transfer of mail data between the old and new data centres is still underway. Considering the transfer of the same amount of data took 50 minutes over the weekend, and we are now at the 5-hour mark, it was impossible for us to predict that this would take so long.

I can assure you that I understand the frustration that you are feeling with this situation, but given the network problems in southern and central Africa that necessitated this emergency move, we had no choice but to act immediately rather than forcing many clients to do without mail until the weekend.

My best estimate at this point, based on how much data has transferred so far (55%) and how much is left, is that the transfer will complete at approximately 13:00 UTC. Assuming this is the case, the server will be back online and accessible at about 13:30 UTC.

I sincerely and deeply apologise for this situation.

NC036: Migration update 11

6 June 2018 04:42:04 +0000

The transfer is taking significantly longer than we anticipated, likely due to the greater distance between the two data centres.

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), OPERATIONAL.NC036OperationalUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: