Server NC031 is back online, although we are waiting for word from the data centre that the issue is definitely completely resolved.
Server and System Status
NC031: Server back online
18 August 2018 01:28:10 +0000NC031: Networking issues at data centre
18 August 2018 01:11:02 +0000At 00:09 UTC on 18 August we became aware of a networking issue at the data centre where NC031 — the primary web server — is located. This is affecting almost all web hosting clients, including the primary NinerNet website. The web server itself is up and running, as far as we know, but the problem is that traffic to and from the data centre is down.
This issue does not affect any email services.
Data centre staff are working on restoring connectivity, and we will post an update as soon as we are aware that the web server is once gain accessible. We apologise for this issue.
NC031: Data centre power failure
19 July 2018 05:09:49 +0000At 23:04 UTC on 18 July connectivity issues were reported at the New York, US, data centre. The cause of this problem was later identified as a power outage. The data centre is currently reporting that power was restored to the data centre at 00:38 UTC on 19 July, but our own logs on the server itself indicate that it was only down between 22:55 and 23:50 UTC on 18 July. We are expecting a full report from the data centre and will post further details here when we receive that.
This affected our primary web server, hosting most clients’ websites, including our own primary website (www.niner.net).
We later soft-rebooted the server at 04:07 UTC on 19 July, and it was back online at 04:08.
This outage affected automated daily back-ups on the server. These have been restored and will run as scheduled at the next scheduled time, which is at midnight UTC on 20 July.
We apologise for this outage. The data centres we select are supposed to have redundant power systems to ensure that this kind of event never happens. However, clearly it did in this case and, as mentioned above, we are expecting a follow-up report from the data centre to explain this failure.
If you have any questions or concerns, please contact us to let us know. Thank-you, and we again apologise for this incident.
NC036: Migration update 25 — Final
18 June 2018 08:54:43 +0000The migration of all email accounts from server NC027 to server NC036 is complete. In fact, it was successfully completed at 04:00 UTC on 4 June. What followed over the next few days was an unprecedented avalanche of misinformation and red herrings that resulted in our moving the new server to another data centre (a move that took ten times longer than the previous move from the data centre where NC027 was located) where the same “problems” experienced by only some of our clients magically reappeared.
We planned the migration to have absolutely no impact on existing email configurations. We did this by pointing legacy sub-domains of the niner.net domain that named server NC027 — e.g., smtp27.niner.net — to server NC036. At the conclusion of the migration these sub-domains were indeed pointing to the new server. In other words, on Monday morning (4 June) email programs would have thought they were still downloading mail from the same server, not realising (or needing to realise) that they were in fact downloading from a new server.
However, it turned out that a significant minority of email programs were somehow misconfigured with settings that worked on the old server, but stopped working when connecting to the new server. Those clients who were using the correct settings experienced no disruption at all, and when those clients with incorrect settings corrected them on the morning of Monday the 11th, the problems were fixed instantly.
Over the rest of that week (11-15 June) we helped a few clients with some issues unique to how they use email, especially where those practices clashed with current best practices for email transmission. We also dealt with some issues of senders whose mail servers were behaving improperly, causing their emails to be blocked because they looked like spammers. This notably affected email from the ZRA, but their emails are once again flowing unimpeded.
We’re monitoring the spam filtering on the new server. Any message that the server identifies as spam will have the subject of the message prefixed to add “[SPAM]“. You can use this to configure your email program or the webmail to deal with spam automatically, by filtering it into your “junk” folder or deleting it entirely. We recommend filtering to the junk folder so that you can catch the occasional legitimate message that is misclassified as spam.
Finally, in recognition of the fact that the emergency migration of the server to a new data centre on 6 June disrupted all clients’ email, and the fact that those clients with misconfigured email programs experienced a week of disruption before the issue was identified, we will be applying a one-week (quarter month) credit to the accounts of all clients hosted on server NC036. We apologise for the difficulties caused, and will apply what was learned this time to future migrations.
Thank-you, as always, for your custom and patience.
NC036: Migration update 24 — Outlook weirdness
11 June 2018 11:35:44 +0000Microsoft are famous for ignoring standards and inventing their own so-called standards, and over the years we’ve seen evidence of that in Outlook.
Numerous clients today report that the following applies to Outlook, at least Outlook 2016:
There is no option for STARTTLS for incoming (POP/IMAP) settings. As such, you need to use the older and deprecated option of using port 995 (POP) and 993 (IMAP) over SSL.
NC036: Migration update 22 — A word about forwarding email
11 June 2018 08:36:59 +0000Over the years we’ve noticed that a certain percentage of our clients are in the habit of forwarding all of their email to external free webmail services — e.g., Yahoo, Hotmail, Gmail, etc. Why do we even notice this? Well, because these free services often delay your email, and so it queues on our server for anywhere between minutes and days. There are complicated reasons for this, but once you realise that you’re not the only one forwarding your email, you can see how these free webmail services might decide to limit the number of messages that they accept from our servers. This is especially noticeable when (not if) a few spams get through and (ironically) the receivers — the very NinerNet clients who have configured their email accounts here to forward to their free webmail provider — complain to the free webmail provider about the spam by clicking the convenient “this is spam” button. The free provider then responds by blocking or limiting mail from our server, making the reporting of the spam by the NinerNet client self-defeating!
Among other reasons, what people who do this are running into here is introducing multiple points of failure. If a message arrives on the NinerNet mail server, it’s made it! It has arrived where it was intended by the sender to be delivered. But now you’ve told our server to forward it somewhere else. It’s like telling a runner at the finish line that he has to do the same race again. And the runner might not make it the second time, just as your email might not make it into your Gmail account.
Right now there are a few dozen emails queued on our server waiting to be accepted by these free email services. Given that some of them have been queued for several days, most of them will likely bounce back to the senders within the next few hours. There is nothing unusual about this; we see it all the time, and it has little (if anything) to do with the mail server migration.
If webmail is your preferred way of accessing your email, we do (obviously) provide webmail on your own domain. (And non-Gmail webmail these days is way better than it used to be!) If you prefer the webmail offered by your free provider of choice, that’s fine, as long as you’re aware of the inherent risks of delayed and bounced email if you choose to forward everything.
If you’d like to discuss alternatives to forwarding your email, let us know and we can provide options to you or address any concerns you may have.
NC036: Migration update 21 — Screenshots
11 June 2018 07:02:05 +0000Here are the promised screenshots that show how an email program like Thunderbird should be configured.
-
- This is the initial screen when you manually add a new account to Thunderbird. (We will have automatic configuration available by next week.) For POP instead of IMAP, please just select POP from the drop-down list to the right of the “Incoming:” heading, where “IAMP” shows in this screenshot.
-
- These are the advanced incoming (IMAP) settings. Nothing materially different here, just more settings that we suggest.
-
- SMTP servers. Here it is possible that you may have too many or too few SMTP servers. If you’re having problems sending, we suggest that you delete any superfluous SMTP servers and create one SMTP server that corresponds with each incoming email account you have.
-
- These are the correct SMTP configuration details.
We will have Outlook screenshots available as soon as possible.
NC036: Migration update 20 — Solutions
11 June 2018 04:08:41 +0000Summary
We suspect that clients having problems sending or receiving email have very old legacy configuration settings. Please see the “Email server settings” section below for the definitively correct settings.
Situation
Over the weekend we took a deep breath and stepped back to re-analyse this problem, and consult with a number of you. Between…
- a move to a new server in a new data centre,
- and then to another data centre to try to outrun the phantom issues at the first data centre,
- the conclusion that Zambian ISPs were somehow complicit (and then even having that explicitly confirmed [and then retracted] by someone in the IT Security Department at MTN!) because we’d had no complaints from other countries,
- and the fact that the new server was somehow processing just as many messages as it normally does despite so many people apparently being unable to send and/or receive,
.. we were awash in red herrings to an extent I have never seen in 22 years.
We’ve taken a look at the behaviour of two of the most used email programs (Thunderbird and Outlook) and come to some conclusions about what might be happening:
- The fact that most clients carried on connecting with no problems tells us that (a) the server was operating normally, but (b) some clients were using old (in some cases very old) settings that were permitted (but not recommended) on the old server, but no longer permitted on the new server due to the ever-increasing need to raise the bar on server security.
- Some email programs (notably Thunderbird and various incarnations of the Apple Mail app) tend to funnel all outgoing email through a single SMTP (outgoing) account. This can lead to situations where someone might be trying to send an email from you@domain1.com, but trying to log in as other-address@domain2.net. Again, with the ongoing need to tighten email security, this is no longer permissible with just about every mail service provider in the world.
- A lot (probably most, actually) of email programs and apps try to second-guess your selection of a port number, often after you think you’ve saved your email configuration.
- Over the years we’ve seen some email programs and apps treat SSL and TLS in odd and unpredictable ways. The existing settings we’ve always given out still work, but in the interests of getting everyone on the same page we’re starting with a clean slate.
So, if you’re having problems sending, it will likely be worth your while to check your SMTP (outgoing) settings; if you’re having problems receiving, it will likely be worth your time to check your POP or IMAP (incoming) settings. I wanted to have some screenshots ready for this post, but I’d rather get the words up now and post screenshots shortly afterwards, so here are the settings you need to use:
Email server settings
- Email address: you@yourdomain.com
- User name: you@yourdomain.com (same as your email address)
- Password: The correct password on your email account. If you’re not sure what it is, please reset it to a new one through the email control panel (admins only). It can also be reset through the webmail.
- Password type: Plain
- Incoming (POP/IMAP) mail:
- Server: pop.niner.net or imap.niner.net
- Port: 110 (POP) or 143 (IMAP)
- Encryption: STARTTLS
- Outgoing (SMTP) mail:
- Server: smtp.niner.net
- Port: 587
- Encryption: STARTTLS
- Authentication: Turned on
- To send email, you must log in with the same user name (address) as the address you’re sending from.
- Some older mail programs may not offer STARTTLS; if that’s the case for you, try TLS and/or SSL, in that order.
Additional information
I can’t emphasise strongly enough how important it is for you to be precise in setting up this configuration. No setting is “close enough”, and your computer is not smart enough to figure it out; it will just tell you there is an error. Although, having said that, I’d like to emphasise that the niner.net sub-domains with “27” in them — i.e., pop27.niner.net, imap27.niner.net and smtp27.niner.net — do still also work, but they will be phased out; do not use them.
In the case of those email programs that like to railroad you into sending all email through a single SMTP account by default, we suggest that you start with a clean slate there too by deleting all of the saved SMTP accounts (unless you have some on systems that are completely separate from NinerNet) and creating a new one for each of your email accounts. Because your email program may not let you delete the “default” SMTP account, you’ll need to make a new SMTP account the new default, and then delete the old default.
We will post helpful screenshots as soon as possible. In the meantime, please check (and, if necessary, update) your email account settings and ensure that they are correct.
Thank-you.
NC033: Maintenance complete
11 June 2018 00:45:06 +0000Server NC033 is back online. It was down between 00:37 and 00:43 UTC.
