NinerNet Communications™
System Status

Server and System Status

NC027: System maintenance

13 March 2014 10:38:17 +0000

We will be upgrading server NC027 over the weekend to increase its capacity. This maintenance will take about 40 minutes, during which the server will be completely inaccessible. However, incoming mail will be spooled on sending systems, and will be delivered when the server comes back up. No email will be lost or bounced.

Here are the details:

  • Start of maintenance window: 19:00 15 March 2014 UTC
  • Length of maintenance window: 1 hour

While the length of the maintenance window is an hour, the actual length of the maintenance should only be about 40 minutes sometime during that hour. If either the maintenance window or the length of the maintenance should change, we will post updates here.

To convert the above time and date to your time zone, please visit the World Time Server website.

If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience while we work to improve the service that we offer to you.

NC027: Investigation findings

26 January 2014 16:00:09 +0000

Server NC027 (the new mail-only server) experienced a great and very brief increase in activity at about 04:00 UTC on 23 January, and we suspect that this lead to the server getting behind in scanning and processing email. The hard drives simply couldn’t keep up.

In order to allow the server to catch up, we disabled virus and spam scanning. Over the course of the coming week we’ll be re-enabling each of these (inbound and outbound) one at a time per day. This will allow us to determine which activity is the bottleneck and make finer adjustments accordingly. We also have a back-up plan to move the accounts on this server to one with faster (solid-state) drives, if necessary.

We apologise again for the inconvenience of significantly delayed email on Thursday. If you have any questions about this incident, please contact NinerNet support. Thank-you.

NC023: Spam cleaned up

26 January 2014 15:45:01 +0000

The spam on server NC023 (the relay server) has been cleaned up and the offending IP address blocked. Most of the legitimate email to destination servers that had been deferred as a result of the spam has also now been delivered; however, some mail servers are still not accepting email from this server, so that email will continue to be delayed a while longer until their systems are automatically updated to realise that this server is no longer spamming. This is currently only affecting about twenty messages.

We apologise for this interruption. If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience.

NC023: Relay server stopped to clean up spam

26 January 2014 13:28:29 +0000

The relay server has been stopped while we clean up a spam outbreak, and identify and block the source. We will post updates here. We expect to have the server back online within the next 30-40 minutes.

NC027: Mail queue almost clear

23 January 2014 14:48:04 +0000

The mail queue will be cleared within the next 20-30 minutes.

Once the queue is clear we’ll be looking for a cause for this incident. There was a great spike in activity on this server around 04:00 UTC, and it’s quite possible that the initial back-up started around that time and didn’t quite manage to catch up.

NC027: New mail delivered immediately

23 January 2014 13:25:21 +0000

Since reconfiguring the mail server we’ve noticed that new email is being delivered right away. It is now only the backlog that is still slowly being processed.

NC027: Performance improvement

23 January 2014 11:55:41 +0000

We have made some adjustments to the configuration of the mail server on NC027 and as a result the mail queue seems to be clearing, albeit very slowly. We’re continuing to work on this to speed up delivery.

Server NC027 performance degraded

23 January 2014 09:07:10 +0000

Server NC027 has a large and growing mail queue, but the server is up. We are investigating and will post updates here.

Email migration: Update 14

29 October 2013 08:57:36 +0000

This is the last post that we’ll refer to as an “update” regarding the email migration that was largely completed three weeks ago … if only so that we don’t end on number 13. It addresses three issues:

  • Outlook 2003,
  • Anti-spam blacklists, and
  • Mail box quotas.

Outlook 2003: During the migration we learnt that Outlook 2003 does not support TLS. This is software that is over a decade old, and Microsoft will stop supporting it in less than six months. Now would be a good time to upgrade. However, apparently a 2004 “hotfix” available from Microsoft will add TLS support to Outlook 2003, but we cannot vouch for this personally, nor are we aware of any clients who have used this.

The anti-spam blacklists used on the old server were not immediately implemented on the new server. They have been now. The amount of spam you receive should drop significantly as a result.

Finally, we have increased mail box quotas across the board, as we try to keep up with the growing number of people using smart phones and tablets who store significant amounts of mail on the server.

As always, if you have any questions, pleaseĀ contact support and we’ll be happy to assist.

Email migration: Update 13

16 October 2013 10:09:43 +0000

Since the migration of many email accounts to the new server, we’ve had reports of email from some regular correspondents (with email hosted outside of NinerNet) to domains hosted on the new server bouncing back to those senders as undeliverable. All of these reports, so far, are about the same improper configuration of Microsoft Exchange mail servers.

A person sending you an email through a mis-configured mail server will receive a bounce message that includes an explanation for the bounce that looks like this:

you@yourdomain.com
nc027.ninernet.net #554 5.7.1 <senderdomain.local>: Helo command rejected: Go away, bad guy (.local).

The problem is the “senderdomain.local” string. In this case “senderdomain” stands in for an actual name — e.g., something that looks like it might be a domain — followed by “.local”. A properly configured mail server that connects to the public Internet is supposed to advertise a “fully-qualified domain name” (FQDN) through the “HELO” (or “EHLO”) command rather than “something.local”, which is not a real domain. Many mail servers, including ours, reject attempts to deliver mail from improperly configured mail servers advertising a “domain” that does not (or cannot) exist. The reason for this is that much spam comes from machines that are improperly configured in this manner. More technical details about this can be read in theĀ Best Practises for Email and Network Operators – Valid HELO domain article.

Your correspondents will likely think that we are blocking their domain specifically (very likely that we are NOT) or that something is otherwise wrong on our mail server. However, it is the other way around; your correspondents experiencing this problem need to talk to their own IT people, perhaps pointing them to this post, as their mail server needs to be reconfigured correctly.

The article Exchange DNS Configuration for Email Delivery includes a number of helpful hints for the Exchange server administrator about how to properly configure an Exchange server to work correctly on the Internet with respect to domains and DNS. About half way down the page are sections entitled SMTP Banner – Exchange 2003 and SMTP Banner – Exchange 2007 that explain how to set the SMTP banner — i.e., the domain that is advertised by the Exchange server when it connects to another mail server to attempt to deliver email. As mentioned previously, this needs to be a proper domain that is resolvable on the Internet, not something that doesn’t exist like “senderdomain.local”.

Our experience is that when an Exchange server is correctly reconfigured, email from that server starts getting through again immediately, and deliveries to other servers that do not block based on this incorrect behaviour are not affected.

Another possible solution to this problem is for the Exchange server to use a smart host, through which all outbound email is delivered to the public Internet. This has a number of advantages, including not having to reconfigure the SMTP banner and the fact that the server administrator doesn’t have to be concerned about their own IP address being added to a block list if (again as a result of mis-configuration) the server inadvertently becomes the source of spam. NinerNet provides this service (relay server / smart host) for USD30 / CAD36 / ZMW165 per month.

Or you could send Microsoft Exchange Server 2007 For Dummies to the sending domain’s server administrator!


Update, 2022-01-24: The information above applies to any domain or sub-domain used in a mail server’s HELO command, not just the specific nonsense sub-domain “senderdomain.local”. If the maintainer of the sending mail server makes up a sub-domain like “mailserver.mydomain.com”, but doesn’t actually create an A record for “mailserver.mydomain.com”, then the effect will be the same, their email will not get through.

Additionally, these days the error message is different. It is as follows:

450 4.7.1 <mailserver.mydomain.com>: Helo command rejected: Host not found

Mail server admins are still making this mistake today, in 2022!

NinerNet home page

Systems at a Glance:


Loc.SystemStatusPing
Server NC023, London, United Kingdom (Relay server), INTERNAL.NC023InternalUp?
Server NC028, Vancouver, Canada (Monitoring server), INTERNAL.NC028InternalUp?
Server NC031, New York, United States of America (Web server), INTERNAL.NC031InternalUp?
Server NC033, Toronto, Canada (Primary nameserver), OPERATIONAL.NC033OperationalUp?
Server NC034, Lusaka, Zambia (Phone server), INTERNAL.NC034InternalUp?
Server NC035, Sydney, Australia (Secondary nameserver), OPERATIONAL.NC035OperationalUp?
Server NC036, Amsterdam, Netherlands (Mail server), DEGRADED.NC036DegradedUp?
Server NC040, Toronto, Canada (Web server), INTERNAL.NC040InternalUp?
Server NC041, New York, United States of America (Web server), OPERATIONAL.NC041OperationalUp?
Server NC042, Seattle, United States of America (Status website), OPERATIONAL.NC042OperationalUp?

Subscriptions:

RSS icon. RSS

Twitter icon. Twitter

Search:

 

Recent Posts:

Archives:

Categories:

Links

Tags:

.co.zm domains .com.zm domains .zam.co domains back-up bounce messages browser warnings connection issues control panel database dns dos attack dot-zm domains down time email email delivery error messages ftp hardware imap mail mailing lists mail relay mail server microsoft migration nameservers network networking performance php phplist pop reboot shaw shaw communications inc. smtp spam spamassassin ssl ssl certificate tls tls certificate viruses webmail web server

Resources:

On NinerNet: