Web server performance on server NC018 has returned to normal. Thank-you for your patience.
Web server performance on server NC018 has returned to normal. Thank-you for your patience.
Some websites on server NC018 are loading quite slowly. This does not seem to be affecting other services such as email.
We are looking into the cause and will report back here.
We are closing this issue, although we have not found a cause. As far as we can tell, everything related to authentication on the server is working. Research on similar occurrences also shows the problem apparently resolving itself after a short time, and also points to a number of possible causes, including faulty operating system and software updates on client machines. However, in this case we saw the problem on different operating systems (Windows, Mac and Android) and different email programs. We also suspected anti-virus software, but again we heard of problems on computers running different anti-virus software.
Oddly, we could not reproduce the problem on any machines in our office, and so we had no primary references on which we could do research.
About the only commonality we could find — other than the server itself, of course — is that we believe all of the reporting clients are on VSATs. However, even that doesn’t fully explain the problem as if, for some reason, VSATs were the issue we should theoretically have seen the same problem on other servers. That said, the problem reported to us was only with downloading email, and the other mail server on which the bulk of our clients are hosted uses slightly different SSL technology; additionally our default instructions for that server do not mention using SSL. We saw a similar issue with VSAT connections several years ago, so if this is related it’s not unprecedented.
Another possibility that can’t be ruled out is network issues that may have interfered with encryption on the connection. Given the number of reports, such an issue (if it existed) would likely have been closer to the server than the client. If this issue reoccurs we will contact our upstream providers to see what they have to say.
While it is unsatisfactory to close an issue without having a smoking gun to identify it, we are going to do so at this time. If you do see this issue again, please contact support and we will again look at the issue.
Thank-you for your patience while we looked at this issue.
We’re now getting emails from clients telling us that the issue has apparently resolved itself. We’re continuing to investigate, but we have not done anything to the server, either before this issue started or since it was reported to us.
Updates will continue to be posted here.
We are getting reports from clients that they are getting SSL-related error messages when connecting to this mail server. We have not yet been able to reproduce these errors ourselves, but we are investigating to find out what’s happening.
We will post further updates here.
The web server on server NC018 sustained a denial-of-service attack between approximately 14:00 and 18:00 UTC on 7 March. This is why websites on this server were unavailable during this period. No other services (email or nameservers) were affected.
We have implemented improved monitoring of this activity on this server. Additionally, we are in discussions with the data centre to move this server to an improved network infrastructure, a move we hope to implement before the end of March.
We apologise for this disruption.
We have had reports that the web server on server NC018 was down in the late afternoon UTC. We restarted the web server at 18:07 UTC and it is up right now.
We’re looking into this and will report back here shortly.
Server NC027 (the new mail-only server) experienced a great and very brief increase in activity at about 04:00 UTC on 23 January, and we suspect that this lead to the server getting behind in scanning and processing email. The hard drives simply couldn’t keep up.
In order to allow the server to catch up, we disabled virus and spam scanning. Over the course of the coming week we’ll be re-enabling each of these (inbound and outbound) one at a time per day. This will allow us to determine which activity is the bottleneck and make finer adjustments accordingly. We also have a back-up plan to move the accounts on this server to one with faster (solid-state) drives, if necessary.
We apologise again for the inconvenience of significantly delayed email on Thursday. If you have any questions about this incident, please contact NinerNet support. Thank-you.
The spam on server NC023 (the relay server) has been cleaned up and the offending IP address blocked. Most of the legitimate email to destination servers that had been deferred as a result of the spam has also now been delivered; however, some mail servers are still not accepting email from this server, so that email will continue to be delayed a while longer until their systems are automatically updated to realise that this server is no longer spamming. This is currently only affecting about twenty messages.
We apologise for this interruption. If you have any questions or concerns, please contact NinerNet support. Thank-you for your patience.
The relay server has been stopped while we clean up a spam outbreak, and identify and block the source. We will post updates here. We expect to have the server back online within the next 30-40 minutes.
Systems at a Glance:
Loc. | System | Status | Ping |
---|---|---|---|
NC023 | Internal | Up? | |
NC028 | Internal | Up? | |
NC031 | Internal | Up? | |
NC033 | Operational | Up? | |
NC034 | Internal | Up? | |
NC035 | Operational | Up? | |
NC036 | Operational | Up? | |
NC040 | Internal | Up? | |
NC041 | Operational | Up? | |
NC042 | Operational | Up? |
Subscriptions:
Search:
Recent Posts:
Archives:
Categories:
Links
Tags:
Resources:
On NinerNet: