We are investigating a database issue on server NC031 that is affecting database-driven websites. We will report details here.
We are investigating a database issue on server NC031 that is affecting database-driven websites. We will report details here.
We are moving the problem website to a new server. In the meantime it will be suspended to avoid further disruptions.
We are experiencing problems with server NC031. These are caused by a very busy website that we transferred to this server about 48 hours ago.
We are assessing the situation and will post updates here. We apologise for the problem.
A configuration change on server NC027 (the mail server) over the weekend to once again introduce virus scanning has, as on previous occasions, overloaded the server. We made this change over the weekend after earlier doubling the capacity of the server and making other changes recently that should have seen a significant increase in server performance. Indeed, testing over the weekend showed a significant improvement over previous attempts to introduce virus scanning.
The result of this is that incoming emails have been delayed this morning, including messages between accounts on the same server. Therefore we have again reversed this configuration change and the mail queue has begun clearing. We hope that it will be clear within the next hour.
We apologise for this inconvenience.
Due to a recent configuration change, the mail queue on server NC027 was not able to keep up with this morning’s (UTC) mail volume, and mail was backlogged and delayed. We have reversed the configuration change and the mail queue is currently clearing.
We apologise for the inconvenience.
We have restarted the web server and are keeping an eye on it, specifically the sources of traffic to it.
We have taken down the web server while we investigate an overload condition. We will have it back up as soon as possible, and will post further updates here.
Email and other services on server NC018 are not affected.
Web server performance on server NC018 has returned to normal. Thank-you for your patience.
Some websites on server NC018 are loading quite slowly. This does not seem to be affecting other services such as email.
We are looking into the cause and will report back here.
Server NC027 (the new mail-only server) experienced a great and very brief increase in activity at about 04:00 UTC on 23 January, and we suspect that this lead to the server getting behind in scanning and processing email. The hard drives simply couldn’t keep up.
In order to allow the server to catch up, we disabled virus and spam scanning. Over the course of the coming week we’ll be re-enabling each of these (inbound and outbound) one at a time per day. This will allow us to determine which activity is the bottleneck and make finer adjustments accordingly. We also have a back-up plan to move the accounts on this server to one with faster (solid-state) drives, if necessary.
We apologise again for the inconvenience of significantly delayed email on Thursday. If you have any questions about this incident, please contact NinerNet support. Thank-you.
Systems at a Glance:
Loc. | System | Status | Ping |
---|---|---|---|
NC023 | Internal | Up? | |
NC028 | Internal | Up? | |
NC031 | Internal | Up? | |
NC033 | Operational | Up? | |
NC034 | Internal | Up? | |
NC035 | Operational | Up? | |
NC036 | Operational | Up? | |
NC040 | Internal | Up? | |
NC041 | Operational | Up? | |
NC042 | Operational | Up? |
Subscriptions:
Search:
Recent Posts:
Archives:
Categories:
Links
Tags:
Resources:
On NinerNet: