1930 BST(+0100) – The search ability of the North American mail archive’s has been restored and the archive search will now be returning results.
There will be delays in indexing of messages submitted to the archive since 1st September meaning that these will not be immediately searchable in the message archive although they have been successfully stored. We have initiated the re-indexing of the messages so these will become available once the indexing is complete.
This issue has been caused by a configuration database cluster issue, at no point has there been any risk to message storage. Our engineering teams are further investigating the cause of the issue, and putting further systems in place to ensure that this does not recur.
1400 BST(+0100) – Our monitoring has detected a condition in the North American Mail Archive servers, affecting the ability for customers in this region to carry out archive searches.
This is only in the North American region, EU customers will be unaffected. The ability of the archiving of messages for all regions is also not affected, only the searching of the message archive.
This is currently being worked on by our engineering team and further updates will be posted when they become available.
We’re excited to announce the release of MAX ServiceDesk v2.4, which went live today (3rd September). This release includes a range of enhanced features, including new Invoicing Export capabilities, Customer Permissions, and Customer Opportunities tracking.
11:15pm Pacific — Our engineers have restarted the affected mail handlers in the Amsterdam datacenter, and we are no longer seeing the network errors. A small percentage of messages may have been queued during this time. We will be updating our monitoring system so that if this particular issue recurs we will be able to detect it and take action quickly.
10:15pm Pacific — We have heard a few reports of mail handler servers at the Amsterdam datacenter being unable to connect to destination mail servers, with the result that some messages are being queued. The operations team is investigating.
9:30pm Pacific — Approximately two hours ago, mail handlers within the Amsterdam datacenter were responding slowly, with a number of network timeout errors generated. The response times stabilized within approximately an hour, and monitoring has been showing normal message processing since then. We will continue to monitor and will take any appropriate further action should that become necessary.
Tomorrow at 5pm BST (Aug 31)/ 1:30am ACST (Sept 1), Active Discovery will have a 10 minute outage while maintenance is performed in the Australian region. MAX RM Services outside of Active Discovery will not be impacted. This maintenance is isolated to the Australian Data Center.