23rd March 2020

Mailbox outage

Title: Mail client Syncing Issue Date: Monday 23rd March 2020, 14:46 UTC

We apologise for the issues that some of our users are continuing to face when attempting to access their hosted mail accounts.

As things stand, any messages sent to your mailbox during this outage will continue to be delivered, however, access to these mailboxes remains limited at present.

Technical Summary:

--

On the afternoon of the 23rd March 2020 at approximately 14:52 UTC, one of our mail platforms encountered an operational error which subsequently sent a significantly higher load to other nodes within the mail cluster. To address this load, our mail team took the decision to disable individual mailboxes deemed to be causing excess load.

Replacement nodes have been introduced into the environment and accounts are in the process of migrating to these devices, however, due to the excess I/O introduced by the excessive load and the account transfers. We have had to temporarily suspend logins to certain accounts on affected nodes.

--

Apologies once again for the frustration must no doubt be causing you. Our support team has received an unprecedented volume of calls and messages today regarding this issue and we appreciate your patience whilst we work to update you during this stressful time. Timed updates can be found below.

  • UPDATE @ 25th March 2020, 15:18 UTC - Our Operations team have enabled IMAP/POP/Webmail. Email deliveries will be resumed shortly. Updates will be provided as soon as possible

  • UPDATE @ 25th March 2020, 14:03 UTC - We are seeing high load again. Users may notice slowness or intermittent issues to login. Updates will be provided as soon as possible

  • UPDATE @ 25th March 2020, 13:04 UTC - We have fully implemented our fix, any email issues are now resolved and no more accounts should be offline. We will continue to monitor the situation and provide updates when we can.

  • UPDATE @ 25th march 2020, 07:35 UTC - Our operations team has ended the maintenance on one of the mail clusters by enabling all of the mailstores. A small number of accounts will still not have webmail or email client access as we prepare for another batch migration to be done during a lull period.

  • UPDATE @ 25th march 2020, 07:06 UTC - Our ops team is still currently working on this issue at hand. At this time, only users on the still disabled mailstore are expected to be completely unable to utilize hosted email. Users from the other mailstores which were enabled will still see a degraded performance at this time.

  • UPDATE @ 25th march 2020, 00:26 UTC - An emergency migration process is being looked at by our operations team to help restore access. Unfortunately, this does still mean that a handful of users will be unable to login to their account via webmail or their email clients. Once again we do apologise for the discomfort this is causing, we are doing our best to resolve the issue as swiftly and smoothly as possible.

  • UPDATE @ 24th march 2020, 22:59 UTC - The operations team is still monitoring the service, unfortunately, it is momentarily experiencing an unusually high load until such behavior stabilises the email deliverability cannot be restored. Rest assured that we are working hard on the issue as we are aware of the implications this is causing for many users.

  • UPDATE @ 24th march 2020, 17:38 UTC - Our operations team is continually working on restoring access to accounts and their mailboxes incrementally. We will update again once we have any further information.

  • UPDATE @ 24th March 2020, 15:09 UTC - A number of accounts have been brought back online though service to these has been observed to be intermittent. We're continuing to restore service to all affected users as swiftly as possible.

  • UPDATE @ 24th March 2020, 13:53 UTC - We are currently still working on implementing the fix. During this time, users may experience problems accessing their mailboxes. For now, email is still being delivered to users inboxes, even if you cannot log in at this time.

  • UPDATE @ 24th March 2020, 12:05 UTC - We are still working on the fix. We will update again once we have more information.

  • UPDATE @ 24th March 2020, 10:50 UTC - The implementation of a fix appears to be progressing well. A further estimate of the completion time has not yet been made. We will update again once we have this information

  • UPDATE @ 24th March 2020, 09:08 UTC - The issue appears to be progressing well. During the maintenance, users may be unable to access our email services and inboxes may appear empty, however email service would resume shortly after. We will post the updates as we receive.

  • UPDATE @ 24th March 2020, 05:20 UTC - Issue still progressing, update when further information is available

  • UPDATE @ 24th March 2020, 03:30 UTC - The issue appears to be progressing well. A further estimate on the completion time has not yet been made. We will update again once we have this information.

  • UPDATE @ 24th March 2020, 01:00 UTC - The issue is still being worked on by our Ops teams, again your patience is appreciated during this time.

  • UPDATE @ 23rd March 2020, 23:00 UTC - Our Ops team are still diligently working on the issue. We appreciate your continued patience.

  • UPDATE @ 23rd March 2020, 21:30 UTC - We are currently having emergency maintenance to bring service back online.

  • UPDATE @ 23rd March 2020, 19:40 UTC - There is no change in the status of service impact, however, the failover is almost complete. Your patience is greatly appreciated.

  • UPDATE @ 23rd March 2020, 18:00 UTC - Our Ops dept are still investigating the issue as a priority. Please accept our apologies for the potential impact this incident may have caused.

  • UPDATE @ 23rd March 2020, 15:18 UTC - Our technical team have identified the cause of the issue preventing some users from logging in via webmail and are working with our mail service team to resolve these errors.