Rackspace Exchange Server Outage.

As a customer affected this weekend by the Rackspace Exchange problems, and having followed comments on Twitter from users unsure how to proceed, I wanted to post something with a few basics that would hopefully help some people who may be struggling on the best way forward.

What’s happened?

Around 7am (GMT) on Friday 2^nd December, Rackspace, by their own admission, shut down access to their hosted exchange email platform. I’m not going to speculate as to why, but it seems they did this to protect their system and data. Further details from Rackspace can be found via their status page here - https://status.apps.rackspace.com/index/viewincidents?group=2

How can I restore email Sending and Receiving?

For now, the best way to continue to use your email addresses to receive and send mail is to move your email service to a new provider. This is also the advice being given by Rackspace themselves. They are suggesting people use Microsoft 365, via their platform and are offering free credit to do this. There have been some users on Twitter saying that solution didn’t work and required email verification (to an email account that is currently inaccessible). As such, my advice would be to go directly to Microsoft.

You could at this point move your email to any other provider, such as GoDaddy, AWS or anyone else who offers an email service.

How do I move my email elsewhere?

The key component here is having access to your domain name. If you have access to your domain name and its DNS settings, you can update the required settings and point your email data to a new provider. Usually, this will involve changing MX records, CNAME records and maybe adding a TXT record to your DNS. If you do use Office 365, they provide clear instruction for this, as will most decent email providers.

Some users have had problems accessing their domain name settings via their domain registrar as they needed email verification to login or making changes. If that is the case, you should speak to the company who controls your domain name ASAP and explain the situation, that you need access to your account / DNS to change mail provider, but don’t have access to email to get logged in or make changes due to the Rackspace problem. Hopefully, at that point they will help you get logged in.

I don’t have access to my DNS.

If you don’t have access to your domain name, then at this point there isn’t much you can do other than speak to Rackspace who might be able to offer an alternative solution. I received an email today offering a "mail forwarding" option from Rackspace; that might be one way to at least still receive email while the problem is investigated and hopefully fixed.

What happens to my Emails, Contacts and Calendar when I move to Office 365?

At the point you setup your new 365 account you’ll get an empty email, calendar and contact list. When you change your DNS, the mail will start to flow into this new account and you can also then send from it.

The complication then comes with old emails. If you're using Outlook for PC, it's likely you’ll have some (usually 12 months) email history stored locally on your PC. If you need ‘read-only’ access to that data in the short term, you should NOT delete your Rackspace account from your PC. By leaving it intact on your PC, you’ll be able to view recent emails (up to the point of Rackspace shutting it down) and also any future calendar appointments, and I believe most of your contacts, as this is stored / cached locally on your PC.

My advice for the time being would be to use the 365 ‘webmail’ interface to access your new Email accounts with 365, so you can leave your Desktop Outlook platform as it was and still access old data stored locally.

If you know how to export mail, you can always export all the data from your PC's Outlook program, then setup your new account and import that data. The only issue here is you’ll only be able to export whatever is cached on your PC. If that isn’t everything and you need all your data, you’ll need to wait for Rackspace to hopefuly restore access. When access is strored you can change your local settings download 'all mail' then export as required.

Can I go back when Rackspace is fixed.

Yes, at the point they restore all the Exchange services, you can also just change your DNS back to the Rackspace settings and your mail will flow back to Rackspace and your original account. The issue here is that any data sent to 365, or another provider, in the meantime will then be with Microsoft, not Rackspace. You could then export that and import back  into Rackspace after if you wanted.

I’ve got a hybrid Exchange and Rackspace Mail setup.

This is a tricky one. I believe at this point the Rackspace mail infrastructure is still working. If, for example, you have 1x Exchange and 5x Rackspace Mailboxes, your best bet would be to speak to Rackspace (if you can) and get the exchange address forwarded elsewhere or to a Rackspace mail inbox as a temporary solution. You won’t be able to reply directly from that account, but will at least receive your mail.

If you decide to move to a new provider, having had a hybrid setup, you’ll need to create a new mailbox for every user, both Exchange and Rackspace mail, in your new setup. As access to Rackspace mail is still ok, you can then make sure that is fully up to date, export the mail and import it into your new mail provider. 

The key point here is that as soon as you change your DNS / MX records, all mail will go to a new provider. If you have 10 mailboxes with Rackspace, you should create the exact same 10 mailboxes at your new provider, to avoid losing emails going forwward. 

What happens to emails sent to me during the downtime?

Emails sent to you between the point Rackspace shut down and the point you start receiving mail to a new provider will either bounce back, or be stored at Rackspace. That isn’t very clear at this point. I’ve started to get a lot bounces now I’ve got mail flowing again. Much of this depends on how many times and for how long the sending server will keep trying and also what Rackspace have done their end to hold / bounce incoming mail. Sorry if that isn’t very clear, but the information on this point from Rackspace is very limited.

It's best to assume at this point that mail sent to Rackspace, after the point of shutdown, will bounce back to the senders. This also means moving to a new provider quickly is important. The sooner you do this the sooner you’ll start to receive new emails and can start to email out to people who may have tried to contact you in the last few days.

What happens to my historic data with Rackspace?

At this stage we aren’t sure. That will depend on what happened and what they can do about it. If they are able to restore the service and data then at some point everyone will get access to the exchange and be able to access / sync history emails down, to export if required.

If the data at Rackspace is lost, damaged and now can’t be restored then unless you have backups / archiving you can expect not to get that data back in future unfortunately.

This is where your local Outlook mail program comes in. As I mentioned earlier, depending on your setup, it will likely have up to 12 months of cached data (more if your're lucky, less if you're not) stored on your local PC. If you leave that account setup as it currently is, still failing to connect to Rackspace, you can at least view those old emails, export them, drag them to another profile etc. The key here is DO NOT delete your Rackspace profile from your PC unless you have to and don’t care about the locally cached data.

I’ve moved to 365 or another provider, should I delete my Rackspace accounts?

NO… At this point, its worth keeping whatever setup you have at Rackspace in place and untouched. If they do fully restore the system, you’ll get access to your historic data and emails. If you delete your account/inboxes at Rackspace, you then won’t get any historic data back if/when they fix the service. This might mean paying twice, i.e. Rackspace and a new provider for a period of time, but if the old data is required then it's a price worth paying/. You may also be able to get some compenssation from Rackspace, but I'm not an expert on that. 

The key here is to ensure you can send / receive mail, via a new provider, but leaving everything locally and at Rackspace as it is, to wait and see what happens and if you can get your data back if they restore the system. That at least gives you options if they can fix everything.

Final thoughts...

This is obviously a huge problem for Rackspace, thousands of customers will be affected and so far, in my opinion, the updates from Rackspace have been very poor and unhelpful. Hopefully, as this goes on they'll start to give more information on next steps and/or the damage done. 

As frustrating, upsetting and stressful as this is for customers, the best thing you can do at this point to protect your email service and business is to get your mail flowing through a new provider. Recieving email in and sending out can be critical for many people and businesses, so this first step to restore your access to sending / recieving new mail is important. Don't just wait for Rackspace to fix this, it could be days or weeks before anything is done. The very fact they are telling their own customers to use Office 365 should tell you that this is VERY serious and won't be back online quickly.

I hope this 'blog' helps a few people. If anyone would like to reach out for further information or help, I'm happy to do so where I can. DM me on Twitter @jamiefryatt.

Good luck!

 

------------------------------
UPDATE 14:51 (GMT) 

Rackspace Created a Support Ticket - Forwarding Option for Hosted Exchange

About 9:30am GMT, I got an email that Rackspace have opened a Support Ticket. I haven't contacted them or support at all yet. Below is the email, which is essentially offering a temporary 'forwarding' option. 

I recieved the support ticket email to an email address that WAS affected by the outage before I switched yesterday. I'm not sure if that is because it was the 'email on account' with Rackspace, it probably is. It would be pretty daft to email all the affected accounts with this... 

-------------------

Hello,

Given the ongoing Hosted Exchange incident, we have created a temporary solution that will help with receiving new incoming emails. We can enable incoming mail forwarding for any Hosted Exchange mailbox on your account to a different, non-Hosted Exchange email address. You can have incoming mail forwarded to an account on Rackspace Email, Gmail, Hotmail, or any other mailbox hosted on an external platform. 

Please note that this configuration is temporary (as detailed below) but will allow you to receive copies of new email sent to your Hosted Exchange account following implementation and during this incident.

If you wish to proceed with this option, please reply to this ticket and provide the following:

For each user, please provide the Hosted Exchange mailboxes that you wish to enable forwarding on, as well as the destination email addresses that each mailbox should forward to in the following format:

To YourUser@YourDomain.com FORWARDS to YourUser@gmail.com

Example:
To Kelly@YourDomain.com FORWARDS to Kelly@gmail.com
To Jo@YourDomain.com FORWARDS to Jo@hotmail.com
To Casey@YourDomain.com FORWARDS to Casey@aol.com
To Kris@YourDomain.com FORWARDS to Kris@yahoo.com

We thank you for your patience and cooperation.

In order for Rackspace to proceed, Customer (or its authorized representative) must either reply to this ticket with the Hosted Exchange email address(es) and the destination email address(es) you would like to forward to, such response being sufficient to evidence that:
o       Customer acknowledges that this forwarding rule will be a temporary solution, is provided on an ‘as is’ and ‘as available’ basis, and Rackspace may remove forwarding at its discretion.
o       Customer acknowledges that once enabled, the destination address for the forwarded email cannot be updated. We can only enable this forwarding rule one time per Hosted Exchange mailbox.
o       Customer acknowledges that Rackspace is not responsible for any external email platform to which you may forward mail, interception or failure of forwarding, or any errors in destination email addresses.  PLEASE CONFIRM THE DESTINATION EMAIL ADDRESSES ARE SPELLED CORRECTLY AND ARE ACTIVE, WORKING MAILBOXES.
o       Customer has read, understood, and agrees to these terms which supplement and amend Customer’s Agreement with Rackspace.
o       The signatory (or responding person) has the legal authority to make this agreement on behalf of Customer.
o       This agreement is effective between Customer and Rackspace. 

IF YOU DO NOT HAVE THE AUTHORITY TO MAKE THIS AGREEMENT ON BEHALF OF CUSTOMER, PRINT THIS TICKET, OBTAIN THE SIGNATURE OF AN AUTHORIZED REPRESENTATIVE OF CUSTOMER, AND ATTACH THE SIGNED DOCUMENT TO THIS TICKET.

PLEASE DO NOT CLOSE THIS TICKET.

Thank you,
Your Rackspace Team