Rackspace Hosted Exchange Ransomware Attack: Email Outage Timeline and Recovery Updates

Rackspace confirmed a Hosted Exchange ransomware attack has knocked out email service to customers. The attack “may result in a loss of revenue for the Hosted Exchange business, which generates approximately $30 million of annual revenue,” Rackspace disclosed on December 6 — four days after the ransomware attack occurred.

Rackspace is a multi-cloud MSP. As of December 9, 2022 at 8:20 a.m. ET, the company is “unable to provide any timeline or expectations for restoration to the Hosted Exchange environment.”

The security incident comes at a critical time for Rackspace. The San Antonio, Texas-based company ranks among the world’s Top 250 Public Cloud MSPs. Still, Rackspace faces growth challenges on multiple business fronts. The evidence: Rackspace’s market valuation is roughly $1.02 billion as of December 4, 2022 — down roughly 65% over the past year, according to SeekingAlpha. Amid that backdrop, the company has overhauled its executive team, reorganized and considered potential asset sales over the past year.

Rackspace Hosted Exchange Outage: Microsoft 365 Workaround

As a workaround, the company is offering free Microsoft 365 subscriptions as a workaround to impacted customers. Moreover, Rackspace has mobilized 1,000 support professionals to help Hosted Exchange customers with the migrations to Microsoft 365 — but that migration process involves manual tasks that have frustrated some customers.

To help mitigate the migration challenges, Rackspace is partnering with “Microsoft’s Fast Track team to add resources to our extended team to better assist customers with troubleshooting and any technical questions.”

Rackspace ($RXT) has hired “world-class external expertise” to assist with the security incident investigation, the company said, though specific MSSP and incident response company names were not disclosed. The incident started early on December 2. As of early December 6, Rackspace does not have an ETA for Hosted Exchange system recovery.

Rackspace Hosted Exchange Security Incident Timeline

Here is a timeline of the Rackspace Hosted Exchange security incident, investigation and email recovery efforts:

  • Friday, December 2, 2:49 a.m. ET: Rackspace discloses that it is investigating an “issue that is affecting our Hosted Exchange environments.”
  • Friday, December 2, 9:38 a.m. ET: Rackspace says “All hands are on the deck & right resources have been engaged and are actively working on the issue.”
  • Friday, December 2, 8:19 p.m. ET: Rackspace provides affected customers with free access to Microsoft Exchange Plan 1 licenses on Microsoft 365 “until further notice.”
  • Saturday, December 3, 1:57 a.m. ET: Rackspace discloses that the Hosted Exchange outage involves a security incident, and the company has no ETA for resolution. The company vows to offer status updates every 12 hours.
  • Saturday, December 3, 2:31 p.m. ET: The company continues to work with “outside experts” to determine the full scope and impact of the incident.
  • Sunday, December 4, 12:37 a.m. ET: The company has engaged “world-class external expertise” in an effort to “minimize negative impacts to customers,” and continues to recommend that Hosted Exchange customers migrate to Microsoft 365.
  • Sunday, December 4, 2;05 p.m. ET: Rackspace is “contacting every Hosted Exchange customer by phone” to assist customers through options, but the company did not say how much time that manual process will require.
  • Monday, December 5, 1:28 a.m. ET: The company continues to recommend migrating to Microsoft 365 as the “best solution” for Hosted Exchange outage customers. Thousands of migrations have now taken place since the Hosted Exchange outage, though the number of customers still dark remains undisclosed. Rackspace did not comment about the status of a potential Hosted Exchange restore.
  • Tuesday, December 6, 8:30 a.m. ET: Rackspace confirmed that the Hosted Exchange security incident was a ransomware attack. The company believes that this incident was isolated to its Hosted Exchange business. The attack “may result in a loss of revenue for the Hosted Exchange business, which generates approximately $30 million of annual revenue in the Apps & Cross Platform segment.  In addition, Rackspace Technology may have incremental costs associated with its response to the incident,” the company said.
  • Thursday, December 6: In multiple updates, Rackspace said it is working with Microsoft to speed Microsoft 365 migrations for affected Hosted Exchange customers, and there’s no timetable for the Hosted Exchange restore amid the ongoing cyberattack investigation.
  • Stay tuned for more updates.

Rackspace Business Evolution

Rackspace was an early leader in the hosting market. But an ill-fated shift to OpenStack failed to compete against public cloud providers such as Amazon Web Services, Microsoft Azure and Google Cloud. In a major business pivot, Rackspace has spent recent years offering multi-cloud managed services and security services for customers that run AWS, Azure and Google Cloud workloads.

Note: Blog originally posted December 4, 2022. Updated regularly thereafter.

 

Return Home

8 Comments

Comments

    Larry Clark:

    They may be saying that they will offer support, but…
    — They disabled their chat support service.
    — The telephone support works — but only in the context that you reach robotic call direction. You can wait for the next available for hours and hours. Or you can get in line for a call-back — I’ve been waiting for the call back for 12 hours and I suspect it will be another 12 – 24 hours of waiting.

    Their solution was that customers convert to Microsoft 365 accounts then manually convert every user from hosted exchange to 365. This could be thousands of individual users one at at time.

    Their normal hosted email is not allowing addition of new mailboxes. I tried that and the “spinner” was going around for hours before I finally gave up.

    Their stock has plunged YTD, and Rackspace insiders are buying up stock at firesale prices.

    Joanne:

    I am ITIL certified. I work in tech. My small business is affected by this outage. I can not use any of the suggestions Rackspace is pushing out to their users because, unknown to me, they are reselling Microsoft 365. The problem is all of the suggestions they are pushing their customer relies on your logging into your Microsoft account. Microsoft recognized my/Rackspace hosted e-mail but they are not recognizing my Microsoft password which only Rackspace has but they are not answering the phone or their chat. My small business is hard down. I am currently trying to use an old g mail account that none of my customers or vendors have. Rackspace if you are reading this all your end users need that Microsoft password that only you have for our accounts.

    Joe Panettieri:

    Larry, Joanne: Thank you for your notes. Sorry to hear you have been affected by the outage. Please let me know if/when Rackspace resolves your issues, or if you find/implement third-party workarounds. We will continue to cover the story, along with Rackspace’s overall business evolution efforts — which were facing multiple challenges even before the outage.
    -jp

    Allen Hurst:

    Joe-Cybersecurity teams would benefit from a pre-configured failover digital communications and file-sharing platform ready in case of an emergency such as what has happened with Rackspace. If Rackspace users are seeking alternatives to Microsoft365 for their SOC and security teams, we would be glad to help implement a highly secure Out of Band Communications application from our partner HighSide.io.

    Allen

    andy:

    i just had my I.T. guy start migration to Go Daddy, they were very helpful and responsive , we will be up and running today.
    only issue is will i get the emails i have not received in 4 days?
    Rackspace has been horrible , and i wonder if they will recover from this.
    i did get Rackspace on the phone this morning , only waited 20 minutes, they were of no help, they just said go to status page for updates. as well they said migrate thru Microsoft platform. we tried that yesterday , and it took 5 hours on hold, we got someone in a foreign country finally, and they put us on hold, and we got disconnected .

    Larry Clark:

    This statement is part of the Rackspace update this morning:

    “We have restored email services to thousands of customers on Microsoft 365. We continue to make progress on restoring email service to every affected customer.”

    Rubbish. Without having to bother reading between the lines, you can tell that they seem to have not done little or nothing to solve the root cause of the problem.

    What Rackspace has done is fob off Microsoft 365 accounts along with a “Kiddy Script” that users are forced to follow to get their accounts back up and running. And who is this hurting the most? Small and mid-size business. (Consider a real estate office that loses their email over a weekend.) Meanwhile, who compensates the actual cost of companies scrambling with their own IT staff over the weekend, or the extra billings by their normal service providers, or the cost of having to bring in a new vendor to un-dork this situation?

    I don’t think that M365 should be considered the end-state solution. It is the space-saver spare in the trunk of your car that will get you down the road. But you really need to buy a new set of tires. Migration to a competent company is probably the best solution — the good providers will be taking notes about this charade.

    (If Rackspace converts those “free” M365 into permanent sales, will they be able to cut even more staff? I would guess that’s one of Microsoft’s selling points.)

    Caucus:

    Hello

    I agree what has been said. What is the alternative to Rackspace? We have over 2million emails in the archive. How do I get those to a new provider. I was lucky out of 30 users I only had two on Exchange. I have 28 mailboxes that are working.
    I can get email from the Archive four hours later and forward it to another mailbox on our accounts.
    So what choice do I have to replace this. I need Exchange for some, IMAP for most and an Archive for all. The archive needs to work for all sends of each mailbox and all receives for each mailbox. I have over eight years of Archives at Rackspace. Who and how is going to take this on?

    Any and all suggestions greatly appreciated

    Joe Panettieri:

    Hi Folks: Just a quick update. In addition to our coverage above, you can track Rackspace status updates here. Thank you for reading MSSP Alert.
    -jp

    Joe Panettieri, co-founder
    Executive VP & Editorial Director
    MSSP Alert

Leave a Reply

Your email address will not be published.