Agency keeps vital healthcare services available with reliable Active Directory recovery

Benefits

• Delivers peace of mind that critical AD services can be quickly restored in a disaster scenario
• Enhances business productivity by enabling the IT team to promptly revert improper changes or deletions to specific AD objects
• Saves the IT team three weeks of work annually by streamlining recovery tasks

The Story

Minimizing downtime is critical today — especially for healthcare organizations.

Cyberattacks, corruption and even innocent mistakes in Active Directory (AD) can bring your business to a standstill in a heartbeat. In fact, even a single improper modification or deletion of an AD object can lead to business disruptions or a security breach.

These events can be enormously costly for any organization, but in the healthcare sector, they can literally mean life or death. That’s why NHS National Services Scotland (NSS), a public healthcare agency, relies on Quest Recovery Manager— to ensure quick and easy recovery of specific AD objects that are accidentally modified or deleted, as well as fast, reliable recovery of an entire forest or domain in case of a disaster

Vital healthcare processes rely on Active Directory.

NHS National Services Scotland provides support services and expert advice to support the efficient and effective operation of NHS Scotland, the country’s public healthcare system. NSS covers both clinical areas, such as the safe supply of blood and tissue, and non-clinical areas like essential digital platforms and cybersecurity. It supports the various NHS Boards, including Public Health Scotland, which has been at the forefront of COVID-19 response.

Active Directory is the backbone of this vital IT ecosystem, so any AD downtime can cost not just money but lives. “Without Active Directory up and running, NSS users would be unable to access IT resources or provide critical services to the rest of the NHS,” explains David McNeill, principal engineer for NHS National Services Scotland. “Moreover, every board’s local Active Directory synchronizes to Azure AD — in fact, we have one of the biggest tenancies in the UK, with a couple hundred thousand users. So if Active Directory went down, our users would lose access not just to on-premises resources but to cloud systems like Microsoft Teams, OneDrive, email and more.”

The impact of such AD downtime would be “massive,” McNeill says. “While much of the clinical operations could continue using cached credentials, we would be unable to do much else. For example, we would not be able to provide shared services like HR and finance, supply COVID-19 statistics to the Scottish government, or enable blood transfusions for patients. In short, if our Active Directory went down, NSS would simply grind to a halt.”

When the management team approached McNeill for a solution, he was ready. “I’d been asking about investing in Quest Recovery Manager for years,” he says. “When I explained that it could have had the other NHS board back up and running in hours rather than days, the purchase got signed off that day.”

Quest Recovery Manager delivers reliable AD backup and fast recovery at both the object and directory level across the entire forest. With its intuitive graphical interface, you can pinpoint and revert unwanted changes to your AD environment in a few clicks. It simplifies and speeds recovery of an entire domain or forest by enabling IT pros to restore DCs simultaneously from a central console and automating manual steps. As a result, organizations can slash recovery time from forest-level AD corruption by up to 95 percent, dramatically reducing downtime, loss of productivity and reputation damage. In short, as the management team at NSS recognized, Recovery Manager is an insurance policy for your AD — one that you can’t afford not to have.

Recovery Manager delivers peace of mind.

Fortunately, NHS National Services Scotland has not suffered a catastrophic incident, so they have not had to use the solution’s full forest recovery capabilities in their production environment. However, knowing that the IT team is fully equipped to deliver such a recovery quickly is priceless — especially for a healthcare organization at the heart of the national response to the pandemic.

“At conferences, I often hear questions about how the financial savings that a solution might deliver. But in the NHS, we don’t think like that,” explains McNeill. “Obviously, we do think about money and whether a solution is a responsible investment for taxpayers and our organization. But when our IT systems are down, it’s not about losing money. It’s about not being able to provide patient care. It’s about not being able to issue blood. It’s about not being able to get crucial statistics to the Scottish government for pandemic decision-making.”

Granular recovery of AD objects takes just a few clicks — saving the IT team weeks of work every year.

While Recovery Manager’s disaster recovery capabilities have primarily served as an insurance policy, its granular recovery functionality is regularly put to use. “Recovery Manager has been an absolute godsend on multiple occasions,” McNeill notes. “For instance, the deletion of several DNS records caused us serious issues, and we were able to use the Quest tool to restore them quickly. Similarly, should an automated HR process delete un-actioned users, we are able to promptly recover them to a selected point in time.”

Moreover, Recovery Manager can restore objects and properties that other tools can’t. In particular, Group Policy is a critical component of Active Directory that controls everything from users’ desktop settings to valid authentication protocols, so being able to promptly restore Group Policy to a known good state is vital for both security and business productivity. “If we make changes to Group Policy and then something goes wrong, we can simply roll back the policy instead of reverting the changes one by one. We could never do that with our previous tool.”

The easy granular restore capabilities in Recovery Manager saves the IT team a great deal of time. “Before we had Recovery Manager, when something was improperly deleted, we would have had to hunt for the right backup and manually restore the right AD objects,” McNeill says. “Now we can locate and restore an AD object in a few clicks, which saves us three weeks of work each year.”

Those time savings are particularly valuable today, with IT talent in short supply and IT teams taking on a wide range of new responsibilities to facilitate remote work. “IT teams everywhere are being pushed to the limit because of the pandemic,” says McNeill. “So having a solution that enables us to quickly restore users, Group Policy, DNS records and more makes all the difference. And it doesn’t just save my team from having to do a lot more work; it also gets the business back up and running quicker, multiplying the value of the tool.”

An incident elsewhere vividly demonstrates the need for fast and reliable AD backup and recovery.

NSS had an Active Directory backup and recovery solution in place — but the IT team recognized that it was woefully insufficient for such a critical component of the IT infrastructure. “We were using another tool to back up our directory,” recalls McNeill. “However, AD recovery with that tool was neither quick nor easy. Moreover, vital components simply were not covered, including Group Policy and DNS. For example, it was a nightmare trying to get it to back up certain DNS records. We knew it just didn’t cut it for full AD disaster recovery scenarios.”

These concerns by the IT team escalated into huge red flags to the management team when another NHS health board suffered an extended outage — corruption in Active Directory quickly replicated across the organization’s domain controllers, rendering 10,000+ team members unable to access the information they needed to carry out key front-line functions.

Unfortunately, even though the IT team had implemented Active Directory in line with best practices and had significant strategic, technical and operational IT experience, they required assistance from Microsoft Professional Support in order to perform the recovery — and restoring normal operations took two agonizingly long days.

Quest delivers best-in-class hybrid Active Directory recovery.

One of the key recommendations in the incident analysis was for the board to implement a third-party AD backup solution designed to facilitate faster recovery in the event of a catastrophic failure. The management team at NHS National Services Scotland decided not to wait for a similar incident to strike their IT ecosystem before acting on that advice. “NSS provides services nationally, so we can’t lose our AD domain for multiple hours, let alone multiple days. That’s just not acceptable,” explains McNeill.

And even though the IT team has not had to perform a disaster recovery, knowing it is available has opened the doors to important initiatives. “Knowing Recovery Manager can quickly restore our AD has enabled us to make critical changes to the IT ecosystem with far less worry,” says McNeill. “I’ve upgraded the domain twice, and Recovery Manager has been my belt and braces. When I’m doing the risk assessment that’s required to get projects signed off by the business, the solution is central for my mitigation strategy. Whenever we’re making a domain upgrade change or even just a schema change, before we start, we make sure to back up the domain using Recovery Manager. Plus, we’ve also used it to prove that replication is working properly.”

Proof of reliable disaster recovery enables compliance with internal policies.

In addition, Recovery Manager enables the IT team to demonstrate the effectiveness of its disaster recovery strategy. “We’ve got a national security operations center (SOC) now, and they require us to prove that we have effective disaster recovery procedures. Before we had the Quest solution, I could only have shown that I could get some of the domain back, such as users and possibly some of DNS. Now, I can show them that we can efficiently get Active Directory back in case of corruption or catastrophe.”

When disaster strikes, speed is critical.

McNeill is extremely satisfied with the Quest solution. “What I like most about Recovery Manager is its speed — we can restore Active Directory objects or an entire forest quickly,” he says. “Plus, it is so easy to use; everyone on the team can use it without any problem.”

In fact, NSS is so happy with the tool that they are in discussions to deploy it to the national NHS program. They are also interested in upgrading to Recovery Manager for Active Directory Disaster Recovery Edition, as well as pairing the on-premises solution with On Demand Recovery, Quest’s Azure-hosted SaaS solution for Azure AD backup and recovery.

About Quest

Quest creates software solutions that make the benefits of new technology real in an increasingly complex IT landscape. From database and systems management, to Active Directory and Office 365 management, and cyber security resilience, Quest helps customers solve their next IT challenge now. Quest Software. Where next meets now.