You might be thinking those are just hypothetical scenarios that never really happened to anyone. So today I’m going to tell you an AD disaster story that happened to a real Quest customer.
One day, the customer, who was running Windows Server 2008, found all of its DCs in an endless reboot cycle. It turned out that a privileged user had gone into a subnet and accidentally changed an IPv6 setting to an invalid IP address. When the Knowledge Consistency Checker (KCC) replication setup process encountered the invalid setting on that DC, it crashed. That caused the DC to reboot — but not before the invalid setting had been replicated to all the other DCs across the entire environment, causing them all to start rebooting repeatedly.
Can you imagine trying to recover from an incident like this? With native tools, getting back to normal could take days, if not weeks — you’d have to reload every DC from scratch, re-promote each one and clean up the metadata, and let them replicate as if they were brand-new DCs coming into the domain. The costs of that much downtime and disruption would be enormous, and as the IT pro responsible, you’d probably need to start polishing your resume. Fortunately, as I mentioned, this company was a Quest customer; with Recovery Manager for Active Directory, their IT team had the domain back up and running in less than an hour.
There are two important takeaways here:
- Accidents can be just as damaging as attacks — The privileged user in this incident had no ill intent, and he wasn’t over-privileged or under-trained. He was just trying to do his job, and he made a mistake. You can put various safeguards in place to minimize the most costly errors — such as requiring two sets of eyes for critical changes — but you can’t eliminate the risk that even well-intentioned and skilled privileged users will sometimes make mistakes.
- Every environment has security vulnerabilities — The odds that someone will change this particular IPv6 setting on one of your DCs is vanishingly small. But I’d wager good money that there are other vulnerabilities in your systems right now that a privileged user could deliberately exploit or accidentally stumble upon, with similarly disastrous consequences. Proper patch management will certainly help you improve Active Directory security — but vendors can only patch the problems they know about, and attackers are actively looking for new ones all the time.
Don’t let this happen to you! Read our ebook: “Three ways a privileged user can hose your Active Directory” to learn about eight AD security best practices, including privileged user management, that will help you reduce the risk that privileged accounts will be misused, either deliberately or accidentally, and help ensure that you can recover quickly if those preventative measures fail.