Check out our recent webcast entitled, Preparing for the Disgruntled Privileged User — Three Ways They Can Hose Your Environment in Minutes with Brian Hymer, Strategic Systems Consultant and windows security expert, Randy Franklin Smith. Brian and Randy dish on how AD can be maliciously taken down — along with the rest of your network. In addition to discussing how preventive controls can reduce the risk from disgruntled privileged users in the first place.
After that, Brian demoes our AD security tools that provide a crucial audit trail, as well as a recovery options, that can help you quickly rebuild your entire Active Directory forest in the event of an attack.
Download the archived webcast or read the webcast transcription below – or both!
+++++++++++++++++++++++++++++++++++++++++++++++++++
Randy F. Smith: Good day, everybody. Randy Franklin Smith here and today we're talking about unfortunately just how easy it is for somebody with the right authority to completely hose your environment and more optimistically or on a positive side what we can do to prevent, and prepare, and recover from that. Today's real training for free is made possible by our long-time and best sponsor Quest. I have my long-time colleague here with me Brian Hymer. Brian, thanks for helping out today.
Brian Hymer: Randy, always a pleasure. Really looking forward to the topics today.
Randy F. Smith: We're going to talk about the fact that hosing AD is not hard, talking about preventing it, being prepared, and finally automation. Now, I want to be clear, my purpose with sharing this information today is to help organizations be forewarned about how easy it is for someone who has or acquires privileged access to cause damage. None of these methods are that complex but after reflecting upon this, I'm not going to provide a step-by-step details on how to perform this or how to overcome some of the attempts by Windows to prevent you from shooting yourself in the foot because I've tested all of these, Bryan and I have, but there's some things that Windows does to try to help prevent you from doing this that you have to overcome as well. With a couple of them, there's a bit of a staging required to make it be really effective in terms of destruction. I don't see the point in sharing the step-by-step details, unless some of you are really facing the need to do this as a proof of concept to convince management that this is for real.
The approach that Brian and I have thought is we would be happy to work with you one-on-one after verifying that we are working with an organization and not an individual and we do NDA and stuff like that. That being said, we don't claim to be great hackers that have discovered little-known methods, but we're just trying to be responsible with all this and avoid ... What's the term? You know, the litigious society that we live in. That being said, let's talk about this. First of all, I just thought the term hosing something, I wondered what the dictionary actually came up with and they do pretty good. I like both definitions of both the Urban Dictionary and good old Webster. It really fits what we're talking about. Here's the first way, number one, denying log-on rights.
There's five ways that you can log on to Windows, there's five logon session types. There is correspondingly five allowed logon rights and five deny rights. Now, if you successfully assign these deny rights in the right way to the right people then you get pretty much what you see there. I did this yesterday and it was bad. I could not even log on locally at the console even with an admin account when I did this right. Now, again, being careful with the details and how deep I get into this, you would have to do this in a bit of a staged way to have full effect. If you are inpatient as a bad guy and you're just in a bad mood, then it's potentially likely that you would only be partially affected, which is really good news. If you think it through, then it's really bad. Users cannot log on to their workstation, users can't logon to member servers from their workstations in case they're able to actually get into their workstation, and admins can't get into the domain controllers even at the local keyboard and screen at the console. You would have to reboot into DSRM mode, directory services recovery mode on each one of these domain controllers to manually recover. Brian?
Brian Hymer: Yeah. Randy, the lines that you got here are great. I was thinking part of what we had done was also member servers and workstations could not authenticate to the domain. Really even your applications would be down from this type of activity.
Randy F. Smith: Yeah, everything. This is a pretty widespread outage and the only way I came up with recovering was to boot into DSRM on the domain controllers and then you'd have to go from there. Folks, Brian is going to show you an absolutely fantastic technology, once I'm done, that I guess, you're going to run a simulation and just show people just how easy and quick it is. It's crazy. This is a bad one and it can be extremely effective. Here's another one and that is DNS because Brian have you ever heard the term that husbands often use, "If mama ain't happy nobody's happy?"
Brian Hymer: That's true.
Randy F. Smith: If DNS is down everything is down. If you delete all of the stuff in DNS and then make it difficult to fix it, then people are still going to be able to log on to their domain controllers possibly, but where do you start inputting all of this back together? Once you delete, once DNS isn't working in whatever method you use, nobody's going to be able to find anything. Workstations won't be able to find domain controllers, they're going to resort to net BIOS name resolution, which may or may not work-
Brian Hymer: Depending on the size of your organization, yeah, definitely.
Randy F. Smith: Your networking and stuff like that. Go ahead.
Brian Hymer: Randy, what I found really interesting and this one is that aside from these changes happening, you only have to make them in one place and then they're going to replicate, because your domain controllers are used to replication, and the DNS cache is still there, so it's got plenty of time to replicate but once DNS has been broken, and that's replicated to all of your DCs then your DNS cache times out, so those hosts that your DNS already knows about it no longer looks. It wants to go back and look at DNS to make sure it's still the same. Now you're really hosed-
Randy F. Smith: Yeah.
Brian Hymer: Because it took enough time to replicate out, it's rather ironic, and then obviously that killed the DNS zones, but then you can't get anywhere because you can't look up anything in DNS.
Randy F. Smith: It's ugly. I actually went and did what I did on each of my DNS servers but with your tests Brian you let it replicate. I hadn't thought of how the cache might allow that to happen. Yeah, it's very ugly. Let's see here. If you were to combine this with some of the others then it just gets even worse. With this one, even if the admin is still logged on to their workstation, or here's the weird thing. Let's say you were logged on already to the console of the domain controller and so you're like, "Well, let me bring up active directory users' computers," guess what? 80 users using the computers even running on that domain controller will fail to connect because you've denied the access to this computer from the network right. When it tries to connect via LDAP to its same local self, it'll fail because that is a network logon right, so it's really nasty. Let's see here. Brian, you want to talk about this one, the bitmask thing?
Brian Hymer: Yeah. You know, Randy, a lot of these that we're talking about are malicious activity but we actually had a customer make this mistake. It was a problem that was in Windows 2008. He actually figured out what went wrong after his DCs all ended up going into an endless reboot. And he went back and talked to Microsoft and they have since patched it, so hopefully everybody's up to date on their patches especially if you're still in Windows 2008 or 2008 R2. But essentially what he did was he went into a subnet and modified an IPv6 setting and put an invalid IP address. And what happened was that the KCC when it comes across that to set up replication, it just crashed. And when the KCC crashes, the DC reboots but not before that data could replicate to another DC. So then you've got another DC that's rebooting and soon that replicates across your entire environment and it brought them down all of these DCs were in an endless reboot.
Randy F. Smith: Yeah. It really is. There is no chance of us exaggerating on anything here. Let's also make a couple of points clear. This isn't just about the disgruntled insider. It's about any external attacker who gains privileged access. This could just be some nihilistic jerk who has no other motives than that. But there's so many hostile entities, there's hacktivists like good old Mr. Robot that wants to erase consumer debt. It could be a hostile foreign state sponsored group, very common today. Brian, I promise you in the next ... It's not going to be that long. We're going to see competitors especially from other political spheres doing stuff like this.
Brian Hymer: I think we already are.
Randy F. Smith: Yeah. Yeah it be. Well I mean, certainly the stealing information, but the next thing is just do a denial of service against a competing business. Disaster for them is good for you. Or just it could be a wronged party, again, classic example Mr. Robot, such and such company, can't remember its name, Evil Company or whatever, Evil Corp, they hurt his dad. Any wronged party out there, and they've got knowledge and motivation, all they want to do is harm and destroy. It's not like stealing information. And the thing about it is harming and destroying is so much easier than stealing information or building something. This is for real stuff. And of course you don't have to do this with Active Directory. There's lots of other ways. I mean, think of if somebody did a well-timed coordinated attack, destructive on your switches. It's bad. Concentrating on Active Directory, what can we do? Forewarning, there's no secrets that we're going to share here. It's all about the fact that you get a tactical advantage if you are forewarned and if you take appropriate steps. No-brainer, right?
Limit privileged access, as a start, don't put anybody in these groups. It's not just about domain admits, it's also the administrators' group, the local administrators' group inside Active Directory itself. There's also that group, group policy creator owners, DNS admins. We've got to create a wall around domain controllers. By that I mean the domain controller's OU. There's also a group called Domain Controllers. There are group policy objects. Whatever GPOs that hit, your domain controllers. Man, you've got to be absolutely careful with those. What software is installed in those domain controllers, because if you install some agent, but other people who aren't domain admins have access to it, then they may very well in effect be domain admins.
Brian was bringing up the obviously point, we have to put somebody in those groups. What I'm really talking about is using a full-fledged privilege account management and privilege session management with human approvals and live supervision for levels of access that would impact your entire domain. We are not doing stuff like this on a daily basis, Brian. I mean we should not be needing to touch domain controllers in any way on a daily basis. If we sequester that level of an access off in privileged accounts that are managed by technology and then we require two people to be present, one person doing the work, one person supervising, even remotely, even just a peer, then at least we've brought this down from being possible by a lone wolf.
Brian Hymer: That's a good point, Randy. I totally agree.
Randy F. Smith: Yeah. And we've got to do the same thing with [inaudible 00:15:35] accounts. [inaudible 00:15:39] all their procedures are good but there should still be a collusion requirement, meaning that in order to make use of that, it has to be two people working together instead of just the lone wolf, because society is proving that the average person has less and less control over impulses. And who knows with the latest generation? Somebody gets angry and they're going to do some wacky stuff. So yeah, Gerald was saying dual control and segregation of duties. These are time-honored principles, but we've only paid lip service to them in the past. You wouldn't believe, maybe you would believe, how many conversations I've had with CEOs and at audit exit meetings, some of them getting red-faced saying, "Well, if there's a problem, I need my people to be there fix it. We trust our people." Great, as long as you don't have a Roger Duronio in your department or again, it's not just the disgruntled insider. Accounts get stolen, right? That's what all this stuff was about with Mimikatz and pass-the-hash and all that other stuff. Brian, that's another thing you brought up is part of limiting privileged access would also be taking steps against these credential artifact attacks. One of the major architectural ways to do that is with Red Forest.
Brian Hymer: Right. That's definitely a very strong way. If you guys aren't looking at ESEA, the Red Forest architecture, you should be.
Randy F. Smith: Okay. So next-
Brian Hymer: Randy, we're talking privileged access. It's really privilege access management is what you're talking about. That's a very important piece, because we're talking a lot about hackers and people with a bone to pick, or an ax to grind. Sometimes these mistakes happen just because the person is inexperienced or it's just a common human mistake. The whole IPv6 problem was not a malicious attack. It was an honest mistake and the technology can be difficult. And so it's important to have not just the ability to regulate privileged access, but the ability to test changes that may cause problems. That's really important too and I think everybody on the phone probably has a test lab that they've used to pre-stage upgrades or changes that they're making in production. But those test labs probably don't look anything like production. And that too can be a problem.
Randy F. Smith: We're taking a hybrid approach of having a dual use domain. It's production and test in one and our management is claiming major savings by that, Brian.
Brian Hymer: Well, good luck with that, Randy.
Randy F. Smith: [inaudible 00:19:04] how that works out for us.
Brian Hymer: I'll talk to you in six months.
Randy F. Smith: Dual use. It's not so sophisticated. Okay but for real though what you say is on point. I've done it myself. The way I discovered the logon rights thing is I did a portion of that to myself of a deny ... accessless computer from the network [inaudible 00:19:31] or something like that. I forgot what it was. Anyway, I did some combination of those rights, and I was able to overcome it, because I didn't do a complete job of it thankfully, but the only way I could do this, I had to use Notepad to manually edit the group policy object and then also change the version number, and then I had to [inaudible 00:19:51] in and change the version number on the GPO there too, with [inaudible 00:19:59] or something like that. And then I was able to get access back without having to do a full restore. Anyway it's to your point that just mistakes can cause this kind of thing too. All right. Let's talk about audit. Are you hearing an echo, Brian?
Brian Hymer: No. Not on my side.
Randy F. Smith: Okay, good. Brian, talk about this. You're the guy on this because of the years, and years that you've worked with change auditor.
Brian Hymer: Yeah. Good segue, Randy. Auditing is really important. It's not that you're going to know how to solve every problem that you come across, but if you don't have an audit trail, then it's possible that lightning could strike you twice. And that's what you try to avoid and that's why having a forensic record is so important. And it's more than just change auditor. Change auditor does a great job of wheeling down to the very important details, but because this landscape is changing all the time, you can't ignore native events, application system security log, directory services logs. You need to be able to see all that information and be able to research it in a quick and efficient way. That's one of the tools that we have. For example, in part of the work that we were doing, Randy, I knew that there were certain commands that were run, and those commands could have been run on workstations. So with our solutions you can collect the security log, the Windows system log, the Windows application log, whatever other logs you want too from your workstations, and search through that, so I was able to go very quickly and search for some of the commands that we did to do this, and find them very quickly and easily.
Randy F. Smith: The thing that I want to point out is it's not about assigning blame, because at that particular moment, while everything is down and the phones are lighting up and people are screaming and shouting, which there are situations where that actually occurs in computer rooms and [inaudible 00:22:18] and so on or [inaudible 00:22:19] I should say but it's about assessing what happened so that you can fix it, because with a lot of these, you're just going to start getting phone calls, maybe emails maybe not. And you know how it is, when a user calls up, it's like, "I can't log in." And that's all you hear, or "The system isn't working." So you just get to know that something is wrong in a very big way, and you're not [inaudible 00:22:57] know even what's wrong, much less how it occurred. And if you can go back and say, "Okay our first [inaudible 00:23:05] port was at 7:30 am. Let's go see what's been changed in our environment shortly before that. Then man, it's just going to put you light years ahead in getting back functionality, Brian.
Brian Hymer: Yeah. I totally agree. If you don't know what happened, you don't really know how to fix it. So you really have to have that audit record to know.
Randy F. Smith: The other thing that I want is to know immediately when key objects, like anything to do with the root of a domain or domain controllers group policy objects at that level, any of the major groups that we're talking about is changed, so that ... and I want that alert even if it's legitimate, Brian, because that is a confirmation that the system is working, you follow me?
Brian Hymer: Yeah.
Randy F. Smith: It's not like you're going to be getting these alerts all the time, because you shouldn't be changing the stuff all the time. It's a great test when you do have to maybe make a change to a GPO at the domain controller's level, or the root level of the domain, that you get an alert, because that shows ... it gives you positive confirmation, that our monitoring process is working. If it ends up being an unapproved change, then obviously you've got an immediate notification, and maybe you've got time to get in there and reverse it, Brian.
Brian Hymer: Yes. That's very true, Randy. I don't think I could add any more to that. It's pretty good.
Randy F. Smith: Fair enough. Well, let's talk about documentation, having up-to-date offline records of your AD structure. You force your domain to trust relationships. What are your DNS servers? What are your subnets and the replication links between them for AD? Where are all your domain controllers, both IP and physical location? Which domain are they domain controller of? What are the flexible single master operations on them? Which ones are global catalogs? [inaudible 00:25:17] all of them? All of that stuff, I mean, having it in paper or up in Dropbox or something like that, right, Brian?
Brian Hymer: Yeah, that's a great thing to do, and that's part of what our Forest Recovery product does. It actually saves key information like that and you can write reports on your Forest Recovery project, how it's set up and even after you run recovery, how that recovery went. [crosstalk 00:25:41] good.
Randy F. Smith: Well yeah. And that's something I want to put you is really, ideally this would be automated and would go get saved like up in Dropbox where you can get at it from anywhere any time.
Brian Hymer: Right. But hopefully the bad guys can't get at it from anywhere and any time.
Randy F. Smith: Right. Likewise. Yeah. Okay. Backup, backup, backup. Back up active directory, don't just rely on recycle bin and protect those backups. I'm not getting more into detail on that because we've done an awesome webinar before on when the active directory recycle bin isn't enough and what are your different ... domain recovery and forest recovery, those are really two different levels of disasters, two different types of backups that you have to prepare for and do, right, Brian?
Brian Hymer: Yeah, absolutely. So, Randy, the DNS example is really good. We went through and removed records that we could have just gone and hose those records, put invalid IP addresses in and then blocked access. Recycle bin's not going to help you. It's not going to restore those attributes.
Randy F. Smith: Yeah. Good point. That's right.
Brian Hymer: So yeah, really important. Get your backups, make sure you've got them.
Randy F. Smith: Recycle bin is a convenience and nothing more. Okay. So backups are great, but I mean you folks know this, right. But the tough thing is, with AD it's not easy to test your backups and your recovery procedures. But you have to regard backups as faulty and it's a proven viable. I mean, there's just no other possibility. And so we have to periodically rebuild our domain in some type of test environment, or I should say our forest if we want it to work when we really need it and if we want to do it quickly when we really need it.
Brian Hymer: It's really true. One of the things we're baking into our product is to check the viability of a backup after the backup completes. And so we actually will mount the backup and go and read an object in the [inaudible 00:27:58]. And if we can't read it, we send an email that says this backup is invalid. But if we read it, then we assume that it's viable. A really good point, Randy.
Randy F. Smith: Bottom line for this is that you're limiting authority in documenting, we got to do all that, but also manually testing our backups in recovery is expensive. It is for anything. It is for even SQL server, but it is much more expensive to do this for active directory in terms of time, resources and also expertise, because this is like double bypass surgery. The thing is, even if you do all this, you're still vulnerable. It's still very time consuming to recover, and you're going to have major losses. Here's the thing. You guys can automate this. You want to show folks how, Brian?
Brian Hymer: Yeah I'd love to show. That's a great segue, Randy. Let me switch over here. I'll get my screen going. Let's see how can we start here. Let me just talk about our solutions portfolio, because we have a number of items that really fit what Randy was just talking about. IT security search is obviously our way of searching data. We have a number of auditing tools within Quest: InTrust, Change Auditor, Enterprise Reporter. Those products are all capable of creating event data or state data on how your environment looks, and then with IT Security search, you get a web page that lets you see all of that information. Active Roles as well. I don't know if there is any Active Roles customers on the line. We've got a lot of really happy customers with that tool. You talked about not having people in the Domain Admins group, Active Roles is the way to do that because it allows you to delegate permissions to Active Directory in a way that's manageable. It's like ... I always like to say it's like Active Directory users are computers on steroids, that's a great tool, one of our best selling tools over the years, and it is really something that a lot of companies couldn't do without. Let's see. Let's drill into some components here, Randy.
We'll talk about IT security search at a little bit higher level, and I've got a new toy here, and I'm gonna try to play with Randy, so let's see how well this works. I've got a little spotlight here that I can use. IT security search gives you a view like a single pane of glass, like having your own little spotlight to look into the data that comes from these five sources and that's Enterprise Reporter, which lets you see current permissions, current group memberships, things like that from Active Directory in your windows servers, Change Auditor, which does that real time auditing of critical changes in your environment and file system activity and others. So there is a great deal that Change Auditor can do for you. InTrust, it can collect native and third-party logs and it can reach beyond the Windows environment.
You can collect from UNIX systems and have syslog pulled in from InTrust. We store that data into a very highly compressed repository and that way, you have that forensic record for years. My customers have 7-10 years worth of data there. Recovery Manager. This is our active directory solution that allows you to restore objects in AD, or automate an entire forest recovery and I'm gonna show that a bit more today. And then of course I talked about a little bit Active Roles, which allows you to manage active directory, not just user lifecycle management, but really a delegation of permissions and authority within the active directory world, and your member servers. IT security search brings all of that together in a Google interface. That's really what it looks, and I think we talked about that, Randy [inaudible 00:32:09] webinar too. Recovery Manager for active directory. It's a great product. We've had it for years. It came up pretty close to when Active Directory itself came out.
And in the beginning what we were doing was managing backups and giving you a way to do a granular level recovery without having to bring a domain controller down. The great thing too was that you could you run comparison reports and see exactly what was happening in your environment, what changes that happened and decide just to change what had changed, right or restore just what had changed. We've expanded this. Most recently, we've added this hybrid ability. Now, a lot of companies are working towards Office 365, and that means that they have an Azure environment, so they have accounts probably hybrid/synced up to Azure. They might even have some accounts that are Azure only, and those Azure accounts have attributes that are specific to Azure. We've built a solution called On Demand Recovery for Azure AD and it integrates right in with Recovery Manager for Active Directory and it allows you to restore those attributes or kick off a restore of an object whether it's a hybridly synced object or an Azure only object just from one interface. So you don't have to go run through two steps.
You can actually get it all done in one swoop. So that's a really nice feature. And then the other piece that we're going to talk about in more depth here is our forest recovery. The Forest Edition version of this product allows you to run health checks against your forest. It also allows you to create a virtual web that we automate the process of taking production DCs and moving them on to a VM host, like VMware or hyper-V, and then configuring those DCs, so that they're number one, further isolated from production, and number two, talking to each other and working as a functional lab. Now, we built this so that our customers could test forest recovery, which is really what the forest edition is all about, but what we found was some of our customers want this tool just for the virtual lab feature, because it is so hard to build a copy of Active Directory for testing that matches your production environment. This is a really great addition. I mean to our tools. Randy, I'm going to talk about these [inaudible 00:34:42], feel free to chime in.
Brian Hymer: Deny Logon Rights, so this is our mascot here. This is Hank the Hacker. I don't know if any of you were down to Ignite this year. If you [inaudible 00:34:55]. I'm wearing my Quest shirt from Ignite right now, but I'm not going to turn my webcam. When you got Deny Access Rights. This is my environment, then I tried to log back in, I obviously got a deny. So this is wait you might see if this started in your environment. Obviously, I logged in, so I tried to log in and I couldn't, so I had to actually switch in this environment, this is logging into my recovery manager server. I had to switch over and log in to the local host. One thing you might want to think about is, especially around your Recovery Solutions is do you know the local [inaudible 00:35:39] account password for the administrator. It might be important. Once I did that, I was able to log in. I did try to bring up our console, and the first thing I found is that my account did not have administrator rights or SA rights to the database. We use a persistence database in our product that keeps track of where the forest recovery is.
Just in case the forest recovery console gets shut down for some reason in the middle of recovery, but it's not required, so I just went ahead and disabled that, and move forward with my forest recovery. This is my verify settings. All of my DCs came back because we're using local system, so even though deny rates were set for all the accounts because I'm running as local system you can't deny a local system, so our forest recovery agents were able to communicate back to the console with no problem. And then I ran the forest recovery process. You can see here, it took an hour and 54 minutes and then everything came back just fine. I think I might want to skip through a couple slides here. Yeah. Here's what I mean. Everything looked fine, but a half an hour later I still had some problems. This is our forest health check. It told me I couldn't connect to these four, five DSs, so I simply RDPed onto one of them, ran a dcdiag, and then based on that, I forced replication between its replication partners, and by the end, things cleaned up, and my forest health check came back nice and clean.
Randy F. Smith: Brian, remind us which scenario are you in here.
Brian Hymer: This is the deny rights. This was a deny access rights scenario.
Randy F. Smith: Okay, so this answers Ross's question that says how will these tools work when deny access hosing it's been enacted.
Brian Hymer: Yeah, yeah. Ross, so the biggest thing was that I needed to know my local SAM account, administrator's account on my recovery server. But from there I didn't need any domain access at all. I was able to recover the forest. The DNS scenario. Let's talk about that real quick. We blocked DNS. We made sure that all of those DCs, the DNS was hosed. I should have gotten a screenshot of that where I went and opened up the DNS client on another DC after making these changes, and it failed to load the DNS nodes for my forest. And that's the MCDS and the standard domain name DNS zones. When I ran a verify settings this time in our console, it came back and said it couldn't connect to this one DC. But the rest of the DCs worked. That's kind of surprising. I know that our product actually stores the DNS entries for these domain controllers in the forest recovery project, just in case this type of scenario happens.
And so, I got a little worried. And so I took the time and I went out to the host file in this DC and I added a host record or actually on my recovery server I had a host record for the server and I went and reran my verify settings and this guy worked. But these last three failed. And so that's when I realized that DNS cache was actually failing out. It wasn't necessarily my product that was having problems, it was just standard DNS cache. I said, "Let's just go for it." I started Forest Recovery anyway. And the forest recovery ran without a problem. So maybe that's something that I can bring back toward developers. Maybe if a connectivity issue happens, that it could go and make sure that the cached data that we have in our forest recovery project actually reaches us domain controllers. I love doing testing like this, Randy. I'm glad that you wanted to do this webinar and you came up with these scenarios. Okay. Let's get in, I'll show you, before we go to resources, let me get in and show you a little bit of the product itself. [inaudible 00:39:58] Can you see my screen okay, my big "Quest. Join the innovation"?
Randy F. Smith: Yes we sure do.
Brian Hymer: Okay.
Randy F. Smith: I've got a couple of other good questions here, Brian.
Brian Hymer: All right.
Randy F. Smith: From Simon, he says, "What authentication method is used between the domain controller's agent and the recovery agent [inaudible 00:40:18], i.e. would it not be possible that the deny rights might [inaudible 00:40:23] the recovery via the agents and there's [inaudible 00:40:30].
Brian Hymer: That's a great question. We're using an SSL communication. And I wish I could tell you all of the details, but Active Directory authentication was not an issue for this communication to work. One of the things that we've always looked for is that we have for example DSRM rights because the first thing we usually do is reboot the domain controllers in the DSRM to restore the [inaudible 00:40:58]. On our next release, we may not even need that. Even if you don't know your DSRM passwords across all are the DCs in your forest, we still are able to do that recovery. We're stepping outside of Active Directory and Kerberos needing to be the authentication method, which makes sense because if we're dealing with restoring a corrupted active directory, you're not going to have that.
Randy F. Smith: Yeah, and with some of these, you could get deeper [inaudible 00:41:30] on a maybe on a one-on-one basis. We're just trying to be careful not to make this super easy for somebody to go out and do, especially if how Recovery Manager to bring it back, so to explain how Recovery Manager works in a couple of these would require to get really detailed into the exploits, if you want to call it that.
Brian Hymer: Right. Yeah. And again we'd be happy to show those to you if we can do it in a one-on-one with an NDA. Here's the problem with doing a demo alive is sometimes things don't work directly.
Randy F. Smith: [crosstalk 00:42:13]
Brian Hymer: Randy, one thing that a hacker might try to do is actually clear events. And so, I wanted to show this just as an example. In IT security search all I did, you noticed, the interface was kind of like Google, all I did was search for any activity that had the word cleared in it as the what criteria. I could explain that in more detail, but I think we've done that in previous webinars. And right away, I can see here. Record's coming back showing me that various logs had been cleared in the environment. Again, this is just a lab, so no need to get worried, but you can see right here that I was able to find that data very, very quickly. Now, the beauty with InTrust is it doesn't matter if you clear the log, because we collect this data in real time, and we actually write the log data to a cache at the same time as it gets written to the Windows event log. And so, even if you went and cleared the log, we've already got the data. That data is also sent in. Now, we used to do that in a gathering job years ago, but now we send that data in real time.
One minute after that happens, it's already on the InTrust server, which hopefully is protected from your attackers. I just wanted to show that as an example. Again, I can double-click on one of these, and I can see all the details for that event. I've got this nice event timeline that shows. My screen is a little bit limited. I'll just come down here, but I can see here that that log was cleared, the system event log was cleared. I could look at when somebody actually did that. Here's Alex made this very same change a while ago. And so, I've got a clear audit trail that he was probably trying to cover his tracks, and then I could go and take a look at what else Alex had been up to. Let's get into the recovery side, a lot more that I could show here. Let me just show one more search executed. And I'll look not just any executes, but I'll look at yesterday. Right. Now, get these two guys out of the way.
Now, I can see exchange commands that were executed. If you look over here on the left, these are called facets and facets are essentially filters and how we categorize the data that we pulled back, so I can see here facets are building out in the who field. It shows me just what accounts did these changes. I can look down here and see that some of these were from service accounts. Oops. Didn't really want to click. I'll step back out of there. Some of these were from an actual users, some of these were from computer accounts. I can see what machine they were done from. I can see what servers, if I look at where. I can see what servers they happened on. Let's just stop that search. Get these out of the way again. I can see what server they were done on here in the where area, and so, I could focus down and say, "Well, I know I'm looking at some things that happened on my member server, this Mem one server, so I can click on there and now, I've got that as my where criteria, but I can see all these individual users that had done changes on that box, and so maybe I want to just focus down on a particular user.
See how quick it is for me to seep through this data? [inaudible 00:45:49] But now I can see the programs that were executed by this particular user yesterday. And I can click on those and see just what program it was. Again, I'm just collecting the security log, but I can see just what executable they ran. Right down here. This one's just the office background task handler. There could be other ones that are a little bit more worrisome than that. But again, if you don't have this audit trail, you can't do this type of work, so it's very important and when you find something, it's pretty easy for me, like one of the things we were doing as a DS query. I could simply add that here and search. Now, I can see just those programs when I did that, if I click on one of these, you can see here, it actually highlights DS query. And I can see the exact command that I ran showing right there in the log. Great for an audit trail. Let's change gears and let's talk about recovery. This is our forest recovery console, and I want you to know this is just a simulation.
I'm not actually running a forest recovery here, these UCs don't exist. The reason for that is watching a forest recovery run is kind of akin to watching paint dry or corn grow. There's not a lot to see. Although it is nice to see the progress in the console. It takes a lot of time to actually do one. So this simulation will run in about five minutes. I'm just going to start by clicking my verify settings button here. And it's just going to reach out to each one of these DCs and run through a number of tests just to make sure that the DCs are available and pre-recovery checks were done. If you're curious as to just what we're checking on each one of these, you can actually [inaudible 00:47:57] over the steps. We show you exactly what we're doing in these popup windows. This is what came back and failed for me in my testing last night. And so, I just decided to go ahead and kick off recovery anyway.
I'll go ahead and do that here I could start recovery. I'll confirm. And then here we just have one more that says, listen, we want to make sure that you realize that this may cause a forest wide failure. Hopefully you are doing it because you're already in a forest wide failure or that you're in a scenario that recovering the forest is faster than trying to manually fix what went wrong, which is another good point. [inaudible 00:48:39] quick round. So I'm just going to let this kick forward. Randy, are there other questions? Are people wondering? Are there things we could answer? [crosstalk 00:48:52]
Randy F. Smith: Yeah back from auditing, somebody asked, you have access to logging levels and audit policies in the domain. Can the requirements be protected, ie. [inaudible 00:49:02] attempts to change it. [crosstalk 00:49:08] Are you even relying on the ADR policy with Change Auditor?
Brian Hymer: With Change Auditor? No. But there are things that show up especially when you're starting to collect from member servers that audit policy would make a big difference on, and so, that's not things like local SAM account changes, but it could be things like user rights, and other changes that happen on member servers. So, yeah, having the ability to protect that is pretty important. Obviously, audit policy is set up the domain controller's policy or on the default domain policy. With Change Auditor you can actually protect those policies and keep them from being modified. So great question.
Randy F. Smith: John asked, "Does the product prevent leaving credential artifacts behind?"
Brian Hymer: Ask that one more time. I'm not sure if I followed the question.
Randy F. Smith: I'm not sure where that fits in here, but credential artifacts like cache credentials and stuff that pass-the-hash uses, like if you ever logged on to the computer in the past. We're talking about audit and recovery [crosstalk 00:50:28] goes up there and cleans up credentials.
Brian Hymer: That's kind of a good thing, like user profiles that may be there from an administrative account, you probably want to clear those out, that sort of thing.
Randy F. Smith: Oh, yeah, you want to do that. But I mean, I don't think it's within the problem domain of what you're trying to solve. That's all.
Brian Hymer: Right. I think with Enterprise Reporter, you might be able to get that user profile data. That's a great question. I'll follow up on that and find out and that way at least you can go out and scan your environment and look for user profiles for privileged accounts and know what you need to go clean up [crosstalk 00:51:04] point.
Randy F. Smith: Fair enough.
Brian Hymer: I'm watching a DC here that's set to reinstall active directory, and I'm just going to change to one that's actually restoring from backup. You can see, a number of steps have happened. We've prepared the DC for a restore. We've copied that backup onto the DC if it wasn't already there. We rebooted in the DF mode and then we isolated the DC while it was in DSR mode. We actually use IPsec and isolate that domain controller. So I can't talk to the network other than this box. And then we do a number of things to protect that box from being modified while it's being restored. We disable any custom filters for passwords, restore the box. It's got that isolation now so we reboot back into normal mode and walk through all these configuration steps. You know, get information about the DC, select preferred DNS server, [inaudible 00:52:04] red pool. These are all steps that are in Microsoft's planning for forest recovery document.
And we automate all of them. And then once we've done all of those changes after the DC has been restored, we come back, we disable that isolation that we placed and then we modify the global catalog, we get the global catalog rebuilt if we need to. And then we eventually restart this box, all is back in normal mode. We eventually had our backs ready for people to be able to log in, and then we just do some final cleanup. All of these steps are automated, that you can see some of these guys are waiting like this, he's going to do a reinstall of active directories. He's waiting for domain controllers to get restored from backup before he does a DC problem. He is going to wait until that's done. Now these guys were done restoring from back ups. Now these other ones will step in.
All this process is automated. I was at a customer the start of the summer. They had a process put in place. They had for domain controllers for a test. It took them two to three guys working about six hours to do the recovery and manually. Then we run our product in and did the same recovery. We finished in, I think it was 28 minutes and 39 seconds. And we didn't have to do anything. We just sat here and watched it like you and I are. Much, much better to have an automated tool to get this done. And we're almost done here.
Randy F. Smith: I was wondering if that was specific to Azure or does that work with AWS?
Brian Hymer: That's a great question. We're only focusing on Azure with the on-demand piece. If you had domain controllers running in AWS as servers, as VMs you could certainly run forest recovery on them the same way with the stool. It's just another subnet. But we aren't doing anything account-wise and in AWS yet. Once forest recovery completes, we give you a number of utilities, a quick access. For example, let's say a FSMO role holder, like my infrastructure master here failed to restore, and it swapped it to Los Angeles. And I didn't want it in Los Angeles, I could simply click here, manage FSMO roles and move that FSMO rule to another DC in the domain. A number of these probably won't work right away that the domain has to settle down after being restored. So you probably want to wait half an hour, maybe an hour before trying these out. But it's nice to have them all there. That's what I have to show today, Randy. I've got some more slides.
Randy F. Smith: How well does recovery work, Brian, Simon's asking, over low bandwidth? Is there a way to manage or limit the bandwidth usage?
Brian Hymer: It's a great question. Part of our best practices with Recovery Manager, I mean this is going to segue me into another discussion too, Randy, so I'm glad you asked, or I'm glad he asked. Part of our best practices is to store the backups on the domain controllers themselves. And that way, you don't have to worry about the time it takes to copy those backups back onto the DC. I mean, you also don't have to worry about being able to authenticate against the file sharing that [inaudible 00:55:58] backups on when the directory is down. Now, that said, we did have a customer incident where ransomware had gone loose in their environment and it actually encrypted their domain controllers, and unfortunately, it also encrypted those backups. It's important to have that data stored in more than one location. Recovery Manager actually allows you to store that data on up to three locations. Every backup you run can be stored on the DC itself, on the recovery manager server, which is where you'd want it for object level recovery and then on a third location of your choosing, just any UNC share.
You could put that onto a cloud share. You could put that into a WORM device, [inaudible 00:56:48] storage of some sort, bare metal recovery solution or a deduplication solution. Any of that would be a good way to go. Great question [crosstalk 00:57:00] answered it. Just a couple last things to show. We do have some resources. Nine best practices for Active Directory security. We did some breakout sessions at Ignite and we have a case study about that. And then using our IT security search feature, we did that webinar with you Randy just recently and that's on-demand webcast now on our website. So we'll make sure these links are available.
Randy F. Smith: Brian, thanks a lot. I mean, it's really impressive what Recovery Manager can do. I'm always blown away with that and thank you for testing us against the scenarios that we came up with.
Brian Hymer: My pleasure. Yeah it's a lot of fun to do that type of testing, Randy. Thanks for providing those.
Randy F. Smith: Well it makes a lot easier for me to be enthusiastic and positive about a sponsored solution when I know it works. Thanks again for making today's real training for free possible. We hope that this was valuable to you folks, and if you need help convincing management about this, and need to get into the nitty gritty how-to details, let us know between Brian and I, we should work something out, but until next time folks, have a great day, and look for the follow-up email to come from Quest on today's webinar. Bye-bye everybody.
Brian Hymer: Thanks everyone. Bye-bye.
