A day in the life of a pre-sales systems engineer ebbs and flows between learning new tricks, doing what you did yesterday again and dealing with issues no one understands. The other day, I had one of the latter. It was a true test of my skills as an engineer and the capabilities of a few specific products that came together in the end for the win — and a win in record time.
A new customer purchased Quest Secure Copy to port data from an old server, which happened to be a domain controller, to a new server with a current operating system. This was preparation to decommission the older server. Not an unusual situation on the surface.
The customer and I had worked together in the pre-sales timeframe to ensure that he understood what needed to be done and how to get the most out of the product. The expectations were set and in the first part of the migration, everything went as expected — no errors. It was the second part of the migration where the results were unexpected, unusual and disappointing.
Having moved all of the group-based data, complete with all the ACLs, shares and permissions for the shares successfully, the customer was eager to move the user data — all 200+ GBs of it. Having listened to the engineering recommendations, the customer made a copy of the successful job then changed the pathing of the source and target for the data migration. Once saved, he fired off the job. The data started to move as expected. Out of curiosity, he checked the permissions on the data in the target location and immediately stopped the job. The security was not being properly migrated. The end result was that the access to the data was a mixture of the user’s ACEs and an ACE which pointed to a local group on the server named Users or Server\Users. This would not allow the domain users to access their own data.
The customer called support, who tried to ID the issue to no avail. It just didn’t make sense. The support rep recalled the customer speaking highly of me and used that knowledge to rope me in for help. This is where the technical story really begins.
Out in the market place, Secure Copy has a few competitors. One such competitor, RoboCopy, is a free tool from Microsoft that would work, but it’s quite user hostile with poor (or, more realistically, no) logging. Secure Copy has superior logging on each and every aspect of the jobs being run. And that’s where Secure Copy came in to help locate the issue in this scenario.
At the beginning on the help session, the customer and I scoured the configuration looking for some mal-configuration. Finding none, we looked in the logs. On the surface, nothing stood out as important. It wasn’t until we looked at the security in the customer’s server where the data will be copied from that we got our first clue. Looking at the security on the parent folder and a few child folders, we saw that the referenced group accounts granting access were Administrators and Users. Keep in mind that this is on a domain controller. I would have expected that the groups referenced to be domain groups so I didn’t pay attention to these settings initially.
We compared logs from the successful job with the job where the security was incorrect. That’s when I realized what the issue was. I found that in the successful job, the groups being granted access were domain groups of the referenced Security Identifier or SID. The format for a domain SID is S-1-5-21-<domain SID>-<Relative ID of the AD object or RID>. The key indicator was the fourth set of numbers or 21. This denotes that the source of the account is a domain account. When comparing the same data in the unsuccessful job, I noticed the SID of the Users group was S-1-5-32-545. The 32 indicates that it is a local built-in account. Ironically, every server has this group.
So, what was happening was that the local group referenced was migrated from Source Server\Users to Target Server\Users as one would expect when performing a SID match. Now, it made sense! Secure Copy was migrating the permissions from one server to another and found a Users group on the target server with the exact same SID and did what it was programmed to do, which is to convert the security from Source Server\Users to Target Server\Users during the migration.
Now that we had found the problem, we had to find the solution. Basically, we just needed to change all references from the Users group to Domain Users and all references from Administrators to Domain Admins. Using the native tools to change the ACLs could cause access errors and would take a long time to complete with no recourse if something went wrong. That’s when we decided to download and use a second Quest solution, Security Explorer.
After installing Security Explorer on the customer’s desktop, we connected to the source server to explore the permissions. The exploration confirmed what we had seen previously with native tools. From there, we used the Security Explorer permissions backup and recovery feature to back up the ACLs to a local file in case we needed to restore the permissions.
From there, we crafted a search of the data looking for all explicit references to the Users group. When that completed, we used the replace function to put the group Domain Users in its place. We crafted a second search looking for the Administrators group; using the replace function to put the Domain Admins group wherever the Administrators group was referenced. The entire process took less than an hour. Not one user lost access to the data during the operation.
With the permissions now set to domain groups rather than local server groups, the customer was able to successfully migrate the data from one server to another with all of the user and group permission intact. The customer will use the additional capabilities in Secure Copy to point the users’ home directories to the new location at the scheduled cutover time.
In this case, it was the little things that made all of the difference. Being able to review the logs of the jobs was the most useful in identifying the actual cause of the issue; which was environmental because of the poor application of security. Having a solution to be able to back up the permissions then change the security quickly and accurately was the other. We never needed to restore the original permissions, but it was a good thing that we had the ability to do so if needed.
As you can see, Quest has the solutions and the know how to get the job done quickly and successfully when it comes to data migrations.