Copying Files: Shouldn't this be Simple?

Look, copying files from one server to another shouldn’t be hard. It just shouldn’t. Even old versions of MS-DOS had XCopy.exe, and modern versions of Windows have the whole fancy Explorer drag-and-drop interface. File copying should be easy, right?

Figure 1 The XCopy command

File Copying: Easy. File Migrating: Hard.

If you’re just dragging some application files from server to server, odds are that Explorer will do the job just fine. It’s when you need to migrate files – as in moving home directories or project folders from one file server to another – that the pain starts.

The biggest problem is those annoying Access Control Lists (ACLs) that we can’t seem to live without these days. Remember when “Everyone:Full Control” was good enough for every file and folder? Ah, the good old days. Today, people get bent out of shape when file permissions aren’t maintained, and XCopy doesn’t do a good job of maintaining them when it copies files. You remember the rules from your Microsoft certification exam, right? Move a file somewhere on the same disk, and permissions stay put; copy the file, or move it to a different disk (or server) and permissions inherit from the new parent folder.

Command-line tools like Robocopy have long helped administrators keep permissions intact, but it’s still not a perfect tool. Even Windows PowerShell doesn’t really solve this problem, since it just relies on the operating system’s underlying, low-level file copying protocols.

Figure 2 PowerShell file copy options

Key Concerns During a File Migration

If you’re moving bunches and bunches of files with highly variable permissions – such as moving users’ home folders to a new file server – you need to look for some specific capabilities in your file copying tool. At the very least, you need to be prepared to deal with these issues manually – painful as that might be.

  • Security. NTFS permissions on files, folders, shares, and so forth all need to remain intact. The trick comes when you’ve used local user accounts, or when you’re moving files across domains. A tool should at least log every file copy operation, and highlight instances where it couldn’t properly create the new ACL. For that matter, I’d like the option to have the tool just re-create local users and groups. I realize we primarily use domain groups for permissions, but it seems like every file server always has one or two local groups that are used for some weird permissions need – like a department that wants some level of autonomous control over “their” server. Just create that stuff on the destination server if it doesn’t already exist.
  • Attributes. All file attributes should remain intact, including the NTFS compression attribute.
  • Override. You’ll doubtless run across a file that even you, the administrator, don’t have access to – often because someone else has taken ownership of the file. Tools that can exercise Windows’ admin override can let you copy the data anyway, and that’s going to save a lot of manual exception-handling for you.
  • Locks. Unless you unplug the server from the network, a big file copy will almost always hit a locked file or two. Stupid Access databases! But a really robust file-copy tool will retry those locked files automatically, and let you know if it isn’t able to move them because they never unlock.

These are the basics to get a bunch of files copied from one place to another with a minimum of fuss – and if you’ve got a tool that handles these concerns, then you’ll also have a minimum of manual effort.

Figure 3 File Permission Properties

Business-Grade File Copying

I’ve recently been helping customers migrate a lot of file servers, and there are a few things we’ve had to manually hack together that I’d like to see become part of a real, business-class file copy tool. Even Robocopy – much as I like it – doesn’t handle these very well.

  • Scheduling. I’m finding that file migrations have to happen during maintenance windows, and those windows seem to be shrinking. Guess we’re all working late hours these days and need our file servers online! What I hate is having to rely on a batch file, scheduled in Task Scheduler, to run at night. About half the time, the batch file hits a problem, aborts, and I’m left with a half-migrated batch of files. I’d rather see a tool that can be scheduled on its own, and that can “just keep going” if it runs into issues (logging the problem for review, of course).
  • Multi-threading. This is a big deal: Old-school tools are single-threaded, and copying tens of thousands of files can take forever, even though the tool is only using a fraction of the server’s processing capability. Get me some multi-threaded action! Copying dozens of files in parallel is the way to go. Think about it: If you’re migrating during a maintenance window anyway, why not leverage the full power of these multi-socket, multi-core servers? Two threads halves your file migration time; get a half-dozen threads running and you can actually fit the migration into the window.
  • Shares. If I never have to re-create another user home directory share, I’ll be thankful. Why don’t most file copy utilities just do this? Scripting the creation of hundreds of shared folders is painful, not to mention migrating the ACLs. It seems like someone writing a really pro-level file migration utility should be able to do this more easily.
  • Email notifications. I’m tired of having to hack together my own status e-mails during a file migration; why can’t a tool just include this?

Caveats and Bonuses

While I’m dreaming, there are some other things I’ve had to script recently that I’d like to see a tool incorporate – mainly because my scripted solution wasn’t entirely satisfactory.

Synchronization is one of those things. Robocopy isn’t 100% reliable at syncing files between two servers, so I’d like to see something that can just copy changed files from one place to another.

Here’s a real annoyance: Offline file stubs. You know, those little “shortcuts” that Windows creates to make the Offline Files feature work? I hate those things. I’d like to be able to exclude them from a file migration, but it’s tough to actually determine if a file is in fact one of these things. Windows does such a good job at making them look like real files, you can’t always distinguish.

Encrypted files have also been annoying. 99% of the time they work, but sometimes they just don’t. I’d like the option to copy them as unencrypted when they can’t be copied as encrypted. Of course, I want that prominently logged, but it’s easier to go re-encrypt them after they’re copied than to manually copy them all. I can write a script to batch-encrypt them, for example, if that’s what I need to do.

What’s Your File Copy Solution?

So that’s my list of file copy (and migration) annoyances and wishes – what about you? What tools are you using, and where do they come up short?