One of my “safer at home” projects is updating my file storage system. The drive I put new camera captures on was getting close to filling up, so I needed to expand my system by purchasing a 8TB Seagate external drive that could hold the contents of a partially full 6TB drive, along with the contents of a partially full 2TB drive.
It took me many hours of copying with CarbonCopyCloner to transfer the files from both drives to the new 8TB drive, which always tests my patience, and my tendency to want to watch the pot to see if it’s boiled.
With the copy complete, the next to do is a major reorganization of my folder structure to better fit my current needs and to work with my backup scheme. As part of this, I’m going to erase and reuse some older backup drives, but before I erase those drives, found myself with a nagging question. Did my computer actually copy all my files correctly to the new drive?
Most file copy operations, including what I did with CarbonCopyCloner, are optimized for speed. They read the file from one location and write it to another without verifying that the file was written correctly. Verifying a copy would take re-reading each file and comparing them, which would take a lot more time. In the case of my 5.5TB of data, it would have to read 11TB total of data. 5.5TB on the original drives, and 5.5TB on the copies.
Since my copies weren’t verified, it’s entirely possible that when I copied my files to this new drive, files that hold decades of work, valuable drum scans, irreplaceable originals and memories, that some did not copy correctly, and I could be losing some data. I used to accept that risk in the past, but experience has made me less willing accept it going forward. So what to do then?
CarbonCopyCloner has an option to compare your backup with your original, but for the size of my archive, it was going to be a very time consuming project, and difficult to organize. Fortunately I remembered Lloyd Chamber’s IntegrityChecker software that was designed to do just what I wanted.
First a couple lines about Lloyd and why I’m trusting his software to check my files. Many years ago I met Lloyd when he attended one of my workshops. He was using 8×10 film at the time and trying to push the bounds of what it could achieve…no minor feat. His film was of fantastic quality, but he was still not satisfied. He’s the type of person who obsesses over details in a way I greatly appreciate. But he’s not just a photographer. He has a couple patents to his name for compression technology he used in his very popular DiskDoubler and RAMDoubler software. He has the knowledge and experience to get very deep in the weeds of some interesting computer and digital imaging problems, and he blogs about lens and camera testing at diglloyd.com.
IntegrityChecker validates files in a very unique way. It creates a cryptological hash for every file on a storage volume that can be used to check if the file has been changed in any way. This lets you check the integrity of files and backups in the most efficient way I know how.
So now I’m in the process of creating hash files for my “original” disks. Once all the hash files are created, I’ll use those to validate that my multiple backups are faithful copies of the “original.” That will let me have peace of mind that I have good copies of all my files, and let me decide which copies are redundant so I can re-use those drive.
This kind of integrity checking is something we should all do, but since it’s not built into the operating software we use, it doesn’t happen unless we seek it out. If this is something you’re interested in, check out IntegrityChecker on Lloyd’s website.
I think it’s important to note that this is more of a “expert” level tool. It’s offered in both GUI and command line versions, and it’s going to take some understanding of the underlying principles of what it’s doing if you want to apply it correctly. Because of that, it’s not a tool for everyone, but it’s one I wish I had started using a lot sooner. For now it’s the easiest way I know to ensure my files copy correctly and don’t change once they are copied. Check it out and see if it belongs in your toolbox.