Preventing silent data corruption

Zeynel Abidin Öztürk -
13 days ago

Updated 12 days ago.

When a thief makes into your house, they will probably leave traces or mess something up, so you'll look for what is missing. But if things go missing by itself with no trace, that's not something you're prepared. Such things just don't happen in the real world, at least we don't consider such a possibility.

The same goes for the files. When you stumble upon an important Word file on your drive with 0 bytes, maybe you're going to think that you didn't copy properly the file at the time, or some other program has broken it. But this is digital world, and your file is probably a victim of "silent corruption".

If you have a "catastrophic" data corruption that even your OS won't boot, you're going to try to fix it and probably restore your data from your backups or cloud. But silent data corruption may make its way to your backups and also your cloud drives like Dropbox. So this kind of data corruption is going to kill your backups too.

File corruption can occur because of various reasons including not shutting down properly, hardware failures, bad sectors, sudden power failures during file operations, malware, malfunctioning anti-virus.

It's surely not nice that such data corruption can still occur easily in 2019. But there should be some solutions, isn't it? There are, at least you may want to try one of them.

What is silent corruption?

It's a term for defining corrupted files without any warning from the operating system. Such files can become empty (0 bytes) files or they can stay in the same size, while their content became filled with garbage.

This kind of corruption is very bad, because even if you backup your data regularly, your backup program will backup corrupt data as well and overwrite your actual, non-corrupted file. If you're aware that a file is corrupted, you can restore it before it's overwritten. But when a file in your e.g. song archive gets corrupted, you won't even notice. It can go unnoticed for months or years.

Windows or your operating system will probably detect corruption when you use a tool like chkdsk, or fsck in Ubuntu Linux. But you probably won't notice what errors chkdsk found and recorded silently on your Windows Event Log.

What is the proper solution for it?

There are "enterprise-grade" solutions for data integrity, like RAID arrays. You can also do this with your home computer, by buying multiple disks of the same model and size. But that isn't going to be a cheap solution and won't be (easily) possible for external USB drives. What is worse, it's said that RAID-1 will not detect or prevent file corruption, it can only help to recover from complete disk failures. So I think it's cumbersome to have RAID, when the software would provide at least some data integrity protection for the casual user.

If the files you want to protect aren't changed at all, and if you don't access them frequently, you can use WinRAR to create a RAR file with volumes and create some recovery volumes (this is different from recovery record, that is used in single-file archives). The recovery volumes can recover the missing or damaged volumes, in case some parts become corrupted.

The proper solution would be using a file system with data integrity features, but unfortunately NTFS (or default Linux file systems) doesn't provide it, at least not adequately, because data corruption is still an issue.

It's worth mentioning that Windows has a tool for finding corrupted system files. The sfc /scannow command will scan Windows system files and fix them by restoring original files if necessary. But as you noticed, this is only designed for fixing Windows system files. And Microsoft stores original versions of these files somewhere (%WinDir%\System32\dllcache) for restoring.

Recommended solution: A Windows utility for detecting file corruption

Google's skilled search engine fell short when I was searching for such a program, but after various searches, I've found a free utility.

Anti-Corruption Defender scans the folders (or entire drives) you selected, and creates checksums for every file. After a few days (or many days) it re-scans these files and compares if checksums are different.

The program will detect modified files, but this doesn't always mean all modified files are corrupted. The program lets you ignore frequently changing files and folders (you can ignore any file by location), so you can focus on files that should stay intact. If the program says a .RAR file (which you don't touch) is different than how it was 30 days ago, then it may mean it's corrupted (or maybe modified by a malware).

Anti-Corruption Defender can work both with local drives and external drives. In the case of external drives, it can scan the drive as you insert it.

I do not recommend adding the entire OS drive, as it'll take very long to scan and will probably slow the system down. Instead, only add the folders where your important files and backups are.

Note: I'm not responsible if data loss or corruption occurs because of using this program. I tested it and I'm using it, but if your hardware is about to fail, it can go down with any heavy activity. Make sure you backup your data first!

The program is portable and stores settings in its folder.

You need to do your backup when using Anti-Corruption Defender. The tool is not a backup program, it only checks the files are staying completely the same. As I said, this is done by creating checksums for files. A checksum is a few bytes of data that is generated from the entire file or any piece of data.

The program doesn't provide any way to restore files in case of file corruption or loss. Anti-Corruption Defender will only let you know that some files are not intact, so you can check them and restore from your backup if necessary. And you will also be able to protect your backup from being overwritten with bad data.

Download: Anti-Corruption Defender

The real solution may be ZFS

ZFS is a file system developed by Oracle and includes strong protection against data corruption, as well as supporting high storage capacities. It provides data integrity features by creating checksums or SHA-256 hash throughout the file system tree.

The "checksumming" is done in such a way that it creates a "tree" and the system self-validates itself. Whenever a data block is accessed, the checksum is calculated and compared with the data. If checksums do not match, ZFS can heal the data from the storage pool. By default, ZFS creates a copy of data but additional redundancy can be provided, so a copy of the data will be written 2 or 3 times.

Various Unix operating systems support ZFS but there is no support for Windows in sight yet. Experimental root support for ZFS was added to Ubuntu recently, in a beta release.

How do you prevent silent data corruption? Were you struck by it, too? If you have an alternative or better method, please share it with us in the comments box below!

Share this page on:

Join our mailing list!

Your email is kept private and we don't spam.


Info