Online storage is NOT THE ANSWER

by pgrobin - 5/8/10 5:28 AM

In Reply to: On-line storage may be the answer by colingbradley

Read the service level agreement with these 'free' online storage companies, you will find that they do not guarantee data integrity. This has been proven multiple times. Using Flickr or other sites, while convenient and useful for most of us, is not an acceptable backup solution. Always backup your image files locally.

There are two good options: Optical storage or RAID 5. Long term storage (more than 2 or 3 years is best done on optical since the degradation of the media is very slow. Magnetic storage is great but only if the health of the drive is verified periodically by reading the files and possibly by re-writing them (some good points have been made about magnetic degradation)). Even optical should be read occasionally since it can and will fail.

Other replies have mentioned the integrity of the hard drive, that silent corruption doesn't happen. Nonsense, silent corruption is a big concern in the enterprise storage industry today. Sun's ZFS file system specifically addresses silent corruption and its prevention. Hard drive electronics do not know anything about the files stored on the drive, only the file system knows what's there. The file system requests data from specific locations on the drive, if the drive can get data from those areas then it has been successful. It has no way of knowing if the data is correct.

To overcome silent corruption one technique is to use CRC checks. But most FSs store the CRC on the same drive as the file, so relying on that information can be problematic. Storing the CRC on a separate drive, as ZFS does, helps. Using a RAID 5 or RAID 6 array is also very helpful because a failure of one drive will not destroy the file.

I've been working with JPEG files since 1992. I have only seen a software issue once, way back in 1995 or '96 (can't remember exactly when). We could not open all of our original JPEGs created with a JPEG plugin with Photoshop 2.0. We were in Photoshop 3.0. The answer was to install the original plugin used to create the files, they then successfully opened in 3.0 and we then re-saved them using the 3.0 file save interface. We've never had the problem again, because JPEG had at that point been stabilized as a format.

The shifting and gray bars reported are due to corrupt files. Even though the user can see the corruption this is still 'silent corruption' because the drive nor the file system reported a problem. But the files are damaged and this is beyond repair, the data doesn't exist on the drive anymore. The original poster didn't say whether the files, once written to the hard drive, were verified. If not then perhaps they became corrupt on transfer, hard to say. That's unlikely but possible. But somehow the drive has been damaged magnetically and changed values of stored data, or the data never made it to the drive correctly. This should be a lesson for everyone reading this post, store your valuable photos on multiple devices. Without tremendous resources to use systems that automate health checks and data integrity this is the best solution. Read those files on a regular schedule, once a year or perhaps every 6 months. Keep optical disks in a cool dry and dark environment to prevent (or at least reduce) the chance of delamination, physical damage, or UV damage. Buy high quality optical disks, saving a few bucks is not the best way to ensure your files remain viable over time. And consider putting the files on two optical disks, creating a double backup. Then take that second optical and store it somewhere other than in your home or office (a safe deposit box is good).