by doublemeat - 4/16/12 6:19 PM
In Reply to: Helpless Help by Allcom
Well put. I too have spent the last 10 or 15 years or so battling USB drives and data corruption - usually massive data corruption, and/or the drives won't even format. (As long as USB drives have existed.)
This has spanned at least:
- A dozen computer and notebooks.
- A couple dozen versions of Windows, Mac OS X, and Linux (Ubuntu usually).
- All of the major USB chipsets controller (and some cheapo ones).
- All of the major Firewire chipsets.
- All of the major controller chipsets (including the brand that many people swear is the only one that works...am spacing on the name at the moment but if memory serves it is the same or similar to the name of an ivy league university).
- 32-bit OSes, 64-bit OSes.
- Bare installs, bare+patched installs, and fully configured installs.
- Commercial name-brand computers, home-built and rigorously tested computers.
- 256mb RAM, 512mb RAM, 1gb RAM, 2gb RAM, 4gb RAM, 8gb RAM, 16gb RAM (not that that should matter).
- PATA drives, SATA/bridge drives, SATA drives. SATA 1, 2, 3.
- Thoroughly tested bare drives + enclosures, name-brand drive+"permanent" enclosures, drop-in docks, bare USB/SATA connectors+power.
- Bus-powered 2.5" drives, externally-powered 2.5" and 3.5" drives.
- New cables, old cables.
- Thick cables, thin cables.
- Cables with an RFI magnet, and (usually) without.
- Short cables, slightly longer cables.
- Pentium II, III, 4, Core2 Duo, Xenon, Core i5 systems.
- ASUS, Gigabyte, some other big brand, and OEM motherboards.
- FAT32, NTFS with default cluster size, NTFS with 64k cluster size, Ext2, Ext3, Ext4.
*** ALMOST ALWAYS *** with data corruption or just as often, with outright volume failure.
The only exceptions I've ever found to this, were with two...JUST TWO...name-brand commercial 2.5" drive+"permanent" enclosure systems. Those were the only ones that ever worked. One is working right now. It works on EVERY system, every time, with any USB chipset, any USB port, any OS, any USB cable. And bus-powered. It's an NTFS volume with default cluster size. It has never failed me. (I can't remember what the previous one was but it eventually died...but had a long and productive service life.)
The one out of only two that ever worked and is working now, is an old 2.5" Seagate FreeAgent Go. (And I hate Seagate. In my 20 TB server, and in my 30 years as a computer power-user, Seagates have failed far and away more frequently than other brands [with the exception of the IBM DeskStar debacle which yes I owned TEN of the bad ones; including their so-called "enterprise" versions - actually those have failed at a slightly higher rate and earlier on average than their equivalent desktop models, at least in my objective experience.)
When I try to use my own drives in external enclosures, I try to buy high-end 5400 RPM drives (for less heat buildup), and test them rigorously on a SATA port before moving to USB enclosure. As for the enclosure, I only buy the best and research the chipset when buying to make sure I get whatever is most highly recommended at the time. And they always fail. (Even though the drive may continue to be perfectly useful and yield a long life when connected back to SATA.)
I do this every now and then, thinking "surely by now they've got the USB drive kinks worked out by now". Nope. Never.
So yeah, it drives me CRAZY when people give such trite answers as "fluctuating power" or USB cables.
I do have a fairly reliable solution for USB HDD storage now though.
Here's what I do:
1) Buy two expensive name-brand 2.5" HDD USB drives. [Neither name-brand nor expensive necessarily has anything at all to do with reliable, but just in case.]
2) Run Solaris 11 Express [formerly OpenSolaris and can still go with OpenIndianna for openness] in a VirtualBox virtual machine (with VT-x hardware assistance).
3) Map the two identical USB drives via "passthrough" raw VMDK, directly to the virtual machine (so Solaris thinks it has two extra permanent HDDs).
4) Configure the two drives as a ZFS mirror.
I should point out this is a very tricky approach. Since the ports, addresses, and physical drive numbers to USB change depending on what drive is plugged into what port, and in what order, the only way to get this to work is to launch the virtual machine via CMD or bash script. In this script, you have to delete the drives from the virtual SATA adapter, delete the raw VMDK files, recreate them pointing to the correct physical drive numbers, then reattach the new raw VMDK files to the virtual SATA adapter. All in CMD or bash script. Once you get the kinks worked out though, this approach is quite solid. Do NOT attempt to just use passthrough USB port mapping. Trust me, it either won't work or will be terribly unreliable.
Also, do stick with Solaris and not BSD or Linux FUSE for ZFS. I tried to get both BSD and ZFS-FUSE working, and neither is satisfactory for this approach in particular. Plus, Solaris 11 just has the best implimentation of ZFS, hands-down. (For example, it is easy to disable write-caching. Which you WILL want to do. With it disabled, you'll get good write performance. With it enabled by default, it will be so slow as unusable. Don't listen to the fear-mongering about disabling it. You WILL NOT lose data in the event of a power outage [or USB unplug]. The only thing that will happen is that whatever transaction being committed, will be lost. Which, is precisely the same thing that happens with any journaled filesystem, and for this setup, will happen to ZFS whether you have write caching enabled or not.)
The main benefit to ZFS is NOT mirroring. You can get that with NTFS. What it gains you - and this is invaluable for USB drives - is error checking. All data is stored with a checksum. In the case of silent corruption (frequent with USB drives), it is detected by the filesystem. If a file on one drive is corrupt, it will read it of the (hopefully not corrupt) second drive, and heal the first automagically. If it can't fix the corruption, it TELLS YOU.
To me, USB drives with important data, without ZFS is just stupid.