I've found that Explorer is often the culprit

Whenever I've been unable to remove a USB drive safely, I first check the task manager to make certain that I've got no background services running (backup/anti-virus) that might be (unknown to me) accessing the drive in question. Once, I've eliminated that possibility, I generally find that Windows Explorer itself has somehow "locked" the drive in question, and simply closing that window will "free" the USB drive so that it can be removed safely. I'm running Windows XP SP2.