If anything on the usb device is still being used by a program on your PC (or Mac) this will happen, even if you don't notice any activity on the drive. Sometimes it can be as innocent as a photo that is still open in Preview (on a Mac) even if it is parked in your dock or hidden under other windows.

The safest thing to do is restart. Sometimes something you aren't aware of will be running and you won't be able to quit or restart. You can try using the Force Quit method (Mac), and if all else fails... bail out by holding down the power button.