Flash Disk Policies tab.

Plug in your USB drive. Go to Device Manager, find your USB drive and (right click) get the Properties. On the Policies tab make sure "Quick removal" is checked. If so, as long as the USB drive is not blinking it should be fine to remove without having to go through the Task Manager USB icon.