Write cache, not read cache
In your definition of cache you said "the computer copies the device into its faster memory and then reading and writing to the faster memory instead of accessing the device each time."
You imply that the entire contents of the device are copied into memory. As far as I know this is not done. Indeed most USB memory devices nowadays are bigger than the available memory. USB memory devices are treated like disks and as such are organised in 'sectors'. This means that reads and writes are done one sector at a time. This can be more data that what was required to be read.
If the device is deemed by Windows to be 'Optimised for performance' then a write cache is used. This is when it is important to remove the device safely. But there can be other writes in the background or files opened so it's always better safely to remove the device.