First thing is you need to enable SSH.
I already wrote how to do that here -> How To Enable SSH for ESXi 4.1
Next open an ssh program like PuTTY.
Once connected you need to run the following command as root.
Note: This turns off the USB passthrough for your VMs.
At this point you can plug in the USB device. 4.1 auto mounts it so, type:
ls /vmfs/volumes/NO\ NAME/
This should show anything on the USB Device.
You can copy, delete whatever you want to do.
Source: Link Link Link
This blog talks about how to use some of the utilities if you are coping a new vm over.
Once you are done, you should turn back on passthrough by typing:
UPDATE: I found this does work find but for only FAT/FAT16 formatted drives. I have no yet found a solution to use FAT32 or NTFS.