Open Windows Explorer... At the top select Tools -> Folder Options -> File Types
Scroll down to the excel file types. The two types you'll need to change are xls and xlsx.
You will want to highlight that extention and click advance -> Select Open -> Click Edit
In the "Application used to perform action:" Box scroll to the end of the box and add "%1". You need the quotes. Uncheck DDE and click ok. Click ok and do the same for xls or xlsx. Now open two excel docs and they will be in two instances!
For Windows 7 and Office 2010:
NOTE: ALWAYS BACK UP YOUR REGISTRY BEFORE MAKING CHANGES. I am not responsible if you screw up your computer changing these settings.
Click the Windows Start button/orb -> Type Regedit
Navigate to HKEY_CLASSES_ROOT\Excel.Sheet.8\shell\Open\command
Double click the (Default) key and it should say:
Change to "C:\Program Files (x86)\Microsoft Office\Office14\EXCEL.EXE" /dde "%1"
Rename command to command2
Repeat the following for Excel.Sheet.12
Full credit goes to this guy for figuring it out since the previous way was not possible any more. Link