Instead of deleting things to the recycle bin, is there a way to just permanently delete the file from my hard?

You can do the shift-delete thing each time, or you can right-click on the Recycle Bin, select properties, and choose "Do not move files to the Recycle Bin. Remove files immediately when deleted". Then you won't have to do the extra shift-key step. Each time you delete something, it will be gone permanently.