The folder you want to drop music files into is data\Sound\Gramophone. (I'm assuming you want more music playable via the u-boat's gramophone, and not new background music for various menus and situations, which would be a whole other process.)
IIRC the game will recognize both the .mp3 and .ogg formats, but most people seem to feel that adding gramophone music as .ogg files gives you a more stable game and is less likely to cause problems. General consensus also seems to be that you should choose one format or the other and not use both.
If you have .mp3 files or files in other audio formats you can convert them to .ogg using any number of different programs, many are available online as freeware or free trial versions.
