Well considering that SH3 and 4 cost next to nothing now, there is no reason not to pick them up, providing you have the time and disk space for them + mods.
However their ages are starting to show (visually) particularly 7 year old Sh3 - it isnt very pretty to look at after playing SHV
but you do get the extra Uboat types and 43-45 campaign years.
If you want to play in PTO with the US fleet boats then Sh4 is pretty much essential.