I've got a similar configuration and SH3 (+GWX3) runs quite satisfactory. I'm no expert but I would assume that some of the settings of your graphics card might be set not correctly.
do you know what your anti-aliasing and anisotropic filters are?
(mine are set to AA: 4x and AF: 8x, but I have a Nvidia 6600GT)
I suppose some of the members here are a greater help than I am.
good luck!
