Well yeah, you should fix you.
If you're dipping the muzzle, or dropping your hands when shooting fast, it's has nothing to do with the sight.
("Shooting fast" as in drawing and shooting or moving the gun and shooting? Or shooting fast as in multiple shots? If the latter, you may also have too much recoil spring (unnecessarily heavy) and that's what's causing the muzzle to dip, and not really your fault and it only shows itself when shooting fast.)
Need much more info to give a good answer past that.
That said, how much are we talking?
How do you use the gun most of the time?
I might be tempted to split the difference, so it shot a little high one way and a little low the other. But if it's mostly used one way and not the other, no.