I've had it happen once with a brake on one of my Remington 700's, it was a cheaper brake from Damage Industries, it was on sale, and I thought how bad could it be? Pretty bad as it turned out, even though it was mounted with shims and light torque, the accuracy was horrible, similar to yours...