Everyone else has mentioned the big contributors. Keep in mind that the trigger lock time on a bolt-action rifle is somewhere around 2-3 milliseconds. For a mil-spec AR trigger that is heavier and gritty, the lock time can be as much as 15 milliseconds. So basically the rifle has more time to move and is more likely to move between the time the trigger is pressed and when the bullet leaves the muzzle.
Combine that with junk ammo, junk barrels, junk optics with little or no magnification, shooters who don't really understand or apply the fundamentals of marksmanship, and the results are as the OP described.