Are you shooting prone/bench? When you fire a shot, do you notice the reticle jumping in a certain direction or is your recoil straight back? I shoot 1/2-3/4" groups with my 22LR @ 100y, but when I shoot my 308 my recoil management is horrible and I end up getting "flyers". When I notice a horizontal reticle shift when I pull the trigger, I notice horizontal variation in my groups. If I get my NPOA fully dialed in and manage my recoil so the recoil jump is only slightly vertical, my groups shrink way down.
And then lastly I let my friend shoot my rifle who shoots much better than me, and through my rifle which isn't setup for him, he shot a good deal smaller groups as well...
On a 308, 100y, FGMM 175gr, I highly doubit it's going to rifle/ammo and it's always the shooter.