What's your OAL? Long ogived VLDs really like to be within about 0.020" of the lands or closer - too much bullet jump and they'll engage the rifling slightly off-center. An example of this might be consistently seeing groups with three shots touching and two shots randomly thrown out.
If you haven't already, I recommend finding an OAL gage from midwayusa or brownells and pushing the cartridges as long as your chamber will allow. Remember to back your powder off a little if you increase OAL because you'll likely see an increase in pressure.
I had a factory remington 308 that loved the 168 Berger VLD. I used 45grains of RL15 with 210M primers in Winchester brass. I seated bullet to fit in the mag. I now shoot 47 grains of Varget with CCI BR2 primers in winchester or Lapua brass.
I have good luck with all my 30 cal's and Berger bullets being seated between .020 - .030 off. the landsz.
First off is the barrel free floating? Is it bed to the stock?
One of those could be the issue.
I would stick with varget as it is a great 308 powder. Try moving up and down .3 grains. Seating depth could be an issue. Try .03 or less off the lands. Check primers, I used 210M and the work but I have changed to CCI BR2.
This year I used Berger's 168 vld in Montana perusing mule deer and while I practice taking shots out to 600 have yet to take a shot over 290. The nice thing about it is the B.C. is so close to the 175 smk they can be interchanged. In other words if you run them at the same velocity as your 175 load, your drops that you are used to holding won’t change.
I started the OAL at 2.825 and ended up dropping them back another .015 and they are now grouping between ½ and ¾ moa consistently. With Lapua brass, Winchester primers and Varget I am getting 2640 fps out of the little tactical. It put the smack down on my mulie.