I'll take a shot at this- but may be off.
Typically any shift in POI is attributed to some sort of stress, contact point, uneven surfaces, you name it. That said, is the barrel nut seated against the upper? You can lap that front surface (I have the 223 tool- which wouldn't help you on a 308).