I don't think it's the scope/mount, because the following rounds are back on target. I'd suggest using the bolt assist, but I strongly suspect you've already tried that.
Do you push the operating handle fully forward before releasing the bolt? That might affect the bolt's battery position. BCG lube may be a factor.
Try doing the reload, then ejecting the chambered round, and checking the bullet tip for any deformation. The mag lips/feed ramps/mag catch may be affecting the first round feeding/chambering process.
Keep in mind, where the gun points is partly a result of the recoil pulse as it fires. Maybe there's something different happening with the first round's recoil pulse. A higher impact could point to a longer barrel dwell time.
Finally, try seating the bullet about .010" deeper for just a few test rounds.
As I understand it, the M4 feed ramps are intended for heavier, longer bullets. Precision rifles get shot with longer bullets, maybe they get shot with some that could be a loaded tad shorter.
FWIW, I set my seater for factory 55FMJ length, loading with HDY 75gr HPBT-Match. They feed and function fine, and shoot great as well in a factory Stag Model 6 Super Varminter. It may not be the precision rifle you're using, but it's what I've got.
For F T/R, single feeding is required. I bought the
Satern single feed magazine follower. As delivered, the follower blocked the bolt from hand cycling rearward until I cut about 1/8" of an inch off the front lip of the follower's upper ramp at a 45 degree angle. Since then, it works like a charm. I don't know if single feeding has any bearing on what you do, but I throw this out there for your consideration.
Greg