Given that you haven't noticed any "trends" to when the flier appears, I would just chock it up to shooter error? I mean no offense, b/c even with those fliers that is still some pretty tight shooting. Even with a top-notch custom rifle, its no easy task to put them all into the same hole.
If you can, chrono a few groups and see if you notice any velocities discrepancies with the fliers.