Focusing on your original question, you can, but will it fit in your stock? I am assuming since you said "varmint barrel" you're talking about a heavier barrel, which may or may not fit the barrel channel on your stock. If the gun is factory and the sporter barrel is factory, then my guess is...