I don't think it makes a "huge" difference, but when you're trying to get it as accurate as possible it's an area for improvement.
I agree with your gunsmith, but who's going to put a factory barrel on a custom action?
One thing I've noticed is that there's a big difference in what gunsmiths...