If I was doing it for a customer I would take it down just enough until the surface was uniform. You will have plenty of tenon length especially if you use an oversized lug. As it is won't have any effect on performance. I just know that someone would eventually remove the barrel and think I may have done a shoddy job. If it was my personal rifle I wouldn't be concerned with it.
Chase