I anneal after every firing. It may be overkill for me to anneal after every firing, but it does eliminate me from second guessing myself as to how many firings I have between annealings. It's hell getting old!
I am still running a factory barrel, which historically have rather large neck area diameters in their chambers. This lets the neck of your brass swell up more than really necessary to release a bullet when firing with most neck wall thicknesses. Thus, to resize my necks, the brass will therefore be worked more than what would have been necessary if shot out of a custom match grade chamber with a tighter neck diameter. The more you cold work brass, or any metal for that matter, the more it will work harden. A custom match grade chamber with the neck area cut to a diameter custom tailored for your favorite brass neck wall thickness could definitely cut down on the frequency of annealing.
I still have plenty of life yet to go in my Savage 10fp barrel, and I am in no rush for a custom barrel as long as I am getting 1/3 MOA accuracy out of it. Annealing every firing for me has cut my group sizes in half compared to what I was originally getting on 4x to 5x fired brass. Annealing also finishes drying my cases after ultrasonically cleaning them anyway, so I get a double benefit out of doing it.