Like others have said, slow down the cyclic rate. Almost all problems are from too fast a cyclic rate. Mag issues, extraction issues, ejection issues... they are all far more likely to be caused by too fast a cyclic rather than too slow a cyclic rate.
Before you start tinkering, please make sure you are using the correct length buffer tube and buffer weight for an AR10.
AR10s have longer BCGs... which means if you want to use a standard ar15 carbine buffer tube you will need a shorter buffer to accommodate the longer BCG in the buffer tube.
When changing stuff around... if you are going to use a standard ar15 carbine buffer tube, you will need a shorter buffer like this one (2.5" OAL):
M5 .308 Carbine Buffer for use in .308 Carbines with aA .308 Carbine Receiver Extension (buffer tube). This shorter .308 buffer is specific to .308 Carbines and will not work in .308 Rifles or AR15 platform firearms.This true mil-spec buffer is AMERICA Made and features three separate weights...
ar15discounts.com
The problem with the shorter buffers is, they have fewer weights in them, thus they can't be as heavy, so they won't be as effective as slowing down your cyclic rate.
If you are going to use a standard length ar15 carbine buffer, you will need an A5 length buffer tube. LMT, Armalite, Vltor, Magpul are the most common sources of these length buffer tubes. These buffer tubes are slightly longer and have enough length to accommodate both the longer ar10 BCG and ar15 carbine buffer.
My recommendation would be to start with an A5 length buffer tube (not sure what you currently have), full mass carrier, H3 buffer and a tubbs flatwire 308 spring.
This setup is cheap, but honestly, they should be, it's a spring and weight. Doesn't need to be complex or expense to work well. This is what I use on my LMT 308 (16" midlength gas). Adding the tubb 308 spring noticeably slowed down the cyclic rate and made the weapon much smoother and more enjoyable to shoot
There is a machine gun enthusiast who did a bunch of cyclic rate testing with different, buffer tubes, buffers and springs. Pretty much everything you can imagine (Tubb springs, springco springs, hydraulic buffers... everything). Very informative, worth your time, especially given what you are going through (link below).
Why the A5? Commonly known benefits of the A5 system: Allows usage of the rifle length spring Extends the LOP for those with long arms Longer buffers with...
c3junkie.com
Hope this helps.