What charging handle is it? It should not come loose. Is the slot where it latches really worn?
As for the other issue, I would look for irregular wear on the locking lugs and bolt. And it might be a good idea to check headspace. Maybe even polish the chamber a little. It sounds weird.
Good...