Carried a couple different ones for 16 years... the 40-50 degree temp shifts and moisture from condensation was more damaging.
As far as hot cars go, I have had LED readouts melt and run. I have had a couple of regular scopes "bubble" as the glue btw the lense groupings overheated.
Never had heat kill a mil thermal, but,
by all means, take the batteries out. Back in the day when the "football thermal" took 16 AA batteries, yes, the batteries caught the heat clap, and dripped all over everything.
Had one pvs 14 get battery heat clap drip...
Has one 9v bushnell rangefinder get it.
Lost a bunch of D cell, C cell, AA, lights battery clap drip ruined.
SO FAR, no CR123 failure....