According to Wikipedia:
The Criminal Investigation strategic plan is composed of four interdependent programs: Legal Source Tax Crimes; Illegal Source Financial Crimes; Narcotics Related Financial Crimes; and Counterterrorism Financing. These four programs are mutually supportive, and encourage utilization of all statutes within CI's jurisdiction, the grand jury process, and enforcement techniques to combat tax, money laundering and currency crime violations. Criminal Investigation must investigate and assist in the prosecution of those significant financial investigations that will generate the maximum deterrent effect, enhance voluntary compliance, and promote public confidence in the tax system.
Legal source tax crimes
Investigating Legal Source Tax Crimes is IRS-CI's primary resource commitment. Legal Source Tax investigations involve taxpayers in legal industries and occupations who earned income legally but chose to evade taxes by violating tax laws.
Illegal source financial crimes
The Illegal Source Financial Crimes Program attempts to detect all tax and tax-related violations, as well as money laundering and currency violations. This program recognizes that money gained through illegal sources is part of the "untaxed underground economy" that threatens the voluntary tax compliance system and undermines public confidence in the tax system.
Narcotics-related financial crimes
The Narcotics-Related Financial Crimes Program was established in 1919, and is one of IRS-CI's oldest initiatives. Its goal is to utilize the financial investigative expertise of its special agents to disrupt and dismantle major drug and money laundering organizations. IRS-CI's role in supporting narcotics related investigations was highlighted in the 1996 book, The Phoenix Solution: Getting Serious About Winning the Drug War, by Vincent T. Bugliosi.
Counterterrorism and espionage
Following the terrorist attacks of September 11, 2001, IRS Criminal Investigation actively participates in federal counterterrorism investigations. In addition to standard investigative support, IRS-CI Special Agents add financial investigative and computer forensic expertise to terrorism investigations. IRS CI's support on investigations related to counterterrorism was highlighted in the 2013 book "Treasury's War: The Unleashing of a New Era of Financial Warfare," by Juan Zarate.
Criminal Investigation also actively participates in high-level espionage investigations, for many of the same reasons special agents originally worked on liquor bootlegger, organized crime, and public corruption investigations. In many high-level, complex investigations criminal actors are well insulated from culpability apportioned through traditional techniques of evidence development. The common weakness of many high level criminal activities remains money or the financial benefit, which is difficult to conceal.
So that's what the IRS-CI special agents are for. Not for the average taxpayer/tax-not-payer, but for actual criminal activity involving money laundering/trafficking and funding naughty people. Doesn't mean they wouldn't get as petty as any other IRS personnel or federal "special agents", but officially speaking there's a legitimate reason. After all, the Secret Service was created to protect the country's financial infrastructure against counterfeiters etc, not just to protect the grand-high-muckamucks.
But if the ATF is willing to shoot my dog, I'm sure some IRS fellow would be just as happy to shoot me. Ditto with FBI, Secret Service, whomever else. In a normal world, one wouldn't have to worry about that sort of thing, of course...