Bypassing the "why" of all this...
Find a decent chest rig that fits your needs. Then, later on down the line, if you decide you need armor, get a plate carrier. You can run the chest rig over the plate carrier like a lot of guys do. Or, if you go with a modular chest rig, you can just...