Many have already pointed out much of the frustration and difficulties involved but there is also great reward. You said you don't need to do it as a sole source of income so it sounds like you really wouldn't have much to lose other than initial investment in machines. If you aren't in a hurry and you find a nice used lathe you won't be risking much there.
I'd recommend you try it but as others have pointed out you should expect to do a fair amount of free work in the beginning. I did lots of free labor on builds for friends and competitive shooters I knew in the beginning. In those days I was also competing a lot and usually did ok so that also got me some attention. Unlike you I am in it for a full time career and only source of income so it has become a consuming drive in my life. I rarely get to shoot now and I just never have enough time to return every phone call and email I get. It requires many more hours than I ever dreamed, cost much more than I expected in the beginning, and my margins on each piece are far less than I expected. I make up for that by working longer and longer hours. Each time I hit what I think is a threshold and can't work anymore, I figure out a way to make adjustments and work more. For the most part if I'm not in my shop, I'm home sleeping or out doing something work related. Not only did I underestimate regular shop expenses, there are surprises I didn't expect such as registration with ITAR and of course the wonderful 11% federal excise tax that I have to pay based on the sale price of every rifle I deliver and they are usually $3,200 to $4,700. That part is a real treat. I can't pass that on to the consumer like sales tax it comes directly out of my margin. I don't mean to ramble but you get the point.
There are a lot of guys out there in garages across the country doing outstanding work for friends and local shooters for side money. I don't think it's difficult to do it on that level. Of course all of this is dependent on you actually being able to perform good quality work. Experienced guys helped me when I was getting started and if you choose to do it I'd help you with advice and answers if you'd like.
Marc