Sounds like you're taking it too personally and letting it get to you which is probably the intent. If it becomes a frequent thing, then yeah, pursue it. Otherwise, you've got to know by now that people do this type of shit randomly for "fun". Your first mistake was giving him fuel by continuing the conversation for some reason. Do you argue with soundboard prank calls when Arnold Schwarzenegger calls you telling you someone's in his house eating his cookies, and asking who is your daddy and what does he do? If so... then disregard my post.
I'm sure he doesn't know you or your wife, so I doubt he was actually insulting you or your wife.