My opinion is that if you feel that way, you should take some action towards your goal: "Whatever you think you can do or believe you can do, begin it. Action has magic, grace and power in it." -- Goethe

If you are interested in trying to teach an occasional course on an adjunct basis, let me know: I will introduce you to a friend who is a dean at a small, private, local college. He might have a place for you - or at least some referrals/advice.