I think anyone who has listened to a recording of themselves talking, especially public speaking, has that feeling of “why do I sound like an idiot”?
Who knows, maybe there’s a few who don’t. I mean, I can’t imagine Morgan Freeman listens to his narration in movies and just says “Wow, that’s terrible, how can anyone listen to that”?
I think the truth is that almost everyone deals with these feelings. But when it comes down to it: do you dislike it because you actually sound bad, or are you just critical of everything that you do?
If it’s is the latter, great! Snap out of it and get your mind off yourself you narcissistic simpleton.
If it’s the former, then, of course, you should work on it. Speaking is so broad though, where do you even begin?
The three categories that come top of mind are: content, voice, and technique.
Content: Does what you say make sense? Is it easy for others to follow? Did you prove the point that you wanted to prove? Did you have a point? (If not, please see question one.) Was what you said relatable to the audience you had?
Voice: Did you use odd or forced voice inflections and/or force laughter in moments that should be left behind? (No, you don’t need to laugh at your joke if no one else did.) Is your overall tone whiny or monotone? Does it sound like you’re reading from a script? Did you trail out on the end of all your sentences?
Technique: Did you say “um” or “like” 14 times before you ended your first sentence? (Please fix this now.) Did you use the same word repeatedly in the same sentence? Did you take 3 minutes to say what should have been said in 10 seconds? Did you ask a question and then elaborate on it for two minutes so that by the end no one knew the question?
So, this list seems kind of tough, right? How could I be so mean to someone who is trying to do a good job speaking? Well, let me tell you a little secret, this entire list was directed at myself.
The last several weeks I have hosted interviews with four different people, and as I listen to the playback of each of the episodes, I cringe as I hear myself talk. Instead of just being content with the mediocracy, I decided to make a list of questions for all of the things that could be improved and I am working on implementing each of them as I continue future episodes. For real, just for reference, I’m going to attach an audio segment from one of my interviews. I had some pretty downright awful sentences.
Here’s a small clip from the last interview I had. I cringe even putting it on here as it is so incredibly awkward to even listen to, but I think it’s important to show the bad right along with the good that I produce.
Sounds like I took lessons from this guy: 
No one’s a perfect speaker, but you’ll never even get past beginner if you don’t fix the things that need fixing. Also, the longer you leave the bad habits in your speech, the harder it will be to take them out.
Start now, learn good habits, and speak so well that no one thinks about it.