As long as you get frustrated, his speech will just get worse. We can pick up on such cues easy, and the added pressure of not upsetting your spouse or the People you love adds much stress. What he needs is patience, understanding, and reassurance. Dating for people who stutter is very difficult and he is likely afraid of losing you adding more pressure and stress. As frustrating as his speech is for you I guarantee its infintly for frustrating for him. And it doesn't take him a long time to process what you are saying, it's taking him a long time to respond.