I've met a few individuals who say their stuttering increases when speaking to other people who stutter. And, I've also met plenty who say their stuttering decreases. It really just seems to depend on the person. It seems like you might be more comfortable if you acknowledged your stuttering to him, since you're speculating on his feelings and you don't know if it's affecting him or not, which sounds like it's causing some stress for you. If you're concerned that he'll notice the change in your speech with him vs. other clients, you could say something referring generally to your speech being inconsistent-- avoiding a specific "And since you stutter too it makes mine worse!"