For me it's quite noticeable the negative effect that caffeine has on my stutter (+ anxiety too). There's also the loop where you use caffeine because you're constantly tired and don't rest enough, which makes things even worse. I'd try to get off of it (in all forms) for at least a month to see how it affects you. But it can be quite hard in the beginning, so maybe try it when you're not too busy with stuff from college.