I completely agree. Unless they're being wildly obnoxious and continue to belittle you after you explain it, just let it go. Stuttering isn't a super well-known condition, some people are bound to mistake it for a joke, nerves or a one-off mistake every now and then.