I think for most people who stutter, things like this pop up all the time, so I think others can relate to even the smallest things being affected in their lives so I think people can relate when you say, "I know it seems small", but to most who stutter, it is not.
I would recommend that you go ahead and do the voice chat. If you want, just tell him/her that you stutter in the beginning and if he/she minds then who cares. Without more information though it is really kind of hard to give advise. I think everyone's situation is a bit unique and each person's way to improving speech/attitude/life is different. However, as a general rule, in my opinion it is probably better not to avoid situations because of a stutter.
"Everybody is a genius. But if you judge a fish by its ability to climb a tree, it will live its whole life believing it is stupid" - Einstein
"All things are possible for the one who believes" - Jesus
"Those who believe in telekinetics, raise my hand" - Vonnegut