Odds are you will never see the people at the table again. You can be quiet and just not really talk to them! One thing I do at stuff like this is wait to the very last moment to sit down, to reduce small talk.