I am also much worse around my extended family. They are a loud bunch who talk over each other and i think that pushes me over the edge of fluency. I have skipped some family gatherings over the years, but not the big ones. That being said, I think you should try it again. They are your family. They love you and accept you. If they don’t, well then f them. But I promise you won’t regret trying to spend more time with family.