Hey close!
I hate these things to, i've never been very close to my distant family, although I have no problem with my immiediate family. I feel like I ought to know them and be able to talk freely with them, but in many ways I find them worse than strangers. What I try to do at these things, is rather than engage the whole intimidating group of 10+ people, is to break it down into smaller groups and deal with them, 2-3 people I can manage, and often you'll see these big groups will decsend into several concurent conversations. Then I just ignore everyone else and can focus in on one conversation.
One tactic that I use a lot, is I have an uncle whos a massive hyperchondriac. I think he just talks about his health for lack of anything else to discuss. But anyway, he will talk for hours about it as we often joke behind his back, im sure theres someone in your family who has an interest and when they get going, they just won't shut up. What I try to do is get into the group he is talking to and just keep him going on it, asking him small questions to stop the conversation dieing. How are you feeling? - Oh, How long have you had that? - And the doctor didn't realise?, you can keep it going for hours! If you feel more confident you can even start to talk back more, throwing in your own stories to keep it going. For example "You know I was at the hospital a few weeks ago and I swear the nurse didn't clean her hands after going to the toilet!".
Equally, since they're your family you probably know a little bit about their past, you can always ask them about that. Quiz them about things they told you before, maybe the said something about booking a holiday - "so did you ever get around to booking that holiday?" and then see where it goes. It'll die out eventually but if your lucky a broad topic like holidays could keep going for hours!
I guess I try to break conversations down into statements, questions and responses. So a statment would be "I booked a holiday the other week", the response would be "oh did you, you know I wish I could go on holiday, but I don't have the money at the moment" and the question would be "so whereabouts are you going, how long for?" If you keep them going with questions, they WILL reply with something. Obviously you don't want to go overboard and look like an inquisition so throw in the occasional response. It will seem and feel more like a normal conversation with a friend.
I know its tough, but try not to fiddle with yourself, its probably causing you to dwell on the situation. Sometimes I have to convince myself that things are going okay and that I need not panic before I can actually settle down into it, and not let little things like people accidentally blanking me kill my confidence.
Hope this helps close, let us know how it goes at the next one! give it a try, what do you have to lose!