Make it a potluck where everyone brings their own dishes. If that doesn't work, think about Asian dishes which are often naturally vegan and gluten free.
I have a good relationship with gluten and am no expert, but I do know that soy sauce often contains a lot of gluten. I've seen gluten-free soy sauce - but just something to keep in mind if thinking about Asian dishes.
Tamari is gluten-free. I agree with having them bring their own food.
I was going to invite some people for a holiday recently and decided not to. This was why: it was my holiday as well and my family had decided what we wanted to eat. I generally go out of my way to accommodate all manner of idiosyncratic eating. This time I said no vegans, no vegetarians who may or may not eat these vegetarian dishes, no picky eaters, no kosher rules, no fake "allergies" (real allergies fine), etc. So we had 1 friend and I was thrilled. Sometimes just have to put oneself first and keep it simple.
Or tell them to cook for you.