Do you have to use nutmeg? Not a big fan.

Definitely not! Just omit it. (Though the taste is pretty subtle in the finished product.)

You may always change an ingredient, if you don't like it or if it disagrees with you.
Some may argue that you can't call a dish by its traditional name once you omit a key ingredient. But that's a naming and tradition issue, not a cooking question.
But either way, it's your choice.
So, omit the nutmeg and see how it tastes.