I usually make the cake and it sits for a couple of hours before it's served and then I refrigerate it. I like it warm to room temp and cold equally as well. I think letting it sit will allow the milks to meld a little before chilling it. I really don't think it will make much of a difference though, so I'd probably do what the recipe I am using tells me to do.
You can't go wrong either way.