When I saw your answer I realized I asked the wrong question. What I meant to ask is how much in advance (before baking) can you prepare this and should you stop before adding the milk? Thanks!
You really shouldn't assemble this too far ahead of time without cooking it, as the potatoes will oxidize and discolor. If you want, you can cook it almost all the way through, then cool, cover and refrigerate. Return to room temp before finishing/reheating in the oven.