Preheat oven to 350 degrees. Grease baking dish with softened butter.
Mix 3/4 cup brown sugar, 1 tbsp ground cinnamon, 1/2 tsp ground nutmeg, and punch of salt together in a bowl. Sprinkle 3 tbsp brown sugar mixture over bottom of dish. Place 6 potato bread slices in even layer in bottom of dish. Brush bread with melted butter and sprinkle with 3 tbsp sugar mixture. Build two more layers the same way.
In separate bowl, whisk 2 1/2 cup of milk and 6 large eggs together. Pour milk mixture over bread and press lightly. Sprinkle with some almond slices and remaining sugar mixture.
Bake casserole for ~30 min. Let cool for 15 minutes. Sprinkle with confectioners’ sugar and/or maple syrup to serve.