For the rabri (sweetened condensed milk), put whole milk and cream in a tall pot, bring to a boil. Lower to medium heat for 45 minutes, then lower to medium-low heat for another 45 minutes, whisking constantly. Will reduce by half. Take off heat, add the cardamom and condensed milk, and whisk. Set aside to cool in the fridge.
For the saffron syrup, combine water, sugar, saffron threads, and rose water in pot and bring to a boil. Whisk enough to dissolve sugar. Take off heat and set aside to cool.
Brush both sides of each Brioche slice with ghee and pan-fry until light golden on each side. Set aside to cool.
Quickly soak each slice with the saffron syrup and put on paper towels to drain excess. Right before serving, microwave toasts for 30 seconds to warm through slightly. Pour the room-temperature rabri on each slice and top with pistachios, cashews, and rose petals.